Visible to the public BoTest: A Framework to Test the Quality of Conversational Agents Using Divergent Input Examples

TitleBoTest: A Framework to Test the Quality of Conversational Agents Using Divergent Input Examples
Publication TypeConference Paper
Year of Publication2018
AuthorsRuane, Elayne, Faure, Théo, Smith, Ross, Bean, Dan, Carson-Berndsen, Julie, Ventresque, Anthony
Conference NameProceedings of the 23rd International Conference on Intelligent User Interfaces Companion
PublisherACM
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-5571-1
Keywordschatbot, conversation agents, conversational agent, Conversational Agent Quality Assessment, Conversational Agent Testing, Human Behavior, Metrics, pubcrawl, Scalability
AbstractQuality of conversational agents is important as users have high expectations. Consequently, poor interactions may lead to the user abandoning the system. In this paper, we propose a framework to test the quality of conversational agents. Our solution transforms working input that the conversational agent accurately recognises to generate divergent input examples that introduce complexity and stress the agent. As the divergent inputs are based on known utterances for which we have the 'normal' outputs, we can assess how robust the conversational agent is to variations in the input. To demonstrate our framework we built ChitChatBot, a simple conversational agent capable of making casual conversation.
URLhttp://doi.acm.org/10.1145/3180308.3180373
DOI10.1145/3180308.3180373
Citation Keyruane_botest:_2018