Visible to the public A Review of Evaluation Techniques for Social Dialogue Systems

TitleA Review of Evaluation Techniques for Social Dialogue Systems
Publication TypeConference Paper
Year of Publication2017
AuthorsCurry, Amanda Cercas, Hastie, Helen, Rieser, Verena
Conference NameProceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents
PublisherACM
Conference LocationNew York, NY, USA
ISBN Number978-1-4503-5558-2
KeywordsAutomatic Evaluation, conversational agents, Evaluation Metrics, Human Behavior, human factors, pubcrawl, Scalability, Social Agents, Social Dialogue Systems
Abstract

In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.

URLhttps://dl.acm.org/doi/10.1145/3139491.3139504
DOI10.1145/3139491.3139504
Citation Keycurry_review_2017