Predicting Interaction Quality in Customer Service Dialogs

Abstract

In this paper, we apply a dialog evaluation Interaction Quality (IQ) framework to human-computer customer service dialogs. IQ framework can be used to predict user satisfaction at an utterance level in a dialog. Such a rating framework is useful for online adaptation of dialog system behavior and increasing user engagement through personalization. We annotated a dataset of 120 human-computer dialogs from two customer service application domains with IQ scores. Our inter-annotator agreement ($\rho =0.72/0.66$) is similar to the agreement observed on the IQ annotations of publicly available bus information corpus. The IQ prediction performance of an in-domain SVM model trained on a small set of call center domain dialogs achieves a correlation of $\rho =0.53{/}0.56$ measured against the annotated IQ scores. A generic model built exclusively on public LEGO data achieves 94%/65% of the in-domain model’s performance. An adapted model built by extending a public dataset with a small set of dialogs in a target domain achieves 102%/81% of the in-domain model’s performance.