Measuring Heterogeneous User Behaviors During the Interaction with Dialog Systems
In this paper, we describe a technique to develop simulated user agents that are able to interact with dialog systems. By means of these agents, it is possible not only to automatically evaluate the overall operation of the dialog system, but also to assess the impact of the user responses on the decisions that are selected by the system. The selection of the user responses by the simulated user agent are based on a statistical model that is automatically learned from a dialog corpus. The complete history of the interaction is considered to carry out this selection. The paper describes the application of this technique to evaluate a practical dialog system providing tourist information and services.
KeywordsDialog systems Agent simulation Human-machine interaction User modeling System evaluation Statistical methodologies
This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485).
- 2.Chung, G.: Developing a flexible spoken dialog system using simulation. In: Proceedings of 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), pp. 63–70 (2004)Google Scholar
- 3.Dutoit, T.: An introduction to text-to-speech synthesis. Kluwer Academic Publishers (1996)Google Scholar
- 4.Engelbrecht, K.: Estimating Spoken Dialog System Quality with User Models. T-Labs Series in Telecommunication Services. Springer, Heidelberg (2012)Google Scholar
- 11.Möller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta,A., Raake, A., Reithinger, N.: MeMo: towards automatic usability evaluationof spoken dialogue services by user error simulations. In: Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech/ICSLP), pp. 1786–1789. Pittsburgh, USA (2006)Google Scholar
- 12.Rumelhart, D.E., Hinton, G.E., Williams, R.J.: PDP: Computational models of cognition and perception, I, chap. Learning internal representations by error propagation, pp. 319–362. MIT Press, Cambridge (1986)Google Scholar
- 13.Schatzmann, J., Thomson, B., Young, S.: Error simulation for training statistical dialogue systems. In: Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2007), pp. 273–282. Kyoto, Japan (2007)Google Scholar