Abstract
User models have become increasingly popular to conduct simulation-based testing of spoken dialog systems. These models usually describe users’ overt behavior, as opposed to the underlying reasons for the observed actions. While such models are useful to generate test data, a causal model might be more generally applicable to different systems and, in addition, allows to derive useful information for data analysis and prediction of user judgments. Thus, a modeling approach trying to explain user behavior is proposed in this paper, which is based on Dörner’s PSI theory. The evaluation shows that the utterances generated by this model are similar to those of real users.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Blackmon, M., Kitajima, M., Polson, P.: Tool for accurately predicting website navigation problems, non-problems, problem severity, and effectiveness of repairs. In: Proceedings of CHI ’05, pp. 31–40 (2005)
Branigan, H., Pickering, M., Pearson, J., McLean, J.: Linguistic alignment between humans and computers. J. Pragmat. 42, 2355–2368 (2010)
Chung, G.: Developing a flexible spoken dialog system using simulation. In: Proceedings of ACL 2004, pp. 93–98 (2004)
Dörner, D.: Die Mechanik des Seelenwagens. Eine neuronale Theorie der Handlungsregulation, 1st Edition, Verlag Hans Huber, Bern (2002)
Eckert, W., Levin, E., Pieraccini, R.: User modeling for spoken dialogue system evaluation. In: Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 80–87 (1997)
Engelbrecht, K.-P.: Estimating spoken dialog system quality with user models. Ph.D. Thesis, Berlin Institute of Technology, Berlin (2012)
Engelbrecht, K.-P., Möller, S.: Correlation between model-based approximations of grounding-related cognition and user judgments. In: Proceedings of Interspeech 2012 (2012)
López-Cózar, R., Callejas, Z., McTear, M.: Testing the performance of spoken dialogue systems by means of an artificially simulated user. Artif. Intell. Rev. 26, 291–323 (2006)
Möller, S.: Quality of Telephone-Based Spoken Dialog Systems. Springer, New York (2005)
Möller, S., Schleicher, R., Butenkov, D., Engelbrecht, K.-P., Gödde, F., Scheffler, T., Roller, R., Reithinger, N.: Usability engineering for spoken dialogue systems via statistical user models. In: Proceedings of IWSDS ’09 (2009)
Pietquin, O.: A framework for unsupervised learning of dialogue strategies. Ph.D. Thesis, TCTS Lab, Mons (2004)
Schatzmann, J., Georgila, K., Young., S.: Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proceedings of SIGDial, pp. 45–54 (2005)
Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S.: Agenda-based user simulation for bootstrapping a pomdp dialogue system. In: Proceedings of HLT/NAACL, pp. 149–152 (2007)
Walker, M., Litman, D., Kamm, C., Abella, A.: Paradise: A framework for evaluating spoken dialogue agents. In: Proceedings of ACL/EACL, pp. 271–280 (1997)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this paper
Cite this paper
Engelbrecht, KP. (2014). A User Model for Dialog System Evaluation Based on Activation of Subgoals. In: Mariani, J., Rosset, S., Garnier-Rizet, M., Devillers, L. (eds) Natural Interaction with Robots, Knowbots and Smartphones. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8280-2_33
Download citation
DOI: https://doi.org/10.1007/978-1-4614-8280-2_33
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-8279-6
Online ISBN: 978-1-4614-8280-2
eBook Packages: EngineeringEngineering (R0)