Measuring Heterogeneous User Behaviors During the Interaction with Dialog Systems

  • David GriolEmail author
  • José Manuel Molina
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 616)


In this paper, we describe a technique to develop simulated user agents that are able to interact with dialog systems. By means of these agents, it is possible not only to automatically evaluate the overall operation of the dialog system, but also to assess the impact of the user responses on the decisions that are selected by the system. The selection of the user responses by the simulated user agent are based on a statistical model that is automatically learned from a dialog corpus. The complete history of the interaction is considered to carry out this selection. The paper describes the application of this technique to evaluate a practical dialog system providing tourist information and services.


Dialog systems Agent simulation Human-machine interaction User modeling System evaluation Statistical methodologies 



This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485).


  1. 1.
    Bishop, C.M.: Neural networks for pattern recognition. Oxford University Press, Bristol (1995)zbMATHGoogle Scholar
  2. 2.
    Chung, G.: Developing a flexible spoken dialog system using simulation. In: Proceedings of 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), pp. 63–70 (2004)Google Scholar
  3. 3.
    Dutoit, T.: An introduction to text-to-speech synthesis. Kluwer Academic Publishers (1996)Google Scholar
  4. 4.
    Engelbrecht, K.: Estimating Spoken Dialog System Quality with User Models. T-Labs Series in Telecommunication Services. Springer, Heidelberg (2012)Google Scholar
  5. 5.
    García, F., Hurtado, L.F., Sanchis, E., Segarra, E.: The incorporation of confidence measures to language understanding. In: Matoušek, V., Mautner, P. (eds.) TSD 2003. LNCS (LNAI), vol. 2807, pp. 165–172. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Griol, D., Callejas, Z., López-Cózar, R., Riccardi, G.: A domain-independent statistical methodology for dialog management in spoken dialog systems. Comput. Speech Lang. 28(3), 743–768 (2014)CrossRefGoogle Scholar
  7. 7.
    Griol, D., Molina, J.M.: Modeling users emotional state for an enhanced human-machine interaction. In: Onieva, E., Santos, I., Osaba, E., Quintian, H., Corchado, E. (eds.) HAIS 2015. LNCS, vol. 9121, pp. 357–368. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  8. 8.
    Lemon, O.: Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation. Comput. Speech Lang. 25, 210–221 (2011)CrossRefGoogle Scholar
  9. 9.
    Lin, B., Lee, L.: Computer aided analysis and design for spoken dialogue systems based on quantitative simulations. IEEE Trans. Speech Audio Process. 9(5), 534–548 (2001)CrossRefGoogle Scholar
  10. 10.
    López-Cózar, R., de la Torre, A., Segura, J., Rubio, A.: Assessment of dialogue systems by means of a new simulation technique. Speech Commun. 40, 387–407 (2003)CrossRefGoogle Scholar
  11. 11.
    Möller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta,A., Raake, A., Reithinger, N.: MeMo: towards automatic usability evaluationof spoken dialogue services by user error simulations. In: Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech/ICSLP), pp. 1786–1789. Pittsburgh, USA (2006)Google Scholar
  12. 12.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: PDP: Computational models of cognition and perception, I, chap. Learning internal representations by error propagation, pp. 319–362. MIT Press, Cambridge (1986)Google Scholar
  13. 13.
    Schatzmann, J., Thomson, B., Young, S.: Error simulation for training statistical dialogue systems. In: Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2007), pp. 273–282. Kyoto, Japan (2007)Google Scholar
  14. 14.
    Schatzmann, J., Weilhammer, K., Stuttle, M., Young, S.: A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowl. Eng. Rev. 21(2), 97–126 (2006)CrossRefGoogle Scholar
  15. 15.
    Tsilfidis, A., Mporas, I., Mourjopoulos, J., Fakotakis, N.: Automatic speech recognition performance in different room acoustic environments with and without dereverberation preprocessing. Comput. Speech Lang. 27(1), 380–395 (2013)CrossRefGoogle Scholar
  16. 16.
    Williams, J., Young, S.: Partially observable markov decision processes for spoken dialog systems. Comput. Speech Lang. 21(2), 393–422 (2007)CrossRefGoogle Scholar
  17. 17.
    Wu, W.L., Lu, R.Z., Duan, J.Y., Liu, H., Gao, F., Chen, Y.Q.: Spoken language understanding using weakly supervised learning. Comput. Speech Lang. 24(2), 358–382 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Computer Science DepartmentCarlos III University of MadridLeganésSpain

Personalised recommendations