Skip to main content
SpringerLink
Log in

Measuring Heterogeneous User Behaviors During the Interaction with Dialog Systems

  • Conference paper
  • First Online:
Highlights of Practical Applications of Scalable Multi-Agent Systems. The PAAMS Collection (PAAMS 2016)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 616))

  • 729 Accesses

Abstract

In this paper, we describe a technique to develop simulated user agents that are able to interact with dialog systems. By means of these agents, it is possible not only to automatically evaluate the overall operation of the dialog system, but also to assess the impact of the user responses on the decisions that are selected by the system. The selection of the user responses by the simulated user agent are based on a statistical model that is automatically learned from a dialog corpus. The complete history of the interaction is considered to carry out this selection. The paper describes the application of this technique to evaluate a practical dialog system providing tourist information and services.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bishop, C.M.: Neural networks for pattern recognition. Oxford University Press, Bristol (1995)

    MATH  Google Scholar 

  2. Chung, G.: Developing a flexible spoken dialog system using simulation. In: Proceedings of 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), pp. 63–70 (2004)

    Google Scholar 

  3. Dutoit, T.: An introduction to text-to-speech synthesis. Kluwer Academic Publishers (1996)

    Google Scholar 

  4. Engelbrecht, K.: Estimating Spoken Dialog System Quality with User Models. T-Labs Series in Telecommunication Services. Springer, Heidelberg (2012)

    Google Scholar 

  5. García, F., Hurtado, L.F., Sanchis, E., Segarra, E.: The incorporation of confidence measures to language understanding. In: Matoušek, V., Mautner, P. (eds.) TSD 2003. LNCS (LNAI), vol. 2807, pp. 165–172. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  6. Griol, D., Callejas, Z., López-Cózar, R., Riccardi, G.: A domain-independent statistical methodology for dialog management in spoken dialog systems. Comput. Speech Lang. 28(3), 743–768 (2014)

    Article  Google Scholar 

  7. Griol, D., Molina, J.M.: Modeling users emotional state for an enhanced human-machine interaction. In: Onieva, E., Santos, I., Osaba, E., Quintian, H., Corchado, E. (eds.) HAIS 2015. LNCS, vol. 9121, pp. 357–368. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  8. Lemon, O.: Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation. Comput. Speech Lang. 25, 210–221 (2011)

    Article  Google Scholar 

  9. Lin, B., Lee, L.: Computer aided analysis and design for spoken dialogue systems based on quantitative simulations. IEEE Trans. Speech Audio Process. 9(5), 534–548 (2001)

    Article  Google Scholar 

  10. López-Cózar, R., de la Torre, A., Segura, J., Rubio, A.: Assessment of dialogue systems by means of a new simulation technique. Speech Commun. 40, 387–407 (2003)

    Article  Google Scholar 

  11. Möller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta,A., Raake, A., Reithinger, N.: MeMo: towards automatic usability evaluationof spoken dialogue services by user error simulations. In: Proceedings of the 9th International Conference on Spoken Language Processing (Interspeech/ICSLP), pp. 1786–1789. Pittsburgh, USA (2006)

    Google Scholar 

  12. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: PDP: Computational models of cognition and perception, I, chap. Learning internal representations by error propagation, pp. 319–362. MIT Press, Cambridge (1986)

    Google Scholar 

  13. Schatzmann, J., Thomson, B., Young, S.: Error simulation for training statistical dialogue systems. In: Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2007), pp. 273–282. Kyoto, Japan (2007)

    Google Scholar 

  14. Schatzmann, J., Weilhammer, K., Stuttle, M., Young, S.: A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowl. Eng. Rev. 21(2), 97–126 (2006)

    Article  Google Scholar 

  15. Tsilfidis, A., Mporas, I., Mourjopoulos, J., Fakotakis, N.: Automatic speech recognition performance in different room acoustic environments with and without dereverberation preprocessing. Comput. Speech Lang. 27(1), 380–395 (2013)

    Article  Google Scholar 

  16. Williams, J., Young, S.: Partially observable markov decision processes for spoken dialog systems. Comput. Speech Lang. 21(2), 393–422 (2007)

    Article  Google Scholar 

  17. Wu, W.L., Lu, R.Z., Duan, J.Y., Liu, H., Gao, F., Chen, Y.Q.: Spoken language understanding using weakly supervised learning. Comput. Speech Lang. 24(2), 358–382 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by Projects MINECO TEC2012-37832-C02-01, CICYT TEC2011-28626-C02-02, CAM CONTEXTS (S2009/TIC-1485).

Author information

Authors and Affiliations

  1. Computer Science Department, Carlos III University of Madrid, Avda. de la Universidad, 30, 28911, Leganés, Spain

    David Griol & José Manuel Molina

Authors
  1. David Griol
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José Manuel Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Griol .

Editor information

Editors and Affiliations

  1. Departamento de Inteligencia Artificial, Universidad Politécnica de Madrid, Madrid, Spain

    Javier Bajo

  2. Universidad de Sevilla, Sevilla, Spain

    María José Escalona

  3. Faculté des Sciences, Université de Sherbrooke, Sherbrooke, Canada

    Sylvain Giroux

  4. University of Poznan, Poznan, Poland

    Patrycja Hoffa-Dąbrowska

  5. Universidad Politécnica de Valencia, Valencia, Spain

    Vicente Julián

  6. Department of Informatics, Universidade do Minho, Braga, Portugal

    Paulo Novais

  7. Fluminense Federal University, Rio de Janeiro, Brazil

    Nayat Sánchez-Pi

  8. Inst. Comp. Sci. & Bus. Info. Sys. (ICB), University of Duisburg-Essen, Essen, Nordrhein-Westfalen, Germany

    Rainer Unland

  9. Departamento de Informática e Estatístic, Universidade Federal de Santa Catarina, Florianópolis S.C.,, Brazil

    Ricardo Azambuja-Silveira

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Griol, D., Molina, J.M. (2016). Measuring Heterogeneous User Behaviors During the Interaction with Dialog Systems. In: Bajo, J., et al. Highlights of Practical Applications of Scalable Multi-Agent Systems. The PAAMS Collection. PAAMS 2016. Communications in Computer and Information Science, vol 616. Springer, Cham. https://doi.org/10.1007/978-3-319-39387-2_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-39387-2_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-39386-5

  • Online ISBN: 978-3-319-39387-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation