Skip to main content
SpringerLink
Log in

Recurrent Neural Network Interaction Quality Estimation

  • Chapter
  • First Online:
Dialogues with Social Robots

Part of the book series: Lecture Notes in Electrical Engineering ((LNEE,volume 427))

Abstract

Getting a good estimation of the Interaction Quality (IQ) of a spoken dialogue helps to increase the user satisfaction as the dialogue strategy may be adapted accordingly. Therefore, some research has already been conducted in order to automatically estimate the Interaction Quality. This article adds to this by describing how Recurrent Neural Networks may be used to estimate the Interaction Quality for each dialogue turn and by evaluating their performance on this task. Here, we will show that RNNs may outperform non-recurrent neural networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Unweighted Average Recall, see Sect. 4.1.

  2. 2.

    A system-user exchange comprises a system turn followed by a user turn.

  3. 3.

    UAR, \(\kappa \) and \(\rho \) are defined in Sect. 4.1.

References

  1. Ultes, S., Heinroth, T., Schmitt, A., Minker, W.: A theoretical framework for a user-centered spoken dialog manager. In: Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. pp. 241—246. Springer (2011)

    Google Scholar 

  2. Ultes, S., Dikme, H., Minker, W.: Dialogue management for user-centered adaptive dialogue. In: Proceedings of IWSDS (2014). http://www.uni-ulm.de/fileadmin/website_uni_ulm/allgemein/2014_iwsds/iwsds2014_lp_ultes.pdf

  3. Ultes, S., Dikme, H., Minker, W.: First insight into quality-adaptive dialogue. In: Proceedings of the LREC, pp. 246–251 (2014)

    Google Scholar 

  4. Ultes, S., Kraus, M., Schmitt, A., Minker, W.: Quality-adaptive spoken dialogue initiative selection and implications on reward modelling. In: Proceedings of SIGDIAL, pp. 374–383. ACL (2015)

    Google Scholar 

  5. Ultes, S., Minker, W.: Improving interaction quality recognition using error correction. In: Proceedings of SIGDIAL, pp. 122–126. ACL (2013). http://www.aclweb.org/anthology/W/W13/W13-4018

  6. Ultes, S., Minker, W.: Interaction quality: a review. Bulletin of Siberian State Aerospace University named after academician M.F. Reshetnev (4), 153–156 (2013). http://www.vestnik.sibsau.ru/images/vestnik/ves450.pdf

  7. Ultes, S., Platero Sánchez, M.J., Schmitt, A., Minker, W.: Analysis of an extended interaction quality corpus. In: Proceedings of IWSDS (2015)

    Google Scholar 

  8. Ultes, S., ElChabb, R., Minker, W.: Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In: Proceedings of IWSDS, pp. 141–150. Springer (2012)

    Google Scholar 

  9. Ultes, S., Minker, W.: Interaction quality estimation in spoken dialogue systems using hybrid-hmms. In: Proceedings of SIGDIAL, pp. 208–217. ACL (2014). http://www.aclweb.org/anthology/W14-4328

  10. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  11. Mesnil, G., He, X., Deng, L., Bengio, Y.: Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In: Proceedings of the INTERSPEECH, pp. 3771–3775 (2013)

    Google Scholar 

  12. Henderson, M., Thomson, B., Young, S.: Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation. In: Proceedings of the SLT, pp. 360–365. IEEE (2014)

    Google Scholar 

  13. Wen, T.H., Gačić, M., Kim, D., Mrkšic, N., Su, P.H., Vandyke, D., Young, S.: Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. In: Proceedings of the ACL, pp. 275–284. (2015)

    Google Scholar 

  14. Su, P.H., Vandyke, D., Gasic, M., Kim, D., Mrksic, N., Wen, T.H., Young, S.: Learning from real users: rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. arXiv:1508.03386 (2015)

  15. Walker, M., Litman, D.J., Kamm, C.A., Abella, A.: Paradise: a framework for evaluating spoken dialogue agents. In: Proceedings of ACL, pp. 271–280. ACL, Morristown, NJ, USA (1997)

    Google Scholar 

  16. Engelbrecht, K.P., Gödde, F., Hartard, F., Ketabdar, H., Möller, S.: Modeling user satisfaction with hidden markov model. In: Proceedings of SIGDIAL, pp. 170–177. ACL, Morristown, NJ, USA (2009)

    Google Scholar 

  17. Higashinaka, R., Minami, Y., Dohsaka, K., Meguro, T.: Issues in predicting user satisfaction transitions in dialogues: Individual differences, evaluation criteria, and prediction models. In: Spoken Dialogue Systems for Ambient Environments, Lecture Notes in Computer Science, vol. 6392, pp. 48–60. Springer, Heidelberg (2010). doi:10.1007/978-3-642-16202-2_5

  18. Hara, S., Kitaoka, N., Takeda, K.: Estimation method of user satisfaction using n-gram-based dialog history model for spoken dialog system. In: Proceedings of LREC. ELRA, Valletta, Malta (2010)

    Google Scholar 

  19. Schmitt, A., Ultes, S.: Interaction quality: assessing the quality of ongoing spoken dialog interaction by experts—and how it relates to user satisfaction. Speech Commun. (2015) accepted for publication

    Google Scholar 

  20. Ultes, S., Schmitt, A., Minker, W.: Towards quality-adaptive spoken dialogue management. In: Proceedings of the NAACL-HLT: SDCTD, pp. 49–52. ACL, Montréal, Canada (2012). http://www.aclweb.org/anthology/W12-1819

  21. Ultes, S., Schmitt, A., Minker, W.: On quality ratings for spoken dialogue systems—experts vs. users. In: Proceedings of NAACL-HLT, pp. 569–578. ACL (2013)

    Google Scholar 

  22. Schmitt, A., Schatz, B., Minker, W.: Modeling and predicting quality in spoken human-computer interaction. In: Proceedings of SIGDIAL, pp. 173–184. ACL, Portland, Oregon, USA (2011)

    Google Scholar 

  23. Raux, A., Bohus, D., Langner, B., Black, A.W., Eskenazi, M.: Doing research on a deployed spoken dialogue system: one year of let’s go! experience. In: Proceedings of ICSLP (2006)

    Google Scholar 

  24. Schmitt, A., Ultes, S., Minker, W.: A parameterized and annotated spoken dialog corpus of the CMU Let’s go bus information system. In: Proceedings of the LREC, pp. 3369–337 (2012)

    Google Scholar 

  25. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)

    Article  Google Scholar 

  26. Lin, T., Horne, B.G., Tiňo, P., Giles, C.L.: Learning long-term dependencies in narx recurrent neural networks. IEEE Trans. Neural Netw. 7(6), 1329–1338 (1996)

    Article  Google Scholar 

  27. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.J.: Phoneme recognition using time-delay neural networks. IEEE Trans. Acoust. Speech Signal Process. 37(3), 328–339 (1989)

    Article  Google Scholar 

  28. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)

    Google Scholar 

  29. Spearman, C.E.: The proof and measurement of association between two things. Am. J. Psychol. 15, 88–103 (1904)

    Google Scholar 

  30. Cohen, J.: Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol. Bull. 70(4), 213 (1968)

    Article  Google Scholar 

  31. De Jesús, O., Hagan, M.T.: Backpropagation algorithms for a broad class of dynamic networks. IEEE Trans. Neural Netw. 18(1), 14–27 (2007)

    Article  Google Scholar 

  32. De Jesús, O., Horn, J.M., Hagan, M.T.: Analysis of recurrent network training and suggestions for improvements. In: Proccedings of the International Joint Conference on Neural Networks (IJCNN’01) 2001. vol. 4, pp. 2632–2637. IEEE (2001)

    Google Scholar 

  33. Horn, J., De Jesús, O., Hagan, M.T.: Spurious valleys in the error surface of recurrent networksanalysis and avoidance. IEEE Trans. Neural Netw. 20(4), 686–700 (2009)

    Article  Google Scholar 

Download references

Acknowledgements

This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012.

Author information

Authors and Affiliations

  1. Institute of Communications Engineering, Ulm University, Ulm, Germany

    Louisa Pragst & Wolfgang Minker

  2. Engineering Department, Cambridge University, Cambridge, UK

    Stefan Ultes

Authors
  1. Louisa Pragst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefan Ultes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolfgang Minker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Louisa Pragst .

Editor information

Editors and Affiliations

  1. Institute of Behavioural Sciences, University of Helsinki Institute of Behavioural Sciences, Helsinki, Finland

    Kristiina Jokinen

  2. University of Helsinki , Helsinki, Finland

    Graham Wilcock

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer Science+Business Media Singapore

About this chapter

Cite this chapter

Pragst, L., Ultes, S., Minker, W. (2017). Recurrent Neural Network Interaction Quality Estimation. In: Jokinen, K., Wilcock, G. (eds) Dialogues with Social Robots. Lecture Notes in Electrical Engineering, vol 427. Springer, Singapore. https://doi.org/10.1007/978-981-10-2585-3_31

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-2585-3_31

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-2584-6

  • Online ISBN: 978-981-10-2585-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation