An Online Algorithm for Applying Reinforcement Learning to Handle Ambiguity in Spoken Dialogues

  • Fangju Wang
  • Kyle Swegles
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5532)


Spoken dialogue systems (SDSs) have been widely used in human-computer communications, including database querying, online trouble shooting advising, etc. A major challenge in building an SDS is to handle ambiguity in natural languages. User queries, questions, descriptions in a natural language may be ambiguous. To be effective in practical applications, an SDS must be able to disambiguate input from its user(s). In our research, we develop an online algorithm for applying reinforcement learning to handle ambiguity in SDSs. We introduce a new user dialogue policy into the framework of reinforcement learning to model user dialogue behavior. Also, differing from the current reinforcement learning algorithms in speech and language processing that are characterized by offline training, our algorithm conducts both offline and online detection of user dialogue behavior. In this paper, we present the online algorithm for reinforcement learning, emphasizing the detection of user dialogue behavior. We also describe the initial implementation and experiments.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Frampton, M., Lemon, O.: Learning more effective dialogue strategies using limited dialogue move features. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pp. 185–192. Association for Computational Linguistics, Sydney (2006)Google Scholar
  2. 2.
    Griol, D., Hurtado, L.F., Segarra, E., Sanchis, E.: A statistical approach to spoken dialog systems design and evaluation. Speech Communication 50, 666–682 (2008)CrossRefGoogle Scholar
  3. 3.
    Jokinen, K., Kerminen, A., Kaipainen, M., Jauhiainen, T., Wilcock, G., Turunen, M., Hakulinen, J., Kuusisto, J., Lagus, K.: Adaptive dialogue systems - interaction with interact. In: Proceedings of the 3rd SIGdial workshop on Discourse and dialogue, vol. 2, pp. 64–73. Association for Computational Linguistics, Philadelphia (2002)CrossRefGoogle Scholar
  4. 4.
    Levin, E., Pieraccini, R., Eckert, W.: A stochastic model of human-machine interaction for learning dialog strategies. IEEE Transactions on Speech and Audio Processing 8, 11–23 (2000)CrossRefGoogle Scholar
  5. 5.
    Litman, D.J., Kearns, M.S., Singh, S., Walker, M.A.: Automatic optimization of dialogue management. In: Proceedings of the 18th conference on Computational linguistics, vol. 1, pp. 502–508. Association for Computational Linguistics, Saarbrücken (2000)CrossRefGoogle Scholar
  6. 6.
    Melichar, M., Cenek, P.: From vocal to multimodal dialogue management. In: Proceedings of the 8th international conference on Multimodal interfaces, pp. 59–67. ACM, Banff (2006)CrossRefGoogle Scholar
  7. 7.
    Mitchell, T.M.: Machine learning. WCB McGraw-Hill, New York (1997)zbMATHGoogle Scholar
  8. 8.
    Scheffler, K., Young, S.: Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In: Proceedings of the second international conference on Human Language Technology Research, pp. 12–19. Morgan Kaufmann Publishers Inc., San Diego (2002)Google Scholar
  9. 9.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2005)Google Scholar
  10. 10.
    Swegles, K., Wang, F.: A hybrid reinforcement learning algorithm for optimizing dialogue strategies in spoken dialogue systems. In: The 26th International Conference on Machine Learning (ICML 2009), Montreal, Canada (submitted, 2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Fangju Wang
    • 1
  • Kyle Swegles
    • 1
  1. 1.University of Guelph, GuelphOntarioCanada

Personalised recommendations