Advertisement

QRFA: A Data-Driven Model of Information-Seeking Dialogues

  • Svitlana VakulenkoEmail author
  • Kate Revoredo
  • Claudio Di Ciccio
  • Maarten de Rijke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11437)

Abstract

Understanding the structure of interaction processes helps us to improve information-seeking dialogue systems. Analyzing an interaction process boils down to discovering patterns in sequences of alternating utterances exchanged between a user and an agent. Process mining techniques have been successfully applied to analyze structured event logs, discovering the underlying process models or evaluating whether the observed behavior is in conformance with the known process. In this paper, we apply process mining techniques to discover patterns in conversational transcripts and extract a new model of information-seeking dialogues, QRFA, for Query, Request, Feedback, Answer. Our results are grounded in an empirical evaluation across multiple conversational datasets from different domains, which was never attempted before. We show that the QRFA model better reflects conversation flows observed in real information-seeking conversations than models proposed previously. Moreover, QRFA allows us to identify malfunctioning in dialogue system transcripts as deviations from the expected conversation flow described by the model via conformance analysis.

Keywords

Conversational search Log analysis Process mining 

Notes

Acknowledgements

The work of S. Vakulenko and C. Di Ciccio has received funding from the EU H2020 program under MSCA-RISE agreement 645751 (RISE_BPM) and the Austrian Research Promotion Agency (FFG) under grant 861213 (CitySPIN). S. Vakulenko was also supported by project 855407 “Open Data for Local Communities” (CommuniData) of the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) under the program “ICT of the Future.” M. de Rijke was supported by Ahold Delhaize, the Association of Universities in the Netherlands (VSNU), and the Innovation Center for Artificial Intelligence (ICAI).

All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

References

  1. 1.
    Adriansyah, A., van Dongen, B.F., van der Aalst, W.M.: Conformance checking using cost-based fitness analysis. In: 15th International Enterprise Distributed Object Computing Conference, pp. 55–64. IEEE Computer Society (2011)Google Scholar
  2. 2.
    Austin, J.L.: How to do Things with Words. Oxford Paperbacks, Oxford University Press, Oxford (1976)Google Scholar
  3. 3.
    Belkin, N.J., Cool, C., Stein, A., Thiel, U.: Cases, scripts, and information-seeking strategies: on the design of interactive information retrieval systems. Expert Syst. Appl. 9(3), 379–395 (1995)CrossRefGoogle Scholar
  4. 4.
    Cohen, W.W., Carvalho, V.R., Mitchell, T.M.: Learning to classify email into “speech acts”. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 309–316 (2004)Google Scholar
  5. 5.
    Core, M.: Coding dialogs with the DAMSL annotation scheme. In: Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pp. 1–8 (1997)Google Scholar
  6. 6.
    Di Ciccio, C., Mecella, M.: Mining artful processes from knowledge workers’ emails. IEEE Internet Comput. 17(5), 10–20 (2013)CrossRefGoogle Scholar
  7. 7.
    Di Ciccio, C., Mecella, M.: A two-step fast algorithm for the automated discovery of declarative workflows. In: Symposium on Computational Intelligence and Data Mining, pp. 135–142. IEEE (2013)Google Scholar
  8. 8.
    Di Ciccio, C., Mecella, M.: On the discovery of declarative control flows for artful processes. ACM Trans. Manag. Inf. Syst. 5(4), 24:1–24:37 (2015)CrossRefGoogle Scholar
  9. 9.
    Goffman, E.: Erving Goffman: Exploring the Interaction Order. Polity Press, Cambridge (1988)Google Scholar
  10. 10.
    Jeong, M., Lin, C., Lee, G.G.: Semi-supervised speech act recognition in emails and forums. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 1250–1259 (2009)Google Scholar
  11. 11.
    Jo, Y., Yoder, M., Jang, H., Rosé, C.P.: Modeling dialogue acts with content word filtering and speaker preferences. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2179–2189 (2017)Google Scholar
  12. 12.
    Leemans, M., van der Aalst, W.M.P.: Discovery of frequent episodes in event logs. In: Ceravolo, P., Russo, B., Accorsi, R. (eds.) SIMPDA 2014. LNBIP, vol. 237, pp. 1–31. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-27243-6_1CrossRefGoogle Scholar
  13. 13.
    Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Process and deviation exploration with inductive visual miner. In: Proceedings of the BPM Demo Sessions 2014 Co-located with the 12th International Conference on Business Process Management, p. 46 (2014)Google Scholar
  14. 14.
    Li, Z., Kiseleva, J., de Rijke, M.: Dialogue generation: from imitation learning to inverse reinforcement learning. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, January 2019Google Scholar
  15. 15.
    Radlinski, F., Craswell, N.: A theoretical framework for conversational search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pp. 117–126. ACM (2017)Google Scholar
  16. 16.
    Richetti, P.H.P., de A.R. Gonçalves, J.C., Baião, F.A., Santoro, F.M.: Analysis of knowledge-intensive processes focused on the communication perspective. In: Carmona, J., Engels, G., Kumar, A. (eds.) BPM 2017. LNCS, vol. 10445, pp. 269–285. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-65000-5_16CrossRefGoogle Scholar
  17. 17.
    Schegloff, E.A.: Sequencing in conversational openings. Am. Anthropologist 70(6), 1075–1095 (1968)CrossRefGoogle Scholar
  18. 18.
    Schiffrin, D.: Approaches to Discourse: Language as Social Interaction. Blackwell Textbooks in Linguistics. Wiley, Hoboken (1994)Google Scholar
  19. 19.
    Searle, J.R.: Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge (1969)CrossRefGoogle Scholar
  20. 20.
    Searle, J.R.: A classification of illocutionary acts. Lang. Soc. 5(1), 1–23 (1976)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Sitter, S., Stein, A.: Modeling the illocutionary aspects of information-seeking dialogues. Inf. Process. Manage. 28(2), 165–180 (1992)CrossRefGoogle Scholar
  22. 22.
    Stolcke, A., et al.: Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput. Linguist. 26(3), 339–373 (2000)CrossRefGoogle Scholar
  23. 23.
    Trippas, J.R., Spina, D., Cavedon, L., Sanderson, M.: How do people interact in conversational speech-only search tasks: a preliminary analysis. In: Proceedings of the 2017 ACM on Conference on Human Information Interaction and Retrieval, pp. 325–328. ACM (2017)Google Scholar
  24. 24.
    Trippas, J.R., Spina, D., Cavedon, L., Joho, H., Sanderson, M.: Informing the design of spoken conversational search: perspective paper. In: Proceedings of the 2018 Conference on Human Information Interaction and Retrieval, pp. 32–41. ACM (2018)Google Scholar
  25. 25.
    Vakulenko, S., de Rijke, M., Cochez, M., Savenkov, V., Polleres, A.: Measuring semantic coherence of a conversation. In: Vrandečić, D., et al. (eds.) ISWC 2018. LNCS, vol. 11136, pp. 634–651. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00671-6_37CrossRefGoogle Scholar
  26. 26.
    van der Aalst, W.M., Adriansyah, A., van Dongen, B.F.: Replaying history on process models for conformance checking and performance analysis. Wiley Interdisciplinary Reviews: Data Min. Knowl. Discov. 2(2), 182–192 (2012)Google Scholar
  27. 27.
    van der Aalst, W.M.P.: The application of petri nets to workflow management. J. Circuits Syst. Comput. 8(1), 21–66 (1998).  https://doi.org/10.1142/S0218126698000043CrossRefGoogle Scholar
  28. 28.
    van der Aalst, W.M.P.: Process Mining: Data Science in Action, 2nd edn. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-662-49851-4CrossRefGoogle Scholar
  29. 29.
    Wang, G.A., Wang, H.J., Li, J., Abrahams, A.S., Fan, W.: An analytical framework for understanding knowledge-sharing processes in online Q&A communities. ACM Trans. Manag. Inf. Syst. 5(4), 18 (2015)Google Scholar
  30. 30.
    Williams, J., Raux, A., Henderson, M.: The dialog state tracking challenge series: a review. Dialogue Discourse 7(3), 4–33 (2016)Google Scholar
  31. 31.
    Winograd, T., Flores, F.: Understanding Computers and Cognition: A New Foundation for Design. Intellect Books, Bristol (1986)zbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Svitlana Vakulenko
    • 1
    Email author
  • Kate Revoredo
    • 2
  • Claudio Di Ciccio
    • 1
  • Maarten de Rijke
    • 3
  1. 1.Vienna University of Economics and BusinessViennaAustria
  2. 2.Graduate Program in InformaticsFederal University of Rio de JaneiroRio de JaneiroBrazil
  3. 3.University of AmsterdamAmsterdamThe Netherlands

Personalised recommendations