Towards Conversation Envisioning for Cognitive Robots

  • Maryam Sadat MirzaeiEmail author
  • Qiang Zhang
  • Stef van der Struijk
  • Toyoaki Nishida
Part of the Springer Series in Cognitive and Neural Systems book series (SSCNS, volume 12)


This chapter proposes a new conceptual framework for visualizing conversational interactions to build a common ground for productive human-robot interaction. We address conversation envisioning as a process of building and maintaining common ground by incorporating new information into the shared information structure. We discuss how we can build a common ground for human robot interaction by using a synthetic evidential study (SES) as an underlying methodology. We review the main ideas in SES, which integrates role playing game, agent play, and in-situ group discussion to uncover tacit thoughts and interaction of participants. We then introduce the idea of conversation envisioning in which we employ virtual reality for graphic recording (VRGR) to extend the framework of SES so that both observers and participants can visualize their thoughts in conversation. We also consider emotion and its relation to common ground to better analyze the conversation. We focus on a bargaining scenario, to illustrate how the framework of SES could be employed to help people realize subtle socio-ethical issues in social activities to design a robot that can engage in social interactions with people as well as other robots as well.


Conversational informatics Common ground Synthetic evidential study Conversation envisioning Cross-cultural negotiation 


  1. Azzam T, Evergreen S, Germuth AA, Kistler SJ (2013) Data visualization and evaluation. N Dir Eval 2013(139):7–32CrossRefGoogle Scholar
  2. Barrett LF (2014) The conceptual act theory: a précis. Emot Rev 6(4):292–297. [Online] Available: CrossRefGoogle Scholar
  3. Barrett LF (2017) How emotions are made: the secret life of the brain. Houghton Mifflin Harcourt, Boston/New YorkGoogle Scholar
  4. Clark HH (1996) Using language. Cambridge University Press, Cambridge/New YorkCrossRefGoogle Scholar
  5. Couper-Kuhlen E (2015) What does grammar tell us about action? Pragmat Q Publ Int Pragmat Assoc (IPrA) 24(3):623–647Google Scholar
  6. Ekman P (2016) What scientists who study emotion agree about. Perspect Psychol Sci 11(1):31–34CrossRefGoogle Scholar
  7. Ekman P, Cordaro D (2011) What is meant by calling emotions basic. Emot Rev 3(4):364–370CrossRefGoogle Scholar
  8. Emmorey K, Tversky B, Taylor HA (2000) Using space to describe space: perspective in speech, sign, and gesture. Spat Cogn Comput 2(3):157–180CrossRefGoogle Scholar
  9. Goffman E (1967) Interaction ritual: essays on face-to-face interaction. Aldine Transaction, OxfordGoogle Scholar
  10. Güth W (1995) On ultimatum bargaining experiments – a personal review. J Econ Behav Organ 27(3):329–344CrossRefGoogle Scholar
  11. Harsanyi JC (2016) Bargaining. Palgrave Macmillan, London, pp 1–10Google Scholar
  12. Hofstede GJ, Jonker CM, Verwaart T (2008) An agent model for the influence of culture on bargaining. In: Proceedings of the 1st International Working Conference on Human Factors and Computational Models in Negotiation, Delft, The Netherlands, 08–09 Dec 2008. ACM, pp 39–46.
  13. Miller GA (1995) Wordnet: a lexical database for English. Commun ACM 38(11):39–41CrossRefGoogle Scholar
  14. Moors A, Ellsworth PC, Scherer KR, Frijda NH (2013) Appraisal theories of emotion: state of the art and future development. Emot Rev 5(2):119–124CrossRefGoogle Scholar
  15. Nakano YI, Reinstein G, Stocky T, Cassell J (2003) Towards a model of face-to-face grounding. In: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, vol 1. Association for Computational Linguistics, pp 553–561Google Scholar
  16. Nishida T, Nakazawa A, Ohmoto Y, Mohammad Y (2014) Conversational informatics: a data-intensive approach with emphasis on nonverbal communication. Springer, TokyoGoogle Scholar
  17. Nishida T, Nakazawa A, Ohmoto Y, Nitschke C, Mohammad Y, Thovuttikul S, Lala D, Abe M, Ookaki T (2015) Synthetic evidential study as primordial soup of conversation. In: International Workshop on Databases in Networked Information Systems. Springer, pp 74–83Google Scholar
  18. Nishida T, Abe M, Ookaki T, Lala D, Thovuttikul S, Song H, Mohammad Y, Nitschke C, Ohmoto Y, Nakazawa A et al (2015) Synthetic evidential study as augmented collective thought process–preliminary report. In: Asian Conference on Intelligent Information and Database Systems, Bali, Indonesia, 23–25 Mar 2015. Springer, pp 13–22. Google Scholar
  19. Nouri E, Georgila K, Traum D (2017) Culture-specific models of negotiation for virtual characters: multi-attribute decision-making based on culture-specific values. AI Soc 32(1):51–63CrossRefGoogle Scholar
  20. Ohmoto Y, Kataoka M, Nishida T (2014) The effect of convergent interaction using subjective opinions in the decision-making process. In: Proceedings of the Cognitive Science Society, vol 36, no. 36Google Scholar
  21. Ohmoto Y, Suyama T, Nishida T (2016a) A method to alternate the estimation of global purposes and local objectives to induce and maintain the intentional stance. In: Proceedings of the Fourth International Conference on Human Agent Interaction, Biopolis, Singapore, 04–07 Oct. ACM, pp 379–385Google Scholar
  22. Ohmoto Y, Ookaki T, Nishida T (2016b) A support system to accumulate interpretations of multiple story timelines. Procedia Comput Sci 96:607–616CrossRefGoogle Scholar
  23. Ookaki T, Abe M, Yoshino M, Ohmoto Y, Nishida T (2015) Synthetic evidential study for deepening inside their heart. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Seoul, South Korea, 10–12 June 2015. Springer, pp 161–170. Google Scholar
  24. Posner J, Russell JA, Peterson BS (2005) The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol 17(3):715–734CrossRefGoogle Scholar
  25. Rubin JZ, Brown BR (1975) The social psychology of bargaining and negotiation. Academic, San DiegoGoogle Scholar
  26. Thompson L (2000) The Mind and heart of the negotiator. Prentice Hall Press, Upper Saddle RiverGoogle Scholar
  27. Tomasello M (2010) Origins of human communication. MIT Press, Cambridge/LondonGoogle Scholar
  28. Traum DR (1994) A computational theory of grounding in natural language conversation. Rochester Univ NY Dept of Computer Science, Technical ReportGoogle Scholar
  29. Tversky B (2011) Visualizing thought. Top Cogn Sci 3(3):499–535CrossRefGoogle Scholar
  30. Wellman HM (2014) Making minds: how theory of mind develops. Oxford University Press, New YorkCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG (outside the USA) 2019

Authors and Affiliations

  • Maryam Sadat Mirzaei
    • 1
    Email author
  • Qiang Zhang
    • 2
  • Stef van der Struijk
    • 2
  • Toyoaki Nishida
    • 2
  1. 1.RIKEN AIPKyotoJapan
  2. 2.Kyoto UniversityKyotoJapan

Personalised recommendations