Abstract
Ambient Intelligence aims at promoting an effective, natural and personalized interaction with the environment services. In order to provide the most appropriate answer to the user requests, an Ambient Intelligence system should model the user by considering not only the cognitive ingredients of his mental state, but also extra-rational factors such as affect, engagement, attitude, and so on. This paper describes a study aimed at building a multimodal framework for recognizing the social response of users during interaction with embodied agents in the context of ambient intelligence. In particular, we describe how we extended a model for recognizing the social attitude in text-based dialogs by adding two additional knowledge sources: speech and gestures. Results of the study show that these additional knowledge sources may help in improving the recognition of the users’ attitude during interaction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Riva, G., Vatalaro, F., Davide, F., Alcaniz, M. (eds.): Ambient Intelligence: the Evolution of Technology, Communication and Cognition Towards the Future of Human-Computer Interaction Emerging Communication (2005)
López-Cózar, R., Callejas, Z.: Multimodal Dialogue for Ambient Intelligence and Smart Environments. In: Nakashima, H., Aghajan, H., Augusto, J.C. (eds.) Intelligence, p. 93808. Springer US (2010)
Nijholt, A., de Ruyter, B., Heylen, D., Privender, S.: Social Interfaces for Ambient Intelligence Environments. In: Aarts, E., Encarnaçao, J. (eds.) True Visions: The Emergence of Ambient Intelligence, ch. 14, pp. 275A–289A. Springer, New York (2006)
Ortiz, A., del Puy Carretero, M., Oyarzun, D., Yanguas, J.J., Buiza, C., Gonzalez, M.F., Etxeberria, I.: Elderly Users in Ambient Intelligence: Does an Avatar Improve the Interaction? In: Stephanidis, C., Pieper, M. (eds.) ERCIM UI4ALL Ws 2006. LNCS, vol. 4397, pp. 99–114. Springer, Heidelberg (2007)
Cesta, A., Cortellessa, G., Giuliani, M.V., Iocchi, L., Leone, G.R., Nardi, D., Pecora, F., Rasconi, R., Scopelliti, M., Tiberio, L.: The RoboCare Assistive Home Robot: Environment, Features and Evaluation, The RoboCare Technical Report, RC-TR-0906-6 (2006)
van Breemen, J.N.: iCat: a generic platform for studying personal robot applications. Paper Presented at the IEEE SMC, Den Haag (2004)
Reeves, B., Nass, C.: The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Cambridge (1996)
Bickmore, T., Cassell, J.: Social Dialogue with Embodied Conversational Agents. In: van Kuppevelt, J., Dybkjaer, L., Bernsen, N. (eds.) Advances in Natural, Multimodal Dialogue Systems, pp. 1–32. Kluwer Academic, New York (2005)
Cassell, J., Prevost, S., Sullivan, J., Churchill, E.: Embodied Conversational Agents. MIT Press, Cambridge (2000)
Paiva, A. (ed.): Empathic Agents. Workshop in conjunction with AAMAS 2004 (2005)
Jensen, F.V.: Bayesian Networks and Decision Graphs. Springer (2001)
Whittaker, S., Walker, M., Moore, J.: Fish or Fowl: A Wizard of Oz evaluation of dialogue strategies in the restaurant domain. In: Language Resources and Evaluation Conference (2002)
Oviatt, S., Adams, B.: Designing and Evaluating Conversational Interfaces With Animated Characters. In: Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.) Embodied Conversational Agents. The MIT Press (2000)
Novielli, N., de Rosis, F., Mazzotta, I.: User Attitude Towards an Embodied Conversational Agent: Effects of the Interaction Mode. Journal of Pragmatics 42(9), 2385–2397 (2010) ISSN: 0378-2166, doi:10.1016/j.pragma.2009.12.016
Andersen, P.A., Guerrero, L.K.: Handbook of Communication and Emotions. Research, theory, applications and contexts. Academic Press (1998)
Polhemus, L., Shih, L.-F., Swan, K.: Virtual interactivity: the representation of social presence in an on line discussion. In: Annual Meeting of the American Educational Research Association (2001)
Swan, K.: Immediacy, social presence and asynchronous discussion. In: Bourne, J., Moore, J.C. (eds.) Elements of Quality Online Education, vol. 3. Sloan Center For Online Education, Nedham (2002)
KinectDTW, http://kinectdtw.codeplex.com/
Markopoulos, P.: Designing ubiquitous computer human interaction: the case of the connected family. In: Isomaki, H., Pirhonen, A., Roast, C., Saariluoma, P. (eds.) Future Interaction Design. Springer (2004)
Norman, D.: The Invisible Computer. MIT Press, Cambridge (1998)
Pentland, A.: Perceptual intelligence. Communications of the ACM 43(3), 35–44 (2000)
Reilly, W.S.N.: Believable Social and Emotional Agents. PhD thesis (1996)
Brahnam, S., De Angeli, A.: Special issue on the abuse and misuse of social agents. Interacting with Computers 20(3), 287–291 (2008)
Bickmore, T.: Relational Agents: Effecting Change through Human-Computer Relationships. PhD Thesis, Media Arts & Sciences, Massachusetts Institute of Technology (2003)
Hoorn, J.F., Konijn, E.A.: Perceiving and Experiencing Fictitional Characters: An integrative account. Japanese Psychological Research 45, 4 (2003)
Cassell, J., Bickmore, T.: Negotiated collusion: modelling social language and its relationship effects in intelligent agents. User Modelling and User-Adapted Interaction 13, 1–2 (2003)
Bailenson, J.N., Aharoni, E., Beall, A.C., Guadagno, R.E., Dimov, A., Blascovich, J.: Comparing behavioral and self-report measures of embodied agents’ social presence in immersive virtual environments. In: Proceedings of 7th Annual International Workshop on PRESENCE, pp. 216–223 (2004)
Bailenson, J.N., Swinth, K.R., Hoyt, C.L., Persky, S., Dimov, A., Blascovich, J.: The independent and interactive effects of embodied agents appearance and behavior on self-report, cognitive and behavioral markers of copresence in Immersive Virtual Environments. Presence 14(4), 379–393 (2005)
Cramer, H., Kemper, N., Amin, A., Wielinga, B., Evers, V.: Give me a hug: the effects of touch and autonomy on people’s responses to embodied social agents. Comp. Anim. Virtual Worlds 20, 437–445 (2009), doi:10.1002/cav.317
Mozziconacci, S., Hermes, D.J.: Role of intonation patterns in conveying emotion in speech. In: Proceedings, International Conference of Phonetic Sciences, San Francisco (1999)
Bosma, W.E., André, E.: Exploiting Emotions to Disambiguate Dialogue Acts. In: Nunes, N.J., Rich, C. (eds.) Proc. 2004 Conference on Intelligent User Interfaces, Funchal, Portugal, January 13, pp. 85–92 (2004)
Litman, D., Forbes, K., Silliman, S.: Towards emotion prediction in spoken tutoring dialogues. In: Proceedings of HLT/NAACL (2003)
De Carolis, B., Cozzolongo, G.: Interpretation of User’s Feedback in Human-Robot Interaction. Journal of Physical Agents 3(2), 47–58 (2009) ISSN: 1888-0258
Sundberg, J., Patel, S., Björkner, E., Scherer, K.R.: Interdependencies among voice source parameters in emotional speech. IEEE Transactions on Affective Computing 2(3) (2011)
Vogt, T., André, E., Bee, N.: EmoVoice — A Framework for Online Recognition of Emotions from Voice. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 188–199. Springer, Heidelberg (2008)
Allan, Pease, B.: The Definitive Book of Body Language. Bantam Books (2006)
Feldman, R.S.: Applications of Nonverbal Behavioral Theories and Research. Lawrence Erlbaum Associates, Inc., New Jersey (1992)
Knapp, M., Hall, J.: Nonverbal communication in human interaction. Holt, Rinehart & Winsten, Inc., Orlando (1992)
Malandro, L.A.: Non-verbal Communication, 2nd edn. Random House, New York (1983,1989)
Efron, D.: Gesture, Race and Culture. Mouton and Co. (1972)
Rehm, M., Nakano, Y., André, E., Nishida, T.: Culture-Specific First Meeting Encounters between Virtual Agents. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 223–236. Springer, Heidelberg (2008)
Prendinger, H., Mori, J., Ishizuka, M.: Recognizing, Modeling, and Responding to Users’ Affective States. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 60–69. Springer, Heidelberg (2005)
Conati, C.: Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16, 555–575 (2002)
Sabourin, J., Mott, B., Lester, J.: Computational Models of Affect and Empathy for Pedagogical Virtual Agents. Standards in Emotion Modeling, Lorentz Center International Center for Workshops in the Sciences (2011)
Caridakis, G., Karpouzis, K., Wallace, M., Kessous, L., Amir, N.: Multimodal user’s affective state analysis in naturalistic interaction. Journal on Multimodal User Interfaces 3(1-2), 49–66 (2009)
Li, X.: Integrating User Affective State Assessment in Enhancing HCI: Review and Proposition. The Open Cybernetics and Systemics Journal 2, 192–205 (2008)
Boersma, P., Weenink, D.: Praat: doing phonetics by computer (version 4.5.15) [computer program] (2007), http://www.praat.org/ (retrieved February 24, 2007)
Martin, B.: Instance-Based learning: Nearest Neighbor With Generalization, Master Thesis, University of Waikato, Hamilton, New Zealand (1995)
Kistler, F., Endrass, B., Damian, I., Dang, C.T., André, E.: Natural interaction with culturally adaptive virtual characters. Journal on Multimodal User Interfaces
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
De Carolis, B., Ferilli, S., Novielli, N. (2012). Recognizing the User Social Attitude in Multimodal Interaction in Smart Environments. In: Paternò, F., de Ruyter, B., Markopoulos, P., Santoro, C., van Loenen, E., Luyten, K. (eds) Ambient Intelligence. AmI 2012. Lecture Notes in Computer Science, vol 7683. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34898-3_16
Download citation
DOI: https://doi.org/10.1007/978-3-642-34898-3_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34897-6
Online ISBN: 978-3-642-34898-3
eBook Packages: Computer ScienceComputer Science (R0)
