Sentic Avatar: Multimodal Affective Conversational Agent with Common Sense

  • Erik Cambria
  • Isabelle Hupont
  • Amir Hussain
  • Eva Cerezo
  • Sandra Baldassarri
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6456)


The capability of perceiving and expressing emotions through different modalities is a key issue for the enhancement of human-computer interaction. In this paper we present a novel architecture for the development of intelligent multimodal affective interfaces. It is based on the integration of Sentic Computing, a new opinion mining and sentiment analysis paradigm based on AI and Semantic Web techniques, with a facial emotional classifier and Maxine, a powerful multimodal animation engine for managing virtual agents and 3D scenarios. One of the main distinguishing features of the system is that it does not simply perform emotional classification in terms of a set of discrete emotional labels but it operates in a continuous 2D emotional space, enabling the integration of the different affective extraction modules in a simple and scalable way.


AI Sentic Computing NLP Facial Expression Analysis Sentiment Analysis Multimodal Affective HCI Conversational Agents 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vesterinen, E.: Affective Computing. In: Digital Media Research Seminar, Helsinki (2001)Google Scholar
  2. 2.
    Pantic, M.: Affective Computing. Encyclopedia of Multimedia Technology and Networking. Hershy 1, 8–14, Idea Group Reference (2005)CrossRefGoogle Scholar
  3. 3.
    Gunes, M., Gunes, H., Piccardi, M., Pantic, M.: From the Lab to the Real World: Affect Recognition Using Multiple Cues and Modalities. In: Affective Computing: Focus on Emotion Expression, Synthesis and Recognition, pp. 185–218 (2008)Google Scholar
  4. 4.
    Riseberg, J., Klein, J., Fernandez, R., Picard, R.: Frustrating the User on Purpose: Using Biosignals in a Pilot Study to Detect the User’s Emotional State. In: CHI, Los Angeles (1998)Google Scholar
  5. 5.
    Ambady, N., Rosenthal, R.: Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: a Meta-Analysis. Psychological Bulletin 11(2), 256–274 (1992)CrossRefGoogle Scholar
  6. 6.
    Camurri, A., Mazzarino, B., Volpe, G.: Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library. In: Gesture Workshop, Genova (2003)Google Scholar
  7. 7.
    Gunes, H., Piccardi. M.: Bi-Modal Emotion Recognition from Expressive Face and Body Gestures. Network and Computer Applications 30(4), 1334–1345 (2007)CrossRefGoogle Scholar
  8. 8.
    Karpouzis, K., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A., Malatesta, L., Kollias, S.: Modeling Naturalistic Affective States Via Facial, Vocal and Bodily Expressions Recognition. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds.) ICMI/IJCAI Workshops 2007. LNCS (LNAI), vol. 4451, pp. 92–116. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  9. 9.
    Pun, T., Alecu, T., Chanel, G., Kronegg, J., Voloshynovskiy, S.: Brain-Computer Interaction Research at the Computer Vision and Multimedia Laboratory. IEEE Trans. on Neural Systems and Rehabilitation Engineering 14(2), 210–213 (2006)CrossRefGoogle Scholar
  10. 10.
    Burleson, W., Picard, R., Perlin, K., Lippincott, J.: A Platform for Affective Agent Research. In: International Conference on Autonomous Agents and Multiagent Systems, New York (2004)Google Scholar
  11. 11.
    Petridis, S., Pantic, M.: Audiovisual Discrimination between Laughter and Speech. In: ICASSP, Las Vegas (2008)Google Scholar
  12. 12.
    Valstar, M., Gunes, H., Pantic, M.: How to Distinguish Posed from Spontaneous Smiles Using Geometric Features. In: ICMI, Nagoya (2007)Google Scholar
  13. 13.
    Truong, K., Van Leeuwen, D.: Automatic Discrimination Between Laughter and Speech. Speech Communication 49, 144–158 (2007)CrossRefGoogle Scholar
  14. 14.
    Matos, S., Birring, S., Pavord, I., Evans, D.: Detection of Cough Signals in Continuous Audio Recordings Using HMM. IEEE Trans. on Biomedical Engineering 53(6), 1078–1083 (2006)CrossRefGoogle Scholar
  15. 15.
    Pal, P., Iyer, A., Yantorno, R.: Emotion Detection from Infant Facial Expressions and Cries. In: Intl Conf. Acoustics, Speech and Signal Processing (2006)Google Scholar
  16. 16.
    Jong-Tae, J., Sang-Wook, S., Kwang-Eun, K., Kwee-Bo, S.: Emotion Recognition Method Based on Multimodal Sensor Fusion Algorithm. In: ISIS, Sokcho-City (2007)Google Scholar
  17. 17.
    Shan, C., Gong, S., McOwan, P.: Beyond Facial Expressions: Learning Human Emotion from Body Gestures. In: BMVC, Warwick (2007)Google Scholar
  18. 18.
    Baldassarri, S., Cerezo, E., Seron, F.: Maxine: a Platform for Embodied Animated Agents. Computers and Graphics 32(4), 430–437 (2008)CrossRefGoogle Scholar
  19. 19.
    Cambria, E., Hussain, A., Havasi, C., Eckl, C.: Sentic Computing: Exploitation of Common Sense for the Development of Emotion-Sensitive Systems. In: Esposito, A., Campbell, N., Vogel, C., Hussain, A., Nijholt, A. (eds.) COST 2102. LNCS, vol. 5967, pp. 153–161. Springer, Heidelberg (2010)Google Scholar
  20. 20.
    Cambria, E., Hussain, A., Havasi, C., Eckl, C.: Common Sense Computing: From the Society of Mind to Digital Intuition and Beyond. In: Fierrez, J., Ortega-Garcia, J., Esposito, A., Drygajlo, A., Faundez-Zanuy, M. (eds.) BioID MultiComm 2009. LNCS, vol. 5707, pp. 252–259. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  21. 21.
    Cambria, E., Grassi, M., Hussain, A., Havasi, C.: Sentic Computing for Social Media Marketing. In: Multimedia Tools and Applications. Springer, Heidelberg (to appear, 2010)Google Scholar
  22. 22.
    Elliott, C.: The Affective Reasoner: A Process Model of Emotions in a Multi-Agent System. The Institute for the Learning Sciences, Technical Report No. 32 (1992)Google Scholar
  23. 23.
    Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, New York (1988)CrossRefGoogle Scholar
  24. 24.
    Wiebe, J., Wilson, T., Claire, C.: Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation 39(2), 165–210 (2005)CrossRefGoogle Scholar
  25. 25.
    Wilson, T., Wiebe, J., Hoffmann, P.: Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. In: HLT-EMNLP, Vancouver (2005)Google Scholar
  26. 26.
    Somasundaran, S., Wiebe, J., Ruppenhofer, J.: Discourse Level Opinion Interpretation. In: COLING, Manchester (2008)Google Scholar
  27. 27.
    Hu, M., Liu, B.: Mining Opinion Features in Customer Reviews. In: AAAI, San Jose (2004)Google Scholar
  28. 28.
    Pang, B., Lee, L.: Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. In: ACL, Ann Arbor (2005)Google Scholar
  29. 29.
    Abbasi, A., Chen, H., Salem, A.: Sentiment Analysis in Multiple Languages: Feature Selection for Opinion Classification in Web Forums. ACM Transactions on Information Systems 26(3), 1–34 (2008)CrossRefGoogle Scholar
  30. 30.
    Cambria, E., Hussain, A., Havasi, C., Eckl, C.: AffectiveSpace: Blending Common Sense and Affective Knowledge to Perform Emotive Reasoning. In: WOMSA at CAEPIA, Seville (2009)Google Scholar
  31. 31.
    Havasi, C., Speer, R., Alonso, J.: ConceptNet 3: a Flexible, Multilingual Semantic Network for Common Sense Knowledge. In: RANLP, Borovets (2007)Google Scholar
  32. 32.
    Strapparava, C., Valitutti, A.: WordNet-Affect: an Affective Extension of WordNet. In: LREC, Lisbon (2004)Google Scholar
  33. 33.
    Havasi, C., Speer, R., Pustejovsky, J., Lieberman, H.: Digital Intuition: Applying Common Sense Using Dimensionality Reduction. IEEE Intelligent Systems 24(4), 24–35 (2009)CrossRefGoogle Scholar
  34. 34.
    Wall, M., Rechtsteiner, A., Rocha, L.: Singular Value Decomposition and Principal Component Analysis. In: Berrar, D., et al. (eds.) A Practical Approach to Microarray Data Analysis, pp. 91–109. Kluwer, Norwell (2003)CrossRefGoogle Scholar
  35. 35.
    Eckart, C., Young, G.: The Approximation of One Matrix by Another of Lower Rank. Psychometrika 1(3), 211–218 (1936)CrossRefzbMATHGoogle Scholar
  36. 36.
    Plutchik, R.: The Nature of Emotions. American Scientist 89(4), 344–350 (2001)CrossRefGoogle Scholar
  37. 37.
    Minsky, M.: The Emotion Machine. Simon and Schuster, New York (2006)Google Scholar
  38. 38.
    Loquendo Audio Speech Recognition,
  39. 39.
    Ekman, P., Dalgleish, T., Power, M.: Handbook of Cognition and Emotion. Wiley, Chihester (1999)Google Scholar
  40. 40.
    Cerezo, E., Hupont, I., Manresa, C., Varona, J., Baldassarri, S., Perales, F., Seron, F.: Real-Time Facial Expression Recognition for Natural Interaction. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 40–47. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  41. 41.
    Witten, I., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  42. 42.
    Wallhoff, F.: Facial Expressions and Emotion Database. Technische Universitat Munchen (2006)Google Scholar
  43. 43.
    Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-Based Database for Facial Expression Analysis. In: ICME, Singapore (2005)Google Scholar
  44. 44.
    Siegel, S., Castellan, N.: Nonparametric Statistics for the Social Siences. McGraw-Hill, New York (1988)Google Scholar
  45. 45.
    Whissell, C.: The Dictionary of Affect in Language. Emotion: Theory, Research and Experience, The Measurement of Emotions 4, 113–131 (1989)Google Scholar
  46. 46.
    Plutchik, R.: Emotion: a Psychoevolutionary Synthesis. Harper and Row, New York (1980)Google Scholar
  47. 47.
    Kumar, P., Yildirim, E.: Minimum-Volume Enclosing Ellipsoids and Core Sets. Journal of Optimization Theory and Applications 126, 1–21 (2005)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Erik Cambria
    • 1
  • Isabelle Hupont
    • 2
  • Amir Hussain
    • 1
  • Eva Cerezo
    • 3
  • Sandra Baldassarri
    • 3
  1. 1.University of StirlingStirlingUK
  2. 2.Aragon Institute of TechnologyZaragozaSpain
  3. 3.University of ZaragozaZaragozaSpain

Personalised recommendations