Skip to main content
SpringerLink
Log in
Book cover

Intelligent Interactive Multimedia Systems and Services in Practice pp 7–34Cite as

On the Use of Multi-attribute Decision Making for Combining Audio-Lingual and Visual-Facial Modalities in Emotion Recognition

  • Chapter
  • First Online:

  • 434 Accesses

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 36))

Abstract

In this chapter, we present and discuss a novel approach that we have developed for the integration of audio-lingual and visual-facial modalities in a bi-modal user interface for affect recognition. Even though researchers acknowledge that two modalities can provide information that is complementary to each other with respect to affect recognition, satisfactory progress has not yet been achieved towards the integration of the two modalities. In our research reported herein, we approach the combination of the two modalities from the perspective of a human observer by employing a multi-criteria decision making theory for dynamic affect recognition of computer users. Our approach includes the specification of the strengths and weaknesses of each modality with respect to affect recognition concerning the 6 basic emotion states, namely happiness, sadness, surprise, anger and disgust, as well as the emotionless state which we refer to as neutral. We present two empirical studies that we have conducted involving human users and human observers concerning the recognition of emotions from audio-lingual and visual-facial modalities. Based on the results of the empirical studies, we assign weights to criteria for the application of a multi-criteria decision making theory. Additionally, the results of the empirical studies provide information that may be used by other researchers in the field of affect recognition and is currently unavailable in the relevant literature.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Leon, E., Clarke, G., Callaghan, V., Sepulveda, F.: A user-independent real-time emotion recognition system for software agents in domestic environments. Eng. Appl. Artif. Intell. 20, 337–345 (2007)

    Article  Google Scholar 

  2. Goleman, D.: Emotional Intelligence. Bantam Books, New York (1995)

    Google Scholar 

  3. Picard, R.W.: Affective computing: challenges. Int. J. Hum. Comput. Stud. 59, 55–64 (2003)

    Article  Google Scholar 

  4. Stathopoulou, I.-O., Tsihrintzis, G.A.: Visual affect recognition. Front. Artificial Intell. Appl. 214, 1–267 (2010)

    Google Scholar 

  5. Morrison, D., Wang, R., De Silva, L.C.: Ensemble methods for spoken emotion recognition in call-centres. Speech Commun. 49, 98–112 (2007)

    Article  Google Scholar 

  6. Pantic, M., Rothkrantz, L.J.M.: Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91, 1370–1390 (2003)

    Article  Google Scholar 

  7. Chen, L.S., Huang, T.S., Miyasato, T., Nakatsu, R.: Multimodal human emotion/expression recognition. In: Proceedings of the 3rd International Conference on Face & Gesture Recognition: IEEE Computer Society (1998)

    Google Scholar 

  8. De Silva, L., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multimodal information. In: IEEE International Conference on Information, Communications and Signal Processing (ICICS’97), pp. 397–401 (1997)

    Google Scholar 

  9. Huang, T.S., Chen, L.S., Tao, H.: Bimodal emotion recognition by man and machine. In: ATR Workshop on Virtual Communication Environments Kyoto, Japan (1998)

    Google Scholar 

  10. Oviatt, S.: User-modeling and evaluation of multimodal interfaces. In: Proceedings of the IEEE, pp. 1457–1468 (2003)

    Google Scholar 

  11. Zeng, Z., Tu, J., Liu, M., Huang, T., Pianfetti, B., Roth, D., Levinson, S.: Audio-visual affect recognition. IEEE Trans. Multimed. 9, 424–428 (2007)

    Article  Google Scholar 

  12. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proceedings of the 6th International Conference On Multimodal Interfaces, State College, PA, USA, ACM (2004)

    Google Scholar 

  13. Liao, W., Zhang, W., Zhu, Z., Ji, Q., Gray, W.D.: Toward a decision-theoretic framework for affect recognition and user assistance. Int. J. Hum.-Comput. Stud. 64, 847–873 (2006)

    Article  Google Scholar 

  14. Alepis, E., Stathopoulou, I.-O., Virvou, M., Tsihrintzis, G., Kabassi, K.: Audio-lingual and visual-facial emotion recognition: Towards a bi-modal interaction system. In: Proceedings of International Conference on Tools with Artificial Intelligence, ICTAI, 2, art. no. 5670096, pp. 274–281 (2010)

    Google Scholar 

  15. Fishburn, P.C.: Additive utilities with incomplete product set: applications to priorities and assignments. Oper. Res. 15, 537–542 (1967)

    Google Scholar 

  16. Nasoz, F., Lisetti, C.L.: MAUI avatars: mirroring the user’s sensed emotions via expressive multi-ethnic facial avatars. J. Vis. Lang. Comput. 17, 430–444 (2006)

    Article  Google Scholar 

  17. Virvou, M., Kabassi, K.: Adapting the human plausible reasoning theory to a graphical user interface. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 34(4), 546–562 (2004)

    Article  Google Scholar 

  18. Alepis, E., Virvou, M.: Object oriented design for multiple modalities in affective interaction. Intell. Syst. Reference Libr. 64, 87–99 (2014)

    Article  Google Scholar 

  19. Kabassi, K., Virvou, M.: A knowledge-based software life-cycle framework for the incorporation of multicriteria analysis in intelligent user interfaces. IEEE Trans. Knowl. Data Eng. 18, 1265–1277 (2006)

    Article  Google Scholar 

  20. Naumann, F.: Data fusion and data quality. In: Proceedings of the New Techniques and Technologies for Statistics (1998)

    Google Scholar 

  21. Kabassi, K., Virvou, M.: A knowledge-based software life-cycle framework for the incorporation of multicriteria analysis in intelligent user interfaces. IEEE Trans. Knowl. Data Eng. 18(9), 1265–1277 (2006), art. no. 1661516

    Google Scholar 

  22. Schütz, W., Schäfer, R.: Bayesian networks for estimating the user’s interests in the context of a configuration task. In: UM2001 Workshop on Machine Learning for User Modeling, pp. 23–36 (2001)

    Google Scholar 

  23. Bohnenberger, T., Jacobs, O., Jameson, A., Aslan, I.: Decision-theoretic planning meets user requirements: enhancements and studies of an intelligent shopping guide. In: Pervasive Computing, pp. 279–296 (2005)

    Google Scholar 

  24. Chin, D.N., Porage, A.: Acquiring user preferences for product customization. In: Proceedings of the 8th International Conference on User Modeling 2001, Springer, Berlin (2001)

    Google Scholar 

  25. Kudenko, D., Bauer, M., Dengler, D.: Group decision making through mediated discussions. In: User Modeling, 2003, pp. 147–147

    Google Scholar 

  26. Vincke, P.: Multicriteria Decision-Aid. Wiley, New York (1992)

    Google Scholar 

  27. Triantaphyllou, E., Mann, S.H.: An examination of the effectiveness of multi-dimensional decision-making methods: a decision-making paradox. Decis. Support Syst. 5, 303–312 (1989)

    Article  Google Scholar 

  28. Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1998)

    Google Scholar 

  29. Ekman, P., Friesen, W.V.: Unmasking the Face. A Guide to Recognizing Emotions from Facial Clues. Prentice-Hall, Englewood Cliffs (1975)

    Google Scholar 

  30. Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  31. Ekman, P.: Emotion in the Human Face. Cambridge University Press, New York (1982)

    Google Scholar 

  32. Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes between emotions. Science 221(4616), 1208–1210 (1983)

    Article  Google Scholar 

  33. Ekman, P., Davidson, R.J.: The Nature of Emotion. Fundamental Questions. Oxford University Press Inc., Oxford (1994)

    Google Scholar 

  34. Ekman, P.: Basic emotions. In: Dalgleish, T., Power, M.J. (eds.) Handbook of Cognition and Emotion. Wiley, Sussex (1999)

    Google Scholar 

  35. Tomkins, S.S.: Affect as amplification: some modifications in theory. In: Plutchik, R., Kellerman, H. (eds.) Emotion: Theory, Research and Experience, vol. 1: Theories of Emotion. Academic Press, New York (1980)

    Google Scholar 

  36. Tomkins, S.S.: Script theory: differential magnification of affects. In: Howe, J.H.E., Dienstbier, R.A. (eds.) Nebraska symposium on motivation, vol. 26. Lincoln University of Nebraska Press, Lincoln (1999)

    Google Scholar 

  37. Plutchik, R.: The Emotions: Facts, Theories, and a New Model. Random House, New York (1962)

    Google Scholar 

  38. Plutchik, R.: A general psychoevolutionary theory of emotion. In: Plutchik, R., Kellerman, H. (eds.) Emotion: Theory, Research and Experience, vol. 1: Theories of Emotion. Academic Press, New York, pp. 3–33 (1980)

    Google Scholar 

  39. Oatley, K., Johnson-Laird, P.N.: Towards a cognitive theory of emotions. Cogn. Emot. 1, 29–50 (1987)

    Article  Google Scholar 

  40. McDougall, W.: An Introduction to Social Psychology. Luce and Co., Boston (1926)

    Google Scholar 

  41. Izard, C.E.: The Face of Emotion. Appleton-Century-Crofts, New York (1972)

    Google Scholar 

  42. Izard, C.E.: Human Emotions. Plenum, New York (1977)

    Book  Google Scholar 

  43. Frijda, N.: The Emotions. Cambridge University Press, New York (1987)

    Google Scholar 

  44. Arnold, M.B.: Emotion and Personality. Columbia University Press, New York (1960)

    Google Scholar 

  45. Weiner, B.: An attributional theory of achievement motivation and emotion. Psychol. Rev. 92, 548–573 (1985)

    Article  Google Scholar 

  46. Martinez A.M.: The AR face database, CVC Technical Report (1998)

    Google Scholar 

  47. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: Proceedings of the 3rd International Conference on Face and Gesture Recognition. IEEE Computer Society (1998)

    Google Scholar 

  48. The Yale Database. http://cvc.yale.edu/projects/yalefaces/yalefaces.html

  49. Kanade, T., Tian, Y., Cohn, J.F.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000: IEEE Computer Society (2000)

    Google Scholar 

  50. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference Multimedia and Expo (ICME’05), Amsterdam, The Netherlands (2005)

    Google Scholar 

  51. Stathopoulou, I.-O., Tsihrintzis, G.A.: Facial expression classification: specifying requirements for an automated system. In: Knowledge-Based Intelligent Information and Engineering Systems, pp. 1128–1135 (2006)

    Google Scholar 

  52. Alepis, E., Virvou, M., Kabassi, K.: Development process of an affective bi-modal Intelligent Tutoring System. Intell. Decis. Technol. 1, 1–10 (2007)

    Google Scholar 

  53. Stathopoulou, I.-O., Tsihrintzis, G.: Emotion recognition from body movements and gestures. In: Smart Innovation, Systems and Technologies, 11 SIST, pp. 295–303 (2011)

    Google Scholar 

  54. Lampropoulos, A.S., Stathopoulou, I.-O., Tsihrintzis, G.A.: Comparative performance evaluation of classifiers for facial expression recognition. Stud. Comput. Intell. 226, 253–263 (2009)

    Article  Google Scholar 

  55. Stathopoulou, I.-O., Tsihrintzis, G.A.: NEU-FACES: a neural network-based face image analysis system. In: Proceedings of the 8th International Conference on Adaptive and Natural Computing Algorithms, Part II Warsaw. Springer, Poland (2007)

    Google Scholar 

  56. Stathopoulou, I.-O., Alepis, E., Tsihrintzis, G., Virvou, M.: On assisting a visual-facial affect recognition system with keyboard-stroke pattern information. Knowl.-Based Syst. 23(4), 350–356 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Software Engineering Lab, Department of Informatics, University of Piraeus, 185 34, Piraeus, Greece

    Maria Virvou, George A. Tsihrintzis, Efthimios Alepis & Ioanna-Ourania Stathopoulou

  2. Department of Restoration and Conservation of Cultural Heritage, Technological Educational Institute of the Ionian Islands, 291 00, Zakynthos, Greece

    Katerina Kabassi

Authors
  1. Maria Virvou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George A. Tsihrintzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Efthimios Alepis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ioanna-Ourania Stathopoulou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Katerina Kabassi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Virvou .

Editor information

Editors and Affiliations

  1. Department of Informatics, University of Piraeus, Piraeus, Greece

    George A. Tsihrintzis

  2. Department of Informatics, University of Piraeus, Piraeus, Greece

    Maria Virvou

  3. Faculty of Education, Science, Technology & Mathematics, University of Canberra, Canberra, Australia

    Lakhmi C. Jain

  4. KES International, Shoreham-by-sea, United Kingdom

    Robert J. Howlett

  5. Nagoya University, Nagoya, Japan

    Toyohide Watanabe

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Virvou, M., Tsihrintzis, G.A., Alepis, E., Stathopoulou, IO., Kabassi, K. (2015). On the Use of Multi-attribute Decision Making for Combining Audio-Lingual and Visual-Facial Modalities in Emotion Recognition. In: Tsihrintzis, G., Virvou, M., Jain, L., Howlett, R., Watanabe, T. (eds) Intelligent Interactive Multimedia Systems and Services in Practice. Smart Innovation, Systems and Technologies, vol 36. Springer, Cham. https://doi.org/10.1007/978-3-319-17744-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-17744-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-17743-4

  • Online ISBN: 978-3-319-17744-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation