Computerized Facial Emotion Expression Recognition

  • Mattis GeigerEmail author
  • Oliver Wilhelm
Part of the Studies in Neuroscience, Psychology and Behavioral Economics book series (SNPBE)


Facial emotion expressions are an important gateway for studying human emotions. For many decades, this research was limited to human ratings of arousal and valence of emotional expressions. Such ratings are very time-consuming and have limited objectivity due to rater biases. By exploiting improvements in machine learning, the demand for a swifter and more objective method to assess facial emotional expressions was met by a plethora of software. These novel approaches are based on theories of human perception and emotion and their algorithms are often trained with massive and almost-generalizable data bases. However, they still face limitations such as 2D recognition and cultural biases. Nevertheless, the accuracy of computerized emotion recognition software has surpassed human raters in many cases. Consequently, such software has become instrumental in psychological research and has delivered remarkable findings, e.g. on human emotional abilities and dynamic expressions. Furthermore, recent developments for mobile devices have introduced such software into daily life, allowing for the immediate and ambulatory assessment of facial emotion expression. These trends provide intriguing new opportunities for studying human emotions, such as photograph-based experience sampling, incidental or implicit data recording in interventions, and many more.


  1. Affectiva (2018) Affectiva Affdex.
  2. Baltrusaitis T, Robinson P, Morency L-P (2016) OpenFace: an open source facial behavior analysis toolkit.
  3. Bruce V, Young A (1986) Understanding face recognition. Br J Psychol 77:305–327. Scholar
  4. Calvo MG, Fernández-Martín A, Recio G, Lundqvist D (2018) Human observers and automated assessment of dynamic emotional facial expressions: KDEF-dyn database validation. Front Psych 9:2052Google Scholar
  5. Cannon WB (1927) The James-Lange theory of emotions: a critical examination and an alternative theory. Am J Psychol 39:106–124. Scholar
  6. Chen C, Crivelli C, Garrod OGB et al (2018) Distinct facial expressions represent pain and pleasure across cultures. Proc Natl Acad Sci U S A 115:E10013–E10021. Scholar
  7. Corneanu C, Simón MO, Cohn JF, Guerrero SE (2016) Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568. Scholar
  8. Darwin C (1987) Expression of the emotions in man and animal. Appleton and Company, New YorkGoogle Scholar
  9. De la Torre F, Chu WS, Xiong X et al (2015) IntraFace. Presented at 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), Ljubljana, Slovenia, 4–8 May 2015Google Scholar
  10. Del Líbano M, Calvo MG, Fernández-Martín A, Recio G (2018) Discrimination between smiling faces: human observers vs. automated face analysis. Acta Psychol 187:19–29. Scholar
  11. Ekman P (1992) An argument for basic emotions. Cogn Emot 6:169–200. Scholar
  12. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial action. Manual for the Facial Action Coding SystemGoogle Scholar
  13. Ekman P, Rosenberg E, Hager J (1998) Facial action coding system affect interpretation database (FACSAID).
  14. Emotient (2016a) Facet Emotient Inc.
  15. Emotient (2016b) FACET 2.0 performance evaluation.
  16. Emotient (2016c) Emotient SDK 4.1 performance evaluation.
  17. Hildebrandt A, Olderbak S, Wilhelm O (2015) Facial emotion expression, individual differences. In: Wright JD (ed) International encyclopedia of the social & behavioral sciences, 2nd edn. Elsevier, Oxford, pp 667–675CrossRefGoogle Scholar
  18. Hinton GE, Sejnowski TJ, Poggio TA (1999) Unsupervised learning: foundations of neural computation. MIT Press, CambridgeCrossRefGoogle Scholar
  19. Jack RE, Blais C, Scheepers C et al (2009) Cultural confusions show that facial expressions are not universal. Curr Biol 19:1543–1548. Scholar
  20. Jaswal VK, Akhtar N (2018) Being vs. appearing socially uninterested: challenging assumptions about social motivation in autism. Behav Brain Sci 1–84.
  21. Jeni LA, Cohn JF, Kanade T (2015) Dense 3D face alignment from 2D videos in real-time. Presented at 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), Ljubljana, Slovenia, 4–8 May 2015Google Scholar
  22. Kulke L, Feyerabend D, Schacht A (2018) Comparing the Affectiva iMotions facial expression analysis software with EMG. PsyArXiv.
  23. Marĉelja S (1980) Mathematical description of the responses of simple cortical cells*. J Opt Soc Am, JOSA 70:1297–1300. Scholar
  24. Mayer JD, Salovey P (1997) What is emotional intelligence? In: Salovey P, Sluyter D (eds) Emotional development and emotional intelligence: implications for educators. Basic Books, New York, pp 3–31Google Scholar
  25. Noldus Information Technology (2018) FaceReader.
  26. Olderbak S, Geiger M, Wilhelm O (2019) A call for revamping socio-emotional ability research in autism. Behav Brain Sci 42Google Scholar
  27. Olderbak S, Hildebrandt A, Pinkpank T et al (2014) Psychometric challenges and proposed solutions when scoring facial emotion expression codes. Behav Res Methods 46:992–1006. Scholar
  28. Reznick JS (1997) Intelligence, language, nature, and nurture in young twins. In: Sternberg RJ, Grigorenko E (eds) Intelligence, heredity, and environment. Cambridge University Press, New YorkGoogle Scholar
  29. Schachter S, Singer J (1962) Cognitive, social, and physiological determinants of emotional state. Psychol Rev 69:379–399CrossRefGoogle Scholar
  30. Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44:695–729. Scholar
  31. Tcherkassof A, Bollon T, Dubois M et al (2007) Facial expressions of emotions: a methodological contribution to the study of spontaneous and dynamic emotional faces. Eur J Soc Psychol 37:1325–1345. Scholar
  32. Trampe D, Quoidbach J, Taquet M (2015) Emotions in everyday life. PLoS ONE 10:e0145450. Scholar
  33. van de Ven R (2016a) Emotion Hero.
  34. van de Ven R (2016b) Emotion Hero. Accessed 4 Oct 2018
  35. Werner P, Al-Hamadi A, Walter S (2017) Analysis of facial expressiveness during experimentally induced heat pain. Presented at 2017 seventh IEEE international conference on affective computing and intelligent interaction workshops and demos (ACIIW). San Antonio, TX, USA, 23–26 October 2017Google Scholar
  36. Wilhelm O, Hildebrandt A, Manske K et al (2014) Test battery for measuring the perception and recognition of facial expressions of emotion. Front Psychol 5.
  37. Williams AC de C (2002) Facial expression of pain: an evolutionary account. Behav Brain Sci 25.
  38. Wilt J, Funkhouser K, Revelle W (2011) The dynamic relationships of affective synchrony to perceptions of situations. J Res Pers 45:309–321. Scholar
  39. Zhang C, Zhang Z (2010) A survey of recent advances in face detectionGoogle Scholar
  40. Zinkernagel A, Alexandrowicz RW, Lischetzke T, Schmitt M (2018) The blenderFace method: video-based measurement of raw movement data during facial expressions of emotion using open-source software. Behav Res Methods. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Individual Differences and Psychological Assessment, Faculty of Engineering, Computer Science and PsychologyInstitute of Psychology and Education, Ulm UniversityUlmGermany

Personalised recommendations