Advertisement

Recognizing Facial Expressions Automatically from Video

  • Caifeng Shan
  • Ralph Braspenning

Abstract

Facial expressions, resulting from movements of the facial muscles, are the face changes in response to a person’s internal emotional states, intentions, or social communications. There is a considerable history associated with the study on facial expressions. Darwin [22] was the first to describe in details the specific facial expressions associated with emotions in animals and humans, who argued that all mammals show emotions reliably in their faces. Since that, facial expression analysis has been a area of great research interest for behavioral scientists [27]. Psychological studies [48, 3] suggest that facial expressions, as the main mode for nonverbal communication, play a vital role in human face-to-face communication. For illustration, we show some examples of facial expressions in Fig. 1.

Keywords

Face Image Local Binary Pattern Gesture Recognition Facial Expression Recognition Facial Action Code System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Ahonen T, Hadid A, Pietikäinen M (2004) Face recognition with local binary patterns. In: European Conference on Computer Vision (ECCV), pp 469–481Google Scholar
  2. [2]
    Ambadar Z, Schooler JW, Cohn JF (2005) Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions. Psychological Science 16(5):403–410CrossRefGoogle Scholar
  3. [3]
    Ambady N, Rosenthal R (1992) Thin slices of expressive behaviour as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin 111(2):256–274CrossRefGoogle Scholar
  4. [4]
    Amin MA, Afzulpurkar NV, Dailey MN, Esichaikul VE, Batanov DN (2005) Fuzzy-c-mean determines the principle component pairs to estimate the degree of emotion from facial expressions. In: International Conference on Natural Computation and International Conference on Fuzzy Systems and Knowledge Discovery, pp 484–493Google Scholar
  5. [5]
    Argyle M (1988) Bodily Communication (2nd ed.). Methuen & Co. Ltd., New YorkGoogle Scholar
  6. [6]
    Balomenos T, Raouzaiou A, Ioannou S, Drosopoulos A, Karpouzis K, Kollias S (2005) Emotion analysis in man-machine interaction systems. In: Machine Learning for Multimodal Interaction, LNCS 3361, pp 318–328Google Scholar
  7. [7]
    Bartlett M, Littlewort G, Frank M, Lainscseck C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1(6):22–35CrossRefGoogle Scholar
  8. [8]
    Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Transactions on Neural Networks 13(6):1450–1464CrossRefGoogle Scholar
  9. [9]
    Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: Machine learning and application to spotaneous behavior. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 568–573Google Scholar
  10. [10]
    Bassili JN (1979) Emotion recognition: The role of facial movement and the relative importance of upper and lower area of the face. Journal of Personality and Social Psychology 37(11):2049–2058CrossRefGoogle Scholar
  11. [11]
    Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):711–720CrossRefGoogle Scholar
  12. [12]
    Braathen B, Bartlett M, Littlewort G, Smith E, Movellan JR (2002) An approach to automatic recognition of spontaneous facial actions. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 231–235Google Scholar
  13. [13]
    Burgoon JK, Buller DB, Woodall WG (1996) Nonverbal Communication: The Unspoken Dialogue. McGraw-Hill, New YorkGoogle Scholar
  14. [14]
    Chang Y, Hu C, Turk M (2003) Mainfold of facial expression. In: IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG), pp 28–35Google Scholar
  15. [15]
    Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 520–527Google Scholar
  16. [16]
    Cohen I, Sebe N, Garg A, Chen L, Huang TS (2003) Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91:160–187CrossRefGoogle Scholar
  17. [17]
    Cohn JF, Schmidt KL (2004) The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing 2:1–12CrossRefMathSciNetGoogle Scholar
  18. [18]
    Cohn JF, Reed LI, Ambadar Z, Xiao J, Moriyama T (2004) Automatic analysis and recognition of brow actions in spontaneous facial behavior. In: IEEE International Conference on Systems, Man, and Cybernetics, p 610Google Scholar
  19. [19]
    Costa M, Dinsbach W, Manstead ASR, Bitti PER (2001) Social presence, embarrassment, and nonverbal behavior. Journal of Nonverbal Behavior 25(4):225–240CrossRefGoogle Scholar
  20. [20]
    Coulson M (2004) Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence.Journal of Nonverbal Behavior 28(2):117–139CrossRefMathSciNetGoogle Scholar
  21. [21]
    Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schroder M (2000) ’feeltrace’: An instrument for recording perceived emotion in real time. In: Proceeding of the ISCA Workshop on Speech and Emotion, pp 19–24Google Scholar
  22. [22]
    Darwin C (1872) The Expression of the Emotions in Man and Animals. John Murray, LondonCrossRefGoogle Scholar
  23. [23]
    Donato G, Bartlett M, Hager J, Ekman P, Sejnowski T (1999) Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(10):974–989CrossRefGoogle Scholar
  24. [24]
    Douglas-Cowie E, Cowie R, Schroder M (2000) A new emotion database: Considerations, sources and scope. In: Proceeding of the ISCA Workshop on Speech and Emotion: A Conceptual Framework for Research, pp 39–44Google Scholar
  25. [25]
    Ekman P, Friesen W (1971) Constants across cultures in the face and emotion. J Personality Social Psychol 17(2):124–129CrossRefGoogle Scholar
  26. [26]
    Ekman P, Rosenberg E (1997) What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS). New York: Oxford Univ. PressGoogle Scholar
  27. [27]
    Ekman P, Friesen WV, Hager JC (2002) The Facial Action Coding System: A Technique for the Measurement of Facial Movement. San Francisco: Consulting PsychologistGoogle Scholar
  28. [28]
    Essa I, Pentland A (1997) Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):757–763CrossRefGoogle Scholar
  29. [29]
    Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognition 36:259–275MATHCrossRefGoogle Scholar
  30. [30]
    de Gelder B (2006) Towards the neurobiology of emotional body language. Nature Reviews Neuroscience 7:242–249CrossRefGoogle Scholar
  31. [31]
    Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression recognition with face registration errors. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), Amsterdam, The NetherlandsGoogle Scholar
  32. [32]
    Gu H, Ji Q (2005) Information extraction from image sequences of real-world facial expressions. Machine Vision and Applications 16:105–115CrossRefGoogle Scholar
  33. [33]
    Gunes H, Piccardi M (2006) Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications (In press)Google Scholar
  34. [34]
    Gunes H, Piccardi M (2006) A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: International Conference on Pattern Recognition (ICPR), vol 1, pp 1148–1153Google Scholar
  35. [35]
    Guo G, Dyer CR (2003) Simultaneous feature selection and classifier training via linear programming: A case study for face expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 346–352Google Scholar
  36. [36]
    He X, Niyogi P (2003) Locality preserving projections. In: Advances in Neural Information Processing Systems (NIPS)Google Scholar
  37. [37]
    Hess U, Blairy S, Kleck RE (1997) The intensity of emotional facial expression and decoding accuracy. Journal of Nonverbal Behavior 21(4):241–257CrossRefGoogle Scholar
  38. [38]
    Kaliouby RE, Robinson P (2004) Real-time inference of complex mental states from facial expressions and head gestures. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp 154–154Google Scholar
  39. [39]
    Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 46–53Google Scholar
  40. [40]
    Kapoor A, Picard RW (2005) Multimodal affect recognition in learning environments. In: ACM International Conference on Multimedia, pp 677–682Google Scholar
  41. [41]
    Kimura S, Yachida M (1997) Facial expression recognition and its degree estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 295–300Google Scholar
  42. [42]
    Lee CS, Elgammal A (2005) Facial expression analysis using nonlinear decomposable generative models. In: IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG), pp 17–31Google Scholar
  43. [43]
    Lien JJ, Kanade T, Cohn JF, Li C (1998) Subtly different facial expression recognition and expression intensity estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 853–859Google Scholar
  44. [44]
    Littlewort G, Bartlett M, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image and Vision Computing 24(6):615–625CrossRefGoogle Scholar
  45. [45]
    Littlewort G, Bartlett MS, Lee K (2006) Faces of pain: Automated measurement of spontaneous facial expressions of genuine and posed pain. In: Joint Symposium on Neural Computation, pp 1–1Google Scholar
  46. [46]
    Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(12):1357–1362CrossRefGoogle Scholar
  47. [47]
    Meeren H, Heijnsbergen C, Gelder B (2005) Rapid perceptual integration of facial expression and emotional body language. Procedings of the National Academy of Sciences of USA 102(45):16,518–16,523Google Scholar
  48. [48]
    Mehrabian A (1968) Communication without words. Psychology Today 2(4):53–56Google Scholar
  49. [49]
    Nayak V, Turk M (2005) Emotional expression in virtual agents through body language. In: International Symposium on Visual Computing, pp 313–320Google Scholar
  50. [50]
    Neagle RJ, Ng K, Ruddle RA (2003) Studying the fidelity requirements for a virtual ballet dancer. In: Vision, Video and Graphics, pp 181–188Google Scholar
  51. [51]
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distribution. Pattern Recognition 29(1):51–59CrossRefGoogle Scholar
  52. [52]
    Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971–987CrossRefGoogle Scholar
  53. [53]
    Oliver N, Pentland A, Berard F (2000) Lafter: a real-time face and lips tracker with facial expression recognition. Pattern Recognition 33:1369–1382CrossRefGoogle Scholar
  54. [54]
    O’Toole AJ, Harms J, Snow SL, Hurst DR, Pappas MR, Ayyad JH, Abdi H (2005) A video database of moving faces and people. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5):812–816CrossRefGoogle Scholar
  55. [55]
    Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Kurihara K (ed) Face Recognition, Advanced Robotics Systems, Vienna, Austria, pp 377–416Google Scholar
  56. [56]
    Pantic M, Patras I (2006) Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics 36(2):433–449CrossRefGoogle Scholar
  57. [57]
    Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1424–1445CrossRefGoogle Scholar
  58. [58]
    Pantic M, Rothkrantz L (2000) Expert system for automatic analysis of facial expression. Image and Vision Computing 18(11):881–905CrossRefGoogle Scholar
  59. [59]
    Pantic M, Rothkrantz L (2003) Toward an affect-sensitive multimodal human-computer interaction. In: Proceeding of the IEEE, vol 91, pp 1370–1390Google Scholar
  60. [60]
    Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man, and Cybernetics 34(3):1449–1461CrossRefGoogle Scholar
  61. [61]
    Pantic M, Sebe N, Cohn J, Huang T (2005) Affective multimodal human-computer interaction. In: ACM International Conference on Multimedia, pp 669–676Google Scholar
  62. [62]
    Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo (ICME), pp 317–321Google Scholar
  63. [63]
    Pavlovic VI, Sharma R, Huang TS (1997) Visual interpretation of hand gestures for human-computer interaction: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7):677–695CrossRefGoogle Scholar
  64. [64]
    Ravindra De Silva P, Osano M, Marasinghe A (2006) Towards recognizing emotion with affective dimensions through body gestures. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 269–274Google Scholar
  65. [65]
    Russell JA (1994) Is there univeral recognition of emotion from facial expression. Psychological Bulletin 115(1):102–141CrossRefGoogle Scholar
  66. [66]
    Schapire RE, Singer Y (1999) Improved boosting algorithms using confidence-rated predictions. Maching Learning 37(3):297–336MATHCrossRefGoogle Scholar
  67. [67]
    Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS (2004) Authentic facial expression analysis. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 517–522Google Scholar
  68. [68]
    Shan C (2007) Inferring facial and body language. PhD thesis, Queen Mary, University of LondonGoogle Scholar
  69. [69]
    Shan C, Gritti T (2008) Learning discriminative lbp-histogram bins for facial expression recognition. In: British Machine Vision Conference (BMVC), Leeds, UKGoogle Scholar
  70. [70]
    Shan C, Gong S, McOwan PW (2005) Appearance manifold of facial expression. In: Sebe N, Lew MS, Huang TS (eds) Computer Vision in Human-Computer Interaction, Lecture Notes in Computer Science, vol 3723, Springer, pp 221–230Google Scholar
  71. [71]
    Shan C, Gong S, McOwan PW (2005) Conditional mutual information based boosting for facial expression recognition. In: British Machine Vision Conference (BMVC), Oxford, UK, vol 1, pp 399–408Google Scholar
  72. [72]
    Shan C, Gong S, McOwan PW (2005) Recognizing facial expressions at low resolution. In: IEEE International Conference on Advanced Video and Signal-Based Surveillance (AVSS), Como, Italy, pp 330–335Google Scholar
  73. [73]
    Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing (ICIP), Genoa, Italy, vol 2, pp 370–373Google Scholar
  74. [74]
    Shan C, Gong S, McOwan PW (2006) A comprehensive empirical study on linear subspace methods for facial expression analysis. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, New York, USA, pp 153–158Google Scholar
  75. [75]
    Shan C, Gong S, McOwan PW (2006) Dynamic facial expression recognition using a bayesian temporal manifold model. In: British Machine Vision Conference (BMVC), Edinburgh, UK, vol 1, pp 297–306Google Scholar
  76. [76]
    Shan C, Gong S, McOwan PW (2007) Beyond facial expressions: Learning human emotion from body gestures. In: British Machine Vision Conference (BMVC), Warwick, UKGoogle Scholar
  77. [77]
    Shan C, Gong S, McOwan PW (2007) Capturing correlations among facial parts for facial expression analysis. In: British Machine Vision Conference (BMVC), Warwick, UKGoogle Scholar
  78. [78]
    Shan C, Gong S, McOwan PW (2008) Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision ComputingGoogle Scholar
  79. [79]
    Skelley J, Fischer R, Sarma A, Heisele B (2006) Recognizing expressions in a new database containing played and natural expressions. In: International Conference on Pattern Recognition (ICPR), pp 1220–1225Google Scholar
  80. [80]
    Tian Y (2004) Evaluation of face resolution for expression analysis. In: International Workshop on Face Processing in Video, pp 82–82Google Scholar
  81. [81]
    Tian Y, Kanade T, Cohn J (2000) Eye-state action unit detection by gabor wavelets. In: International Conference on Multimodal Interfaces (ICMI), pp 143–150Google Scholar
  82. [82]
    Tian Y, Kanade T, Cohn J (2001) Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2):97–115CrossRefGoogle Scholar
  83. [83]
    Tian Y, Brown L, Hampapur A, Pankanti S, Senior A, Bolle R (2003) Real world real-time automatic recognition of facial expression. In: IEEE workshop on performance evaluation of tracking and surveillance (PETS), AustraliaGoogle Scholar
  84. [84]
    Tian Y, Kanade T, Cohn J (2005) Handbook of Face Recognition, Springer, chap 11. Facial Expression AnalysisGoogle Scholar
  85. [85]
    Tong Y, Liao W, Ji Q (2006) Inferring facial action units with causal relations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1623–1630Google Scholar
  86. [86]
    Turk M, Pentland AP (1991) Face recognition using eigenfaces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)Google Scholar
  87. [87]
    Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop, p 149Google Scholar
  88. [88]
    Valstar M, Pantic M, Ambadar Z, Cohn JF (2006) Spontaneous vs. posed facial behavior: Automatic analysis of brow actions. In: International Conference on Multimodal Interfaces (ICMI), pp 162–170Google Scholar
  89. [89]
    Wallhoff F (2006) Facial expressions and emotion database. http://www.mmk.ei.tum.de/waf/fgnet/feedtum.html
  90. [90]
    Yacoob Y, Davis LS (1996) Recognizing human facial expression from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6):636–642CrossRefGoogle Scholar
  91. [91]
    Yeasin M, Bullot B, Sharma R (2004) From facial expression to level of interests: A spatio-temporal approach. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 922–927Google Scholar
  92. [92]
    Yin L, Wei X, Sun Y, Wang J, Rosato M (2006) A 3d facial expression database for facial behavior research. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG)Google Scholar
  93. [93]
    Zeng Z, Fu Y, Roisman GI, Wen Z, Hu Y, Huang TS (2006) Spontaneous emotional facial expression detection. Journal of Multimedia (JMM) 1(5):1–8Google Scholar
  94. [94]
    Zhang G, Huang X, Li SZ, Wang Y, Wu X (2004) Boosting local binary pattern (lbp)-based face recognition. In: Chinese Conference on Biometric Recognition, pp 179–186Google Scholar
  95. [95]
    Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5):1–16MATHCrossRefMathSciNetGoogle Scholar
  96. [96]
    Zhang Z, Lyons MJ, Schuster M, Akamatsu S (1998) Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp 454–461Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Philips ResearchEindhovenThe Netherlands

Personalised recommendations