Advertisement

Understanding Facial Expression

  • Shaogang Gong
  • Tao Xiang

Abstract

Facial expression is a natural and efficient means for humans to communicate their emotions and intentions, as communication is primarily carried out face to face. Expression can be recognised by either static face images in isolation, or sequences of face images. For the former, it is assumed that the static visual appearance of a face contains enough information for conveying an expression. The latter exploits information from facial movement generated by expressions. Computer-based automatic facial expression recognition considers two problems: face image representation and expression classification. A good representational scheme aims to derive a set of features from face images that can most effectively capture the characteristics of facial expression. The optimal features should not only minimise visual appearance differences from the same type of expression, known as within-class variations, but also maximise differences between two different types of expressions, known as between-class variations. If indiscriminative image features are selected for a representation, it is difficult to achieve good recognition regardless the choice of a classification mechanism. In this chapter, we consider the problems of how to construct a suitable representation and design an effective classification model for both static image based and dynamic sequence based automatic facial expression recognition.

Keywords

Support Vector Machine Facial Expression Linear Discriminant Analysis Face Image Local Binary Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. In: European Conference on Computer Vision, Prague, Czech Republic, May 2004, pp. 469–481 (2004) Google Scholar
  2. Bartlett, M., Littlewort, G., Fasel, I., Movellan, R.: Real time face detection and facial expression recognition: Development and application to human computer interaction. In: Workshop on Human-Computer Interaction, Madison, USA, June 2003, pp. 53–60 (2003) Google Scholar
  3. Bartlett, M., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: IEEE Conference on Computer Vision and Pattern Recognition, San Diego, USA, June 2005, pp. 568–573 (2005) Google Scholar
  4. Bassili, J.N.: Emotion recognition: The role of facial movement and the relative importance of upper and lower area of the face. J. Pers. Soc. Psychol. 37(11), 2049–2058 (1979) CrossRefGoogle Scholar
  5. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 711–720 (1997) CrossRefGoogle Scholar
  6. Belkin, M., Niyogi, P.: Laplacian eigenmaps and spectral techniques for embedding and clustering. In: Advances in Neural Information Processing Systems, pp. 585–591 (2001) Google Scholar
  7. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006) MATHGoogle Scholar
  8. Chang, Y., Hu, C., Turk, M.: Manifold of facial expression. In: IEEE International Workshop on Analysis and Modeling of Faces and Gestures, Nice, France, October 2003, pp. 28–35 (2003) Google Scholar
  9. Cohen, I., Sebe, N., Garg, A., Chen, L., Huang, T.S.: Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91, 160–187 (2003) CrossRefGoogle Scholar
  10. Donato, G., Bartlett, M., Hager, J., Ekman, P., Sejnowski, T.: Classifying facial actions. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 974–989 (1999) CrossRefGoogle Scholar
  11. Essa, I., Pentland, A.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997) CrossRefGoogle Scholar
  12. Feng, X., Hadid, A., Pietikäinen, M.: A coarse-to-fine classification scheme for facial expression recognition. In: International Conference on Image Analysis and Recognition, pp. 668–675 (2004) CrossRefGoogle Scholar
  13. Feng, X., Pietikäinen, M., Hadid, T.: Facial expression recognition with local binary patterns and linear programming. Pattern Recognit. Image Anal. 15(2), 546–548 (2005) Google Scholar
  14. Fidaleo, D., Trivedi, M.: Manifold analysis of facial gestures for face recognition. In: ACM SIGMM Multimedia Biometrics Methods and Application Workshop, Berkley, USA, pp. 65–69 (2003) Google Scholar
  15. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997) MathSciNetMATHCrossRefGoogle Scholar
  16. Gong, S., McKenna, S., Collins, J.J.: An investigation into face pose distributions. In: IEEE International Conference on Automatic Face and Gesture Recognition, Killington, USA, October 1996, pp. 265–270 (1996) Google Scholar
  17. Gong, S., Psarrou, A., Romdhani, S.: Corresponding dynamic appearances. Image Vis. Comput. 20(4), 307–318 (2002) CrossRefGoogle Scholar
  18. Guo, G., Dyer, C.R.: Simultaneous feature selection and classifier training via linear programming: A case study for face expression recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, Madison, USA, June 2003, pp. 346–352 (2003) Google Scholar
  19. Hadid, A., Pietikäinen, M., Ahonen, T.: A discriminative feature space for detecting and recognizing faces. In: IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, June 2004, pp. 797–804 (2004) Google Scholar
  20. He, X., Niyogi, P.: Locality preserving projections. In: Advances in Neural Information Processing Systems, pp. 153–160 (2003) Google Scholar
  21. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.: Face recognition using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 328–340 (2005) CrossRefGoogle Scholar
  22. Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification. Technical report, National Taiwan University, Taipei (2003) Google Scholar
  23. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53 (2000) CrossRefGoogle Scholar
  24. Li, Y., Gong, S., Liddell, H.: Constructing facial identity surfaces for recognition. Int. J. Comput. Vis. 53(1), 71–92 (2003) CrossRefGoogle Scholar
  25. Liao, S., Fan, W., Chung, C.S., Yeung, D.Y.: Facial expression recognition using advanced local binary patterns, Tsallis entropies and global appearance features. In: IEEE International Conference on Image Processing, pp. 665–668 (2006) Google Scholar
  26. Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999) CrossRefGoogle Scholar
  27. McKenna, S., Gong, S., Würtz, R.P., Tanner, J., Banin, D.: Tracking facial motion using Gabor wavelets and flexible shape models. In: IAPR International Conference on Audio-Video Based Biometric Person Authentication, Crans-Montana, Switzerland, March 1997, pp. 35–43 (1997) CrossRefGoogle Scholar
  28. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distribution. Pattern Recognit. 29(1), 51–59 (1996) CrossRefGoogle Scholar
  29. Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002) CrossRefGoogle Scholar
  30. Padgett, C., Cottrell, G.: Representing face images for emotion classification. In: Advances in Neural Information Processing Systems, pp. 894–900 (1997) Google Scholar
  31. Pantic, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. 36(2), 433–449 (2006) CrossRefGoogle Scholar
  32. Pantic, M., Rothkrantz, L.: Expert system for automatic analysis of facial expression. Image Vis. Comput. 18(11), 881–905 (2000) CrossRefGoogle Scholar
  33. Pantic, M., Rothkrantz, L.: Facial action recognition for facial expression analysis from static face images. IEEE Trans. Syst. Man Cybern. 34(3), 1449–1461 (2004) CrossRefGoogle Scholar
  34. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, pp. 5–10 (2005) Google Scholar
  35. Romdhani, S., Gong, S., Psarrou, A.: Multi-view nonlinear active shape model using kernel pca. In: British Machine Vision Conference, Nottingham, UK, September 1999, pp. 483–492 (1999) Google Scholar
  36. Saul, L., Roweis, S.: Think globally, fit locally: Unsupervised learning of low dimensional manifolds. J. Mach. Learn. Res. 4, 119–155 (2003) MathSciNetGoogle Scholar
  37. Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Mach. Learn. 37(3), 297–336 (1999) MATHCrossRefGoogle Scholar
  38. Shan, C., Gong, S., McOwan, P.: Appearance manifold of facial expression. In: Sebe, N., Lew, M.S., Huang, T.S. (eds.) Computer Vision in Human-Computer Interaction. Lecture Notes in Computer Science, pp. 221–230. Springer, Berlin (2005a) CrossRefGoogle Scholar
  39. Shan, C., Gong, S., McOwan, P.: Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing, Genoa, Italy, September 2005, pp. 370–373 (2005b) Google Scholar
  40. Shan, C., Gong, S., McOwan, P.: Dynamic facial expression recognition using a Bayesian temporal manifold model. In: British Machine Vision Conference, Edinburgh, UK, September 2006, pp. 297–306 (2006a) Google Scholar
  41. Shan, C., Gong, S., McOwan, P.: A comprehensive empirical study on linear subspace methods for facial expression analysis. In: IEEE Workshop on Vision for Human-Computer Interaction, New York, USA, June 2006, pp. 153–158 (2006b) Google Scholar
  42. Shan, C., Gong, S., McOwan, P.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009) CrossRefGoogle Scholar
  43. Tenenbaum, J.B., Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000) CrossRefGoogle Scholar
  44. Tian, Y.: Evaluation of face resolution for expression analysis. In: IEEE Workshop on Face Processing in Video, Washington, DC, USA, June 2004 Google Scholar
  45. Turk, M., Pentland, A.: Face recognition using eigenfaces. In: IEEE Conference on Computer Vision and Pattern Recognition, Maui, USA, June 1991, pp. 586–591 (1991) Google Scholar
  46. Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Computer Vision and Pattern Recognition Workshop, pp. 149–154 (2006) Google Scholar
  47. Valstar, M., Patras, I., Pantic, M.: Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: Computer Vision and Pattern Recognition Workshop, pp. 76–84 (2005) Google Scholar
  48. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995) MATHCrossRefGoogle Scholar
  49. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998) MATHGoogle Scholar
  50. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Conference on Computer Vision and Pattern Recognition, Kauai, USA, December 2001, pp. 511–518 (2001) Google Scholar
  51. Viola, P., Jones, M., Snow, D.: Detecting pedestrians using patterns of motion and appearance. In: IEEE International Conference on Computer Vision, Nice, France, October 2003, pp. 734–741 (2003) CrossRefGoogle Scholar
  52. Würtz, R.P.: Multilayer Dynamic Link Networks for Establishing Image Point Correspondences and Visual Object Recognition. Verlag Harri Deutsch, Frankfurt am Main (1995) MATHGoogle Scholar
  53. Zhang, Z., Lyons, M.J., Schuster, M., Akamatsu, S.: Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, April 1998, pp. 454–459 (1998) CrossRefGoogle Scholar
  54. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonUK
  2. 2.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonUK

Personalised recommendations