Recognizing Facial Expressions Using Model-Based Image Interpretation

  • Matthias Wimmer
  • Christoph Mayer
  • Bernd Radig
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5398)


Even if electronic devices widely occupy our daily lives, human-machine interaction still lacks intuition. Therefore, researchers intend to resolve these shortcomings by augmenting traditional systems with aspects of human-human interaction and consider human emotion, behavior, and intention.

This publication focusses on one aspect of this challenge: recognizing facial expressions. Our approach achieves real-time performance and provides robustness for real-world applicability. This computer vision task comprises of various phases for which it exploits model-based techniques that accurately localize facial features, seamlessly track them through image sequences, and finally infer facial expressions visible. We specifically adapt state-of-the-art techniques to each of these challenging phases. Our system has been successfully presented to industrial, political, and scientific audience in various events.


Support Vector Machine Facial Expression Feature Point Recognition Rate Emotion Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chibelushi, C.C., Bourel, F.: Facial expression recognition: A brief tutorial overview. In: Fisher, R. (ed.) CVonline: On-Line Compendium of Computer Vision (January 2003)Google Scholar
  2. 2.
    Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.: Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding (CVIU) special issue on face recognition 91(1-2), 160–187 (2003)CrossRefGoogle Scholar
  3. 3.
    Cohn, J., Zlochower, A., Lien, J.J.-J., Kanade, T.: Featurepoint tracking by optical flow discriminates subtle differences in facial expression. In: Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition, April 1998, pp. 396–401 (1998)Google Scholar
  4. 4.
    Cohn, J., Zlochower, A., Lien, J.J.-J., Kanade, T.: Automated face analysis by feature point tracking has high concurrent validity with manual facs coding. Psychophysiology 36, 35–43 (1999)CrossRefGoogle Scholar
  5. 5.
    Cootes, T.F., Taylor, C.J.: Active shape models – smart snakes. In: Proceedings of the 3rd British Machine Vision Conference, pp. 266–275. Springer, Heidelberg (1992)Google Scholar
  6. 6.
    Edwards, G.J., Cootes, T.F., Taylor, C.J.: Face recognition using active appearance models. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1407, pp. 581–595. Springer, Heidelberg (1998)Google Scholar
  7. 7.
    Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Cole, J. (ed.) Nebraska Symposium on Motivation 1971, Lincoln, NE, vol. 19, pp. 207–283. University of Nebraska Press (1972)Google Scholar
  8. 8.
    Ekman, P.: Facial expressions. In: Dalgleish, T., Power, M. (eds.) Handbook of Cognition and Emotion, John Wiley & Sons Ltd, New York (1999)Google Scholar
  9. 9.
    Ekman, P., Friesen, W.: The Facial Action Coding System: A Technique for The Measurement of Facial Movement. Consulting Psychologists Press, San Francisco (1978)Google Scholar
  10. 10.
    Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 757–763 (1997)CrossRefGoogle Scholar
  11. 11.
    Fischer, S., Döring, S., Wimmer, M., Krummheuer, A.: Experiences with an emotional sales agent. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 309–312. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  12. 12.
    Friesen, W.V., Ekman, P.: Emotional Facial Action Coding System. University of California at San Francisco (1983); unpublished manuscriptGoogle Scholar
  13. 13.
    Hanek, R.: Fitting Parametric Curve Models to Images Using Local Selfadapting Seperation Criteria. PhD thesis, Department of Informatics, Technische Universität München (2004)Google Scholar
  14. 14.
    Ikehara, C.S., Chin, D.N., Crosby, M.E.: A model for integrating an adaptive information filter utilizing biosensor data to assess cognitive load. In: Brusilovsky, P., Corbett, A.T., de Rosis, F. (eds.) UM 2003. LNCS, vol. 2702, pp. 208–212. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  15. 15.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: International Conference on Automatic Face and Gesture Recognition, France, pp. 46–53 (March 2000)Google Scholar
  16. 16.
    Lisetti, C.L., Schiano, D.J.: Automatic facial expression interpretation: Where human interaction, articial intelligence and cognitive science intersect. Pragmatics and Cognition, Special Issue on Facial Information Processing and Multidisciplinary Perspective (1999)Google Scholar
  17. 17.
    Littlewort, G., Fasel, I., Bartlett, M.S., Movellan, J.R.: Fully automatic coding of basic expressions from video. Technical report, University of California, San Diego, INC MPLab (March 2002)Google Scholar
  18. 18.
    Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: Fifth International Conference on Multimodal Interfaces, Vancouver, pp. 258–264 (2003)Google Scholar
  19. 19.
    Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  20. 20.
    Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  21. 21.
    Schuller, B., Wimmer, M., Arsic, D., Rigoll, G., Radig, B.: Audiovisual behavior modeling by combined feature spaces. In: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Honolulu, Hawaii, USA, April 2007, vol. 2, pp. 733–736 (2007)Google Scholar
  22. 22.
    Schweiger, R., Bayerl, P., Neumann, H.: Neural architecture for temporal emotion classification. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 49–52. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  23. 23.
    Sebe, N., Lew, M.S., Cohen, I., Garg, A., Huang, T.S.: Emotion recognition using a cauchy naive bayes classifier. In: Proceedings of the 16th International Conference on Pattern Recognition, vol. 1, pp. 17–20. IEEE Computer Society, Washington (2002)Google Scholar
  24. 24.
    Sheldon, E.M.: Virtual agent interactions. PhD thesis, Elizabeth Sheldon, Major Professor-Linda Malone (2001)Google Scholar
  25. 25.
    Tian, Y.-L., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 97–115 (2001)CrossRefGoogle Scholar
  26. 26.
    Vick, R.M., Ikehara, C.S.: Methodological issues of real time data acquisition from multiple sources of physiological data. In: Proceedings of the 36th Annual Hawaii International Conference on System Sciences, p. 129. IEEE Computer Society, Washington (2003)Google Scholar
  27. 27.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, Kauai, Hawaii, vol. 1, pp. 511–518 (2001)Google Scholar
  28. 28.
    Wimmer, M.: Model-based Image Interpretation with Application to Facial Expression Recognition. PhD thesis, Technische Universitat München, Institute for Informatics (December 2007)Google Scholar
  29. 29.
    Wimmer, M., Stulp, F., Pietzsch, S., Radig, B.: Learning local objective functions for robust face model fitting. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 30(8), 1357–1370 (2008)CrossRefGoogle Scholar
  30. 30.
    Wimmer, M., Stulp, F., Tschechne, S., Radig, B.: Learning robust objective functions for model fitting in image understanding applications. In: Chantler, M.J., Trucco, E., Fisher, R.B. (eds.) Proceedings of the 17th British Machine Vision Conference (BMVC), vol. 3, pp. 1159–1168. BMVA, Edinburgh (September 2006) (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Matthias Wimmer
    • 1
  • Christoph Mayer
    • 1
  • Bernd Radig
    • 1
  1. 1.Image Understanding and Knowledge-Based Systems ChairTechnische Universität MünchenGermany

Personalised recommendations