Similarity Features for Facial Event Analysis

  • Peng Yang
  • Qingshan Liu
  • Dimitris Metaxas
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5302)


Each facial event will give rise to complex facial appearance variation. In this paper, we propose similarity features to describe the facial appearance for video-based facial event analysis. Inspired by the kernel features, for each sample, we compare it with the reference set with a similarity function, and we take the log-weighted summarization of the similarities as its similarity feature. Due to the distinctness of the apex images of facial events, we use their cluster-centers as the references. In order to capture the temporal dynamics, we use the K-means algorithm to divide the similarity features into several clusters in temporal domain, and each cluster is modeled by a Gaussian distribution. Based on the Gaussian models, we further map the similarity features into dynamic binary patterns to handle the issue of time resolution, which embed the time-warping operation implicitly. The haar-like descriptor is used to extract the visual features of facial appearance, and Adaboost is performed to learn the final classifiers. Extensive experiments carried on the Cohn-Kanade database show the promising performance of the proposed method.


Facial Expression Training Sample Local Binary Pattern Facial Expression Recognition Facial Appearance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Fasel, B., Luettin, J.: Automatic Facial Expression Analysis: A Survey. Pattern Recognition 36, 259–275 (2003)CrossRefzbMATHGoogle Scholar
  2. 2.
    Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)CrossRefGoogle Scholar
  3. 3.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual and spontaneous expressions. In: Int. Conf. on Multimodal interfaces (2007)Google Scholar
  4. 4.
    Shan, C., Gong, S., McOwan, P.W.: Conditional mutual information based boosting for facial expression recognition. In: British Machine Vision Conference (2005)Google Scholar
  5. 5.
    Bartlett, M., Littlewort, G., Fasel, I., Movellan, J.: Real time face detection and facial expression recognition: Development and applications to human computer interaction. In: Computer Vision and Pattern Recognition Workshop on Human-Computer Interaction (2003)Google Scholar
  6. 6.
    Pantic, M., Rothkrantz, J.: Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man and Cybernetics (2004)Google Scholar
  7. 7.
    Shan, C., Gong, S., McOwan, P.W.: Robust facial expression recognition using local binary patterns. In: IEEE Int. Conf. on Image Processing (2005)Google Scholar
  8. 8.
    Bassili, J.: Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. J. Personality Socical Psychol. 37 (1979)Google Scholar
  9. 9.
    Ambadar, Z., Schooler, J., Cohn, J.F.: Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expression. Psychological Science (2005)Google Scholar
  10. 10.
    Black, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Computer Vision 25, 23–48 (1997)CrossRefGoogle Scholar
  11. 11.
    Yacoob, Y., Davis, L.: Computing spatio-temporal representations of human faces. Computer Vision and Pattern Recognition (1994)Google Scholar
  12. 12.
    Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.: Facial expression recognition from video sequences: Temporal and static modeling. Computer Vision and Image Understanding 91, 160–187 (2003)CrossRefGoogle Scholar
  13. 13.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)CrossRefGoogle Scholar
  14. 14.
    Tian, Y.: Evaluation of face resolution for expression analysis. In: Computer Vision and Pattern Recognition Workshop on Face Processing in Video (2004)Google Scholar
  15. 15.
    Yeasin, M., Bullot, B., Sharma, R.: From facial expression to level of interest: A spatio-temporal approach. Computer Vision and Pattern Recognition (2004)Google Scholar
  16. 16.
    Torre, F., Yacoob, Y., Davis, L.: A probabilistic framework for rigid and non-rigid appearance based tracking and recognition. In: The Fourth IEEE Int. Conf. on Automatic Face and Gesture Recognition (2001)Google Scholar
  17. 17.
    Cohn, J.: Automated analysis of the configuration and timing of facial expression. What the face reveals (2nd edition): Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS), 388 – 392 (2005)Google Scholar
  18. 18.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models: their training and application. Comput. Vis. Image Underst. 61, 38–59 (1995)CrossRefGoogle Scholar
  19. 19.
    Hu, C., Chang, Y., Feris, R., Turk, M.: Manifold based analysis of facial expression. In: Computer Vision and Pattern Recognition Workshop (2004)Google Scholar
  20. 20.
    Lee, C.S., Elgammal, A.: Facial expression analysis using nonlinear decomposable generative models. In: Zhao, W., Gong, S., Tang, X. (eds.) AMFG 2005. LNCS, vol. 3723, pp. 17–31. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  21. 21.
    Ke, Y., Sukthankar, R., Hebert, M.: Efficient visual event detection using volumetric features. In: IEEE International Conference on Computer Vision (2005)Google Scholar
  22. 22.
    Cui, X., Liu, Y., Shan, S., Chen, X., Gao, W.: 3d haar-like features for pedestrian detection. In: IEEE International Conference on Multimedia and Expo. (2007)Google Scholar
  23. 23.
    Yang, P., Liu, Q., Metaxas, D.N.: Boosting coded dynamic features for facial action units and facial expression recognition. Computer Vision and Pattern Recognition (2007)Google Scholar
  24. 24.
    Tversky, A.: Features of similarity. Psychological Review (1977)Google Scholar
  25. 25.
    Liu, Q., Jin, H., Tang, X., Lu, H., Ma, S.: A new extension of kernel feature and its application for visual recognition. Neurocomput. 71, 1850–1856 (2008)CrossRefGoogle Scholar
  26. 26.
    Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2, 121–167 (1998)CrossRefGoogle Scholar
  27. 27.
    Daugman, J.: Demodulation by complex-valued wavelets for stochastic pattern recognition. Int’l J. Wavelets, Multiresolution and Information Processing (2003)Google Scholar
  28. 28.
    Viola, P., Jones, M.: Robust real-time object detection. Int. J. Computer Vision 57, 137–154Google Scholar
  29. 29.
    Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24, 971–987 (2002)CrossRefzbMATHGoogle Scholar
  30. 30.
    Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG 2000) (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Peng Yang
    • 1
  • Qingshan Liu
    • 1
    • 2
  • Dimitris Metaxas
    • 1
  1. 1.Rutgers UniversityPiscatawayUSA
  2. 2.National Laboratory of Pattern RecognitionChinese Academy of SciencesBeijingChina

Personalised recommendations