Sparse Temporal Representations for Facial Expression Recognition

  • S. W. Chew
  • R. Rana
  • P. Lucey
  • S. Lucey
  • S. Sridharan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7088)


In automatic facial expression recognition, an increasing number of techniques had been proposed for in the literature that exploits the temporal nature of facial expressions. As all facial expressions are known to evolve over time, it is crucially important for a classifier to be capable of modelling their dynamics. We establish that the method of sparse representation (SR) classifiers proves to be a suitable candidate for this purpose, and subsequently propose a framework for expression dynamics to be efficiently incorporated into its current formulation. We additionally show that for the SR method to be applied effectively, then a certain threshold on image dimensionality must be enforced (unlike in facial recognition problems). Thirdly, we determined that recognition rates may be significantly influenced by the size of the projection matrix Φ. To demonstrate these, a battery of experiments had been conducted on the CK+ dataset for the recognition of the seven prototypic expressions − anger, contempt, disgust, fear, happiness, sadness and surprise − and comparisons have been made between the proposed temporal-SR against the static-SR framework and state-of-the-art support vector machine.


sparse representation classification facial expression recognition temporal framework 


  1. 1.
    Kobayashi, H., Hara, F., Ikeda, S., Yamada, H.: A basic study of dynamic recognition of human facial expressions. In: 2nd IEEE International Workshop on Robot and Human Communication, pp. 271–275 (1993)Google Scholar
  2. 2.
    Pantic, M., Patras, I.: Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 3358–3363 (2005)Google Scholar
  3. 3.
    Valstar, M., Pantic, M.: Fully automatic facial action unit detection and temporal analysis. In: Computer Vision and Pattern Recognition Workshop CVPRW 2006, pp. 149–149 (2006)Google Scholar
  4. 4.
    Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 1683–1699 (2007)CrossRefGoogle Scholar
  5. 5.
    Wright, J., Yang, A., Ganesh, A., Sastry, S., Ma, Y.: Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2008)Google Scholar
  6. 6.
    Bociu, I., Pitas, I.: A new sparse image representation algorithm applied to facial expression recognition. In: Proceedings of the 14th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, 2004, p. 539 (2004)Google Scholar
  7. 7.
    Zafeiriou, S., Petrou, M.: Sparse representations for facial expressions recognition via l1 optimization. In: Computer Vision and Pattern Recognition Workshops (CVPRW), p. 32 (2010)Google Scholar
  8. 8.
    Cotter, S.: Recognition of occluded facial expressions using a fusion of localized sparse representation classifiers. In: Digital Signal Processing Workshop and IEEE Signal Processing Education Workshop (DSP/SPE), p. 437 (2011)Google Scholar
  9. 9.
    Mahoor, M., Zhou, M., Veon, K.L., Mavadati, S., Cohn, J.: Facial action unit recognition with sparse representation. In: Automatic Face & Gesture Recognition and Workshops, p. 336 (2011)Google Scholar
  10. 10.
    Lucey, P., Cohn, J., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: Proceedings of the IEEE Workshop on CVPR for Human Communicative Behavior Analysis (2010)Google Scholar
  11. 11.
    Yang, P., Liu, Q., Cui, X., Metaxas, D.: Facial expression recognition using encoded dynamic features. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  12. 12.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 915–928 (2007)CrossRefGoogle Scholar
  13. 13.
    Du, R., Wu, Q., He, X., Jia, W., Wei, D.: Facial expression recognition using histogram variances faces. In: 2009 Workshop on Applications of Computer Vision, WACV (2009)Google Scholar
  14. 14.
    Ying, Z.-L., Wang, Z.-W., Huang, M.-W.: Facial Expression Recognition Based on Fusion of Sparse Representation. In: Huang, D.-S., Zhang, X., Reyes García, C.A., Zhang, L. (eds.) ICIC 2010. LNCS, vol. 6216, pp. 457–464. Springer, Heidelberg (2010)Google Scholar
  15. 15.
    Candes, E.: The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique 346, 589–592 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Hubel, D.: Eye, Brain and Vision. Freeman (1987)Google Scholar
  17. 17.
    Rockinger, O., Fechner, T.: Pixel-level image fusion: The case of image sequences. In: Proc. SPIE, vol. 3374, pp. 378–388 (1998)Google Scholar
  18. 18.
    Canga, E.F.: Image fusion. Project report, Dept. Electronic and Electrical Eng., Univ. of Bath. (2002) Google Scholar
  19. 19.
    Cootes, T., Edwards, G., Taylor, C.: Active Appearance Models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 681–685 (2001)CrossRefGoogle Scholar
  20. 20.
    Ekman, P.: Emotion in the human face. Cambridge University Press (1982)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • S. W. Chew
    • 1
  • R. Rana
    • 1
    • 3
  • P. Lucey
    • 2
  • S. Lucey
    • 3
  • S. Sridharan
    • 1
  1. 1.Speech Audio Image and Video Technology LaboratoryUniversity of TechnologyQueenslandAustralia
  2. 2.Disney ResearchPittsburgh
  3. 3.Commonwealth Science and Industrial Research Organisation (CSIRO)Australia

Personalised recommendations