Spatiotemporal Features for Effective Facial Expression Recognition

  • Hatice Çınar Akakın
  • Bülent Sankur
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6553)


We consider two novel representations and feature extraction schemes for automatic recognition of emotion related facial expressions. In one scheme facial landmark points are tracked over successive video frames using an effective detector and tracker to extract landmark trajectories. Features are extracted from landmark trajectories using Independent Component Analysis (ICA) method. In the alternative scheme, the evolution of the emotion expression on the face is captured by stacking normalized and aligned faces into a spatiotemporal face cube. Emotion descriptors are then 3D Discrete Cosine Transform (DCT) features from this prism or DCT & ICA features. Several classifier configurations are used and their performance determined in detecting the 6 basic emotions. Decision fusion applied to classifiers improved the recognition performance of best classifier by 9 percentage points. The proposed method was evaluated user independently on the Cohn-Kanade facial expression database and a state-of-the-art 95.34 % recognition performance is achieved.


Facial expression analysis spatiotemporal features face prism 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: Survey of an emerging domain. Image and Vision Computing 27, 1743–1759 (2009)CrossRefGoogle Scholar
  2. 2.
    Gatica-Perez, D.: Automatic nonverbal analysis of social interaction in small groups: A review. Image and Vision Computing 27, 1775–1787 (2009)CrossRefGoogle Scholar
  3. 3.
    Sebe, N., Lew, M., Sun, Y., Cohen, I., Gevers, T., Huang, T.: Authentic facial expression analysis. Image and Vision Computing 25, 1856–1863 (2007)CrossRefGoogle Scholar
  4. 4.
    Zeng, Z., Pantic, M., Roisman, G., Huang, T.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 39–58 (2009)CrossRefGoogle Scholar
  5. 5.
    Hupont, I., Cerezo, E., Baldassarri, S.: Facial emotional classifier for natural interaction, vol. 7, pp. 1–12 (2008)Google Scholar
  6. 6.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 915–928 (2007)CrossRefGoogle Scholar
  7. 7.
    Zhang, Y., Ji, Q.: Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 699–714 (2005)CrossRefGoogle Scholar
  8. 8.
    Wang, T., James Lien, J.J.: Facial expression recognition system based on rigid and non-rigid motion separation and 3D pose estimation. Pattern Recognition 42, 962–977 (2009)CrossRefGoogle Scholar
  9. 9.
    Pantic, M., Patras, I.: Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 36, 433–449 (2006)CrossRefGoogle Scholar
  10. 10.
    Yang, P., Liu, Q., Metaxas, D.N.: Boosting encoded dynamic features for facial expression recognition. Pattern Recognition Letters 30, 132–139 (2009)CrossRefGoogle Scholar
  11. 11.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: Machine learning and application to spontaneous behavior. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 568–573 (2005)Google Scholar
  12. 12.
    Dornaika, F., Davoine, F.: Simultaneous facial action tracking and expression recognition in the presence of head motion. International Journal of Computer Vision 76, 257–281 (2008)CrossRefGoogle Scholar
  13. 13.
    Tsalakanidou, F., Malassiotis, S.: Real-time 2D+3D facial action and expression recognition. Pattern Recognition 43, 1763–1775 (2010)CrossRefGoogle Scholar
  14. 14.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Mea-surement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  15. 15.
    Tian, Y.L.: Evaluation of face resolution for expression analysis. In: CVPR Workshop on Face and Video, p. 82 (2004)Google Scholar
  16. 16.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns a comprehensive study. Image and Vision Computing 27 (2009)Google Scholar
  17. 17.
    Zhou, F., De la Torre, F., Cohn, J.: Unsupervised discovery of facial events. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2010)Google Scholar
  18. 18.
    Bartlett, M.S., Littlewort, G., Frank, M.G., Lainscsek, C., Fasel, I.R., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1, 22–35 (2006)CrossRefGoogle Scholar
  19. 19.
    Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2000, pp. 46–53 (2000)Google Scholar
  20. 20.
    Zeng, Z., Fu, Y., Roisman, G.I., Wen, Z., Hu, Y., Huang, T.S.: Spontaneous emotional facial expression detection. Journal of Multimedia 1, 1–8 (2006)CrossRefGoogle Scholar
  21. 21.
    Zeng, Z., Hu, Y., Fu, Y., Huang, T.S., Roisman, G.I., Wen, Z.: Audio-visual emotion recognition in adult attachment interview. In: Proceedings of the 8th International Conference on Multimodal Interfaces, pp. 139–145. ACM, New York (2006)CrossRefGoogle Scholar
  22. 22.
    Akakin, H.C., Sankur, B.: Analysis of Head and Facial Gestures Using Facial Landmark Trajectories. In: Fierrez, J., Ortega-Garcia, J., Esposito, A., Drygajlo, A., Faundez-Zanuy, M. (eds.) BioID_MultiComm2009. LNCS, vol. 5707, pp. 105–113. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  23. 23.
    Martinez, A.M.: Matching expression variant faces. Vision Research, 1047–1060 (2003)Google Scholar
  24. 24.
    Oja, E.: Independent component analysis: algorithms and applications. Neural Networks 13, 411–430 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Hatice Çınar Akakın
    • 1
  • Bülent Sankur
    • 1
  1. 1.Electrical & Electronics Engineering DepartmentBogazici UniversityBebekTurkey

Personalised recommendations