Advertisement

Multimedia Tools and Applications

, Volume 71, Issue 3, pp 1953–1974 | Cite as

Utilizing 3D flow of points for facial expression recognition

  • Ruchir Srivastava
  • Sujoy Roy
Article
  • 219 Downloads

Abstract

This paper presents an approach to recognize Facial Expressions of different intensities using 3D flow of facial points. 3D flow is the geometrical displacement (in 3D) of a facial point from its position in a neutral face to that in the expressive face. Experiments are performed on 3D face models from the BU-3DFE database. Four different intensities of expressions are used for analyzing the relevance of intensity of the expression for the task of FER. It was observed that high intensity expressions are easier to recognize and there is a need to develop algorithms for recognizing low intensity facial expressions. The proposed features outperform difference of facial distances and 2D optical flow. Performances of two classifiers, SVM and LDA are compared wherein SVM performs better. Feature selection did not prove useful.

Keywords

Facial expression recognition 3D flow 3D facial models Expression intensity 

References

  1. 1.
    Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: CVPR workshop on computer vision and pattern recognition for human-computer interactionGoogle Scholar
  2. 2.
    Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: machine learning and application to spontaneous behavior. In: IEEE computer society conference on computer vision and pattern recognitionGoogle Scholar
  3. 3.
    Bartlett MS, Littlewort G, Frank MG, Lainscsek C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. J Multimed 1(6):22–35CrossRefGoogle Scholar
  4. 4.
    Berretti S, Bimbo AD, Amor PPBB, Daoudi M (2010) A set of selected SIFT features for 3D facial expression recognition. In: Proc. 20th international conference on pattern recognition, pp 4125–4128Google Scholar
  5. 5.
    Chandrasiri NP, Naemura T, Ishizuka M, Harashima H (2004) Internet communication using real-time facial expression analysis and synthesis. IEEE Multimed 11(3):20–29CrossRefGoogle Scholar
  6. 6.
    Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machinesGoogle Scholar
  7. 7.
    Chang Y, Vieira M, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: Analysis and modeling of faces and gestures: second international workshopGoogle Scholar
  8. 8.
    Datchu D, Rothkrantz LJM (2005) Facial expression recognition with relevance vector machines. In: Proc. IEEE international conference on multimedia and expoGoogle Scholar
  9. 9.
    Dimensional Imaging (2011) Retrieved October 8, 2011. http://www.di3d.com/
  10. 10.
    Dornaika F, Davoine F (2008) Simultaneous facial action tracking and expression recognition in the presence of head motion. Int J Comput Vis 76(3):257–281CrossRefGoogle Scholar
  11. 11.
    Dornaika F, Raducanu B (2008) Facial expression recognition for HCI applications. Encycl Artif Intell 2:625–631CrossRefGoogle Scholar
  12. 12.
    Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129CrossRefGoogle Scholar
  13. 13.
    Ekman P, Friesen WV, Hager JC (1978) Facial action coding system. Consulting Psychologists Press, Palo Alto, CAGoogle Scholar
  14. 14.
    Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognit 36(1):259–275CrossRefMATHGoogle Scholar
  15. 15.
    Fasel B, Monay F, Gatica-Perez D (2004) Latent semantic analysis of facial action codes for automatic facial expression recognition. In: Proc. of the 6th ACM SIGMM international workshop on Multimedia information retrieval ACM New York, NY, USA, pp 181–188Google Scholar
  16. 16.
    Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3–4):143–166CrossRefMATHGoogle Scholar
  17. 17.
    Gokturk S, Bouguet J, Tomasi C, Girod B (2002) Model-based face tracking for view independent facial expression recognition. In: Proceedings of the fifth IEEE international conference on automatic face and gesture recognitionGoogle Scholar
  18. 18.
    Gong B, Wang Y, Liu J, Tang X (2009) Automatic facial expression recognition on a single 3D face by exploring shape deformation. In: Proc. 17th ACM international conference on multimedia, pp 569–572Google Scholar
  19. 19.
    Hahnel M, Wiratanaya A, Kraiss K (2006) Facial expression modeling from still images using a single generic 3d head model. Lecture notes in computer science, vol 4174, pp 324–333CrossRefGoogle Scholar
  20. 20.
    Hsu C, Chang C, Lin C (2003) A practical guide to support vector classification.Google Scholar
  21. 21.
    Kinect (2011) Retrieved October 10, 2011. http://www.xbox.com/en-US/kinect
  22. 22.
    Lin D, Yan S, Tang X (2005) Comparative study: face recognition on unspecific persons using linear subspace methods. In: Proc. IEEE international conference on image processingGoogle Scholar
  23. 23.
    Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2010) Local 3d shape analysis for facial expression recognition. In: Proc. IEEE international conference on pattern recognitionGoogle Scholar
  24. 24.
    Mpiperis I, Malassiotis S, Petridis V, Strintzis M (2008) 3D facial expression recognition using swarm intelligence. In: IEEE international conference on acoustics, speech and signal processing, pp 2133–2136Google Scholar
  25. 25.
    Mpiperis I, Malassiotis S, Strintzis M (2008) Bilinear models for 3-D face and facial expression recognition. IEEE Trans Inf Forensics Secur 3(3):498–511CrossRefGoogle Scholar
  26. 26.
    Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell PAMI 22(12):1424–1445CrossRefGoogle Scholar
  27. 27.
    Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: Proc. 2nd IEEE international conference on biometrics: theory, applications and systemsGoogle Scholar
  28. 28.
    Sha T, Song M, Bu J, Chen C, Tao D (2011) Feature level analysis for 3D facial expression recognition. Neurocomputing 74(12–13):2135–2141CrossRefGoogle Scholar
  29. 29.
    Soyel H, Demirel H (2007) Facial expression recognition using 3d facial feature distances. In: ICIAR. Lecture Notes in computer science, vol. 4633. Springer Berlin / Heidelberg, pp 831–838Google Scholar
  30. 30.
    Soyel H, Demirel H (2010) Optimal feature selection for 3D facial expression recognition using coarse-to-fine classification. Turkish J. Electr Eng Comput Sci 18(6):1031–1040Google Scholar
  31. 31.
    Srivastava R, Roy S (2009) 3D facial expression recognition using residues. In: Proc. IEEE region 10 conference, TENCONGoogle Scholar
  32. 32.
    Srivastava R, Sim T, Yan S, Ranganath S (2010) Feature selection for facial expression recognition using deformation modeling. In: Proc. international conference on digital image processingGoogle Scholar
  33. 33.
    Srivastava R, Roy S, Yan S, Sim T (2011) Multi-actor emotion recognition in movies using a bimodal approach. In: Proc. 17th international conference on multimedia modelingGoogle Scholar
  34. 34.
    Stylianou G, Lanitis A (2009) Image based 3D face reconstruction: a survey. Int J Image Graph 9(2):217–250CrossRefGoogle Scholar
  35. 35.
    Sun Y, Yin L (2008) 3D Spatio-Temporal face recognition using dynamic range model sequences. In: Proceedings IEEE computer society conference on computer vision and pattern recognition workshopsGoogle Scholar
  36. 36.
    Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Proceedings of the 10th European conference on computer visionGoogle Scholar
  37. 37.
    Sun Y, Chen X, Rosato M, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Part A Syst Humans 40(3):461–474CrossRefGoogle Scholar
  38. 38.
    Suwa M, Sugie N, Fujimora K (1978) A preliminary note on pattern recognition of human emotional expression. In: International joint conference on pattern recognition, pp 408–410Google Scholar
  39. 39.
    Tang H, Huang T (2008) 3D facial expression recognition based on automatically selected features. In: Proc. computer vision and pattern recognition workshops, pp 1–8Google Scholar
  40. 40.
    Tang H, Huang T (2008) 3D facial expression recognition based on properties of line segments connecting facial feature points. In: Proc. 8th IEEE international conference on automatic face and gesture recognition, pp 1–6Google Scholar
  41. 41.
    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86CrossRefGoogle Scholar
  42. 42.
    Venkatesh Y, Kassim A, Murthy O (2009) A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA. Pattern Recogn Lett 30(12):1128–1137CrossRefGoogle Scholar
  43. 43.
    Wang J, Yin L, Wei X, Sun Y (2006) 3d facial expression recognition based on primitive surface feature distribution In: Proc. IEEE international conference on computer vision and pattern recognitionGoogle Scholar
  44. 44.
    Weiss SM, Indurkhya N (1998) Predictive data mining: a practical guide. Morgan Kaufmann, San MateoMATHGoogle Scholar
  45. 45.
    Whitehill J, Bartlett M, Movellan J (2008) Automatic facial expression recognition for intelligent tutoring systems. In: Proc. IEEE computer society conference on computer vision and pattern recognition workshops, pp 1–6Google Scholar
  46. 46.
    Whitehill J, Bartlett M, Movellan J (2008) Measuring the perceived difficulty of a lecture using automatic facial expression recognition. In: Proc. 9th International conference on Intelligent Tutoring SystemsGoogle Scholar
  47. 47.
    Wu T, Bartlett MS, Movellan J (2010) Facial expression recognition using gabor motion energy filters. In: IEEE CVPR workshop on computer vision and pattern recognition for human communicative behavior analysisGoogle Scholar
  48. 48.
    Yabui T, Kenmochi Y, Kotani K (2003) Facial expression analysis from 3D range images; comparison with the analysis from 2D images and their integration. In: International conference on image processingGoogle Scholar
  49. 49.
    Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimed 8(3):500–507CrossRefGoogle Scholar
  50. 50.
    Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: Proc. of the 7th international conference on automatic face and gesture recognition, IEEE computer society, pp 211–216Google Scholar
  51. 51.
    Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proc. 18th international conference on pattern recognition, pp 1248–1251Google Scholar
  52. 52.
    Zeng ZH, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell PAMI 31(1):39–58CrossRefGoogle Scholar
  53. 53.
    Zhan Y, Shen D (2005) Design efficient support vector machine for fast classification. Pattern Recogn 38(1):157–161CrossRefGoogle Scholar
  54. 54.
    Zhao X, Di Huang ED, Chen L (2010) Automatic 3D facial expression recognition based on a Bayesian belief net and a statistical facial feature model. In: IEEE international conference on pattern recognitionGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringNational University of SingaporeSingaporeSingapore
  2. 2.Institute for Infocomm ResearchSingaporeSingapore

Personalised recommendations