Skip to main content
SpringerLink
Log in

Utilizing 3D flow of points for facial expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents an approach to recognize Facial Expressions of different intensities using 3D flow of facial points. 3D flow is the geometrical displacement (in 3D) of a facial point from its position in a neutral face to that in the expressive face. Experiments are performed on 3D face models from the BU-3DFE database. Four different intensities of expressions are used for analyzing the relevance of intensity of the expression for the task of FER. It was observed that high intensity expressions are easier to recognize and there is a need to develop algorithms for recognizing low intensity facial expressions. The proposed features outperform difference of facial distances and 2D optical flow. Performances of two classifiers, SVM and LDA are compared wherein SVM performs better. Feature selection did not prove useful.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: development and applications to human computer interaction. In: CVPR workshop on computer vision and pattern recognition for human-computer interaction

  2. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial expression: machine learning and application to spontaneous behavior. In: IEEE computer society conference on computer vision and pattern recognition

  3. Bartlett MS, Littlewort G, Frank MG, Lainscsek C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. J Multimed 1(6):22–35

    Article  Google Scholar 

  4. Berretti S, Bimbo AD, Amor PPBB, Daoudi M (2010) A set of selected SIFT features for 3D facial expression recognition. In: Proc. 20th international conference on pattern recognition, pp 4125–4128

  5. Chandrasiri NP, Naemura T, Ishizuka M, Harashima H (2004) Internet communication using real-time facial expression analysis and synthesis. IEEE Multimed 11(3):20–29

    Article  Google Scholar 

  6. Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines

  7. Chang Y, Vieira M, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: Analysis and modeling of faces and gestures: second international workshop

  8. Datchu D, Rothkrantz LJM (2005) Facial expression recognition with relevance vector machines. In: Proc. IEEE international conference on multimedia and expo

  9. Dimensional Imaging (2011) Retrieved October 8, 2011. http://www.di3d.com/

  10. Dornaika F, Davoine F (2008) Simultaneous facial action tracking and expression recognition in the presence of head motion. Int J Comput Vis 76(3):257–281

    Article  Google Scholar 

  11. Dornaika F, Raducanu B (2008) Facial expression recognition for HCI applications. Encycl Artif Intell 2:625–631

    Article  Google Scholar 

  12. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129

    Article  Google Scholar 

  13. Ekman P, Friesen WV, Hager JC (1978) Facial action coding system. Consulting Psychologists Press, Palo Alto, CA

    Google Scholar 

  14. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognit 36(1):259–275

    Article  MATH  Google Scholar 

  15. Fasel B, Monay F, Gatica-Perez D (2004) Latent semantic analysis of facial action codes for automatic facial expression recognition. In: Proc. of the 6th ACM SIGMM international workshop on Multimedia information retrieval ACM New York, NY, USA, pp 181–188

    Google Scholar 

  16. Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3–4):143–166

    Article  MATH  Google Scholar 

  17. Gokturk S, Bouguet J, Tomasi C, Girod B (2002) Model-based face tracking for view independent facial expression recognition. In: Proceedings of the fifth IEEE international conference on automatic face and gesture recognition

  18. Gong B, Wang Y, Liu J, Tang X (2009) Automatic facial expression recognition on a single 3D face by exploring shape deformation. In: Proc. 17th ACM international conference on multimedia, pp 569–572

  19. Hahnel M, Wiratanaya A, Kraiss K (2006) Facial expression modeling from still images using a single generic 3d head model. Lecture notes in computer science, vol 4174, pp 324–333

    Article  Google Scholar 

  20. Hsu C, Chang C, Lin C (2003) A practical guide to support vector classification.

  21. Kinect (2011) Retrieved October 10, 2011. http://www.xbox.com/en-US/kinect

  22. Lin D, Yan S, Tang X (2005) Comparative study: face recognition on unspecific persons using linear subspace methods. In: Proc. IEEE international conference on image processing

  23. Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2010) Local 3d shape analysis for facial expression recognition. In: Proc. IEEE international conference on pattern recognition

  24. Mpiperis I, Malassiotis S, Petridis V, Strintzis M (2008) 3D facial expression recognition using swarm intelligence. In: IEEE international conference on acoustics, speech and signal processing, pp 2133–2136

  25. Mpiperis I, Malassiotis S, Strintzis M (2008) Bilinear models for 3-D face and facial expression recognition. IEEE Trans Inf Forensics Secur 3(3):498–511

    Article  Google Scholar 

  26. Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell PAMI 22(12):1424–1445

    Article  Google Scholar 

  27. Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: Proc. 2nd IEEE international conference on biometrics: theory, applications and systems

  28. Sha T, Song M, Bu J, Chen C, Tao D (2011) Feature level analysis for 3D facial expression recognition. Neurocomputing 74(12–13):2135–2141

    Article  Google Scholar 

  29. Soyel H, Demirel H (2007) Facial expression recognition using 3d facial feature distances. In: ICIAR. Lecture Notes in computer science, vol. 4633. Springer Berlin / Heidelberg, pp 831–838

    Google Scholar 

  30. Soyel H, Demirel H (2010) Optimal feature selection for 3D facial expression recognition using coarse-to-fine classification. Turkish J. Electr Eng Comput Sci 18(6):1031–1040

    Google Scholar 

  31. Srivastava R, Roy S (2009) 3D facial expression recognition using residues. In: Proc. IEEE region 10 conference, TENCON

  32. Srivastava R, Sim T, Yan S, Ranganath S (2010) Feature selection for facial expression recognition using deformation modeling. In: Proc. international conference on digital image processing

  33. Srivastava R, Roy S, Yan S, Sim T (2011) Multi-actor emotion recognition in movies using a bimodal approach. In: Proc. 17th international conference on multimedia modeling

  34. Stylianou G, Lanitis A (2009) Image based 3D face reconstruction: a survey. Int J Image Graph 9(2):217–250

    Article  Google Scholar 

  35. Sun Y, Yin L (2008) 3D Spatio-Temporal face recognition using dynamic range model sequences. In: Proceedings IEEE computer society conference on computer vision and pattern recognition workshops

  36. Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Proceedings of the 10th European conference on computer vision

  37. Sun Y, Chen X, Rosato M, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Part A Syst Humans 40(3):461–474

    Article  Google Scholar 

  38. Suwa M, Sugie N, Fujimora K (1978) A preliminary note on pattern recognition of human emotional expression. In: International joint conference on pattern recognition, pp 408–410

  39. Tang H, Huang T (2008) 3D facial expression recognition based on automatically selected features. In: Proc. computer vision and pattern recognition workshops, pp 1–8

  40. Tang H, Huang T (2008) 3D facial expression recognition based on properties of line segments connecting facial feature points. In: Proc. 8th IEEE international conference on automatic face and gesture recognition, pp 1–6

  41. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  42. Venkatesh Y, Kassim A, Murthy O (2009) A novel approach to classification of facial expressions from 3D-mesh datasets using modified PCA. Pattern Recogn Lett 30(12):1128–1137

    Article  Google Scholar 

  43. Wang J, Yin L, Wei X, Sun Y (2006) 3d facial expression recognition based on primitive surface feature distribution In: Proc. IEEE international conference on computer vision and pattern recognition

  44. Weiss SM, Indurkhya N (1998) Predictive data mining: a practical guide. Morgan Kaufmann, San Mateo

    MATH  Google Scholar 

  45. Whitehill J, Bartlett M, Movellan J (2008) Automatic facial expression recognition for intelligent tutoring systems. In: Proc. IEEE computer society conference on computer vision and pattern recognition workshops, pp 1–6

  46. Whitehill J, Bartlett M, Movellan J (2008) Measuring the perceived difficulty of a lecture using automatic facial expression recognition. In: Proc. 9th International conference on Intelligent Tutoring Systems

  47. Wu T, Bartlett MS, Movellan J (2010) Facial expression recognition using gabor motion energy filters. In: IEEE CVPR workshop on computer vision and pattern recognition for human communicative behavior analysis

  48. Yabui T, Kenmochi Y, Kotani K (2003) Facial expression analysis from 3D range images; comparison with the analysis from 2D images and their integration. In: International conference on image processing

  49. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimed 8(3):500–507

    Article  Google Scholar 

  50. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: Proc. of the 7th international conference on automatic face and gesture recognition, IEEE computer society, pp 211–216

  51. Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proc. 18th international conference on pattern recognition, pp 1248–1251

  52. Zeng ZH, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell PAMI 31(1):39–58

    Article  Google Scholar 

  53. Zhan Y, Shen D (2005) Design efficient support vector machine for fast classification. Pattern Recogn 38(1):157–161

    Article  Google Scholar 

  54. Zhao X, Di Huang ED, Chen L (2010) Automatic 3D facial expression recognition based on a Bayesian belief net and a statistical facial feature model. In: IEEE international conference on pattern recognition

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 117576, Singapore

    Ruchir Srivastava

  2. Institute for Infocomm Research, 1 Fusionopolis Way, #21-01 Connexis, Singapore, 138632, Singapore

    Sujoy Roy

Authors
  1. Ruchir Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sujoy Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruchir Srivastava.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Srivastava, R., Roy, S. Utilizing 3D flow of points for facial expression recognition. Multimed Tools Appl 71, 1953–1974 (2014). https://doi.org/10.1007/s11042-012-1322-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-012-1322-7

Keywords

Search

Navigation