Multimedia Tools and Applications

, Volume 76, Issue 6, pp 7921–7946 | Cite as

Recognition of facial expressions based on salient geometric features and support vector machines

  • Deepak Ghimire
  • Joonwhoan LeeEmail author
  • Ze-Nian Li
  • Sunghwan Jeong


Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.


Facial points Geometric features AdaBoost Extreme learning machine Support vector machines Facial expression recognitions 


  1. 1.
    Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–188CrossRefGoogle Scholar
  2. 2.
    Aifanti N, Papachristou A, Delpoulos A (2010) The MUG facial expression database. In Proceeding of 11th international workshop on image analysis for multimedia interactive services, pp 1–4Google Scholar
  3. 3.
    Asthana A, Saragih J, Wagner M, Goecke R (2009) Evaluating AAM fitting methods for facial expression recognition. In Proceeding of the international conference on affective computing and intelligent interaction, pp 1–8Google Scholar
  4. 4.
    Blome DS (2003) Elastic bunch graph matching. M.Sc. Thesis, Colorado State University: Fort Collins, CO, USAGoogle Scholar
  5. 5.
    Bouguet JY (1999) Pyramidal implementation of the Lucas-Kanade feature tracker. Technological Report, Intel Corporation, Microprocessor Research LabGoogle Scholar
  6. 6.
    Chang Y, Hu C, Feris R, Turk M (2006) Manifold based analysis of facial expression. Image Vis Comput 24:605–614CrossRefGoogle Scholar
  7. 7.
    Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines. Available online: Accessed on 16 Jan 2015
  8. 8.
    Choi HC, Oh SY (2006) Realtime facial expression recognition using active appearance model and multilayer perceptron. In Proceeding of the international joint conference SICE-ICASE, pp 5924–5927Google Scholar
  9. 9.
    Cid F, Moreno J, Bustos P, Nunez P (2014) Muecas: a multi-sensor robotic head for affective human robot interaction and imitation. Sensors 14:7711–7737CrossRefGoogle Scholar
  10. 10.
    Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans Affect Comput 5:418–431CrossRefGoogle Scholar
  11. 11.
    Ekman P (1994) Strong evidence of universal in facial expressions: a reply to Russell’s mistaken critique. Psychol Bull 115:268–287CrossRefGoogle Scholar
  12. 12.
    Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275CrossRefzbMATHGoogle Scholar
  13. 13.
    Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13:7714–7734CrossRefGoogle Scholar
  15. 15.
    Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10:443–458CrossRefGoogle Scholar
  16. 16.
    Hsu CW, Chang CC, Lin CJ (2010) A practical guide to support vector classification. Technical Report. Department of Computer Science, National Taiwan University, TaiwanGoogle Scholar
  17. 17.
    Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501CrossRefGoogle Scholar
  18. 18.
    Kotisa I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1033–1046Google Scholar
  19. 19.
    Kotisa I, Pitas I (2007) Facial expression recognition in image sequence using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187MathSciNetCrossRefGoogle Scholar
  20. 20.
    Li Y, Wang S, Zhao Y, Ji Q (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22:2559–2573CrossRefGoogle Scholar
  21. 21.
    Liu W, Lu J, Wang Z, Song H (2008) An expression space model for facial expression analysis. In Proceeding of IEEE congress in image and signal processing, pp 680–684Google Scholar
  22. 22.
    Mehrabian A (1968) Communication without words. Psychol Today 2:53–56Google Scholar
  23. 23.
    Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115:541–558CrossRefGoogle Scholar
  24. 24.
    Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424–1445CrossRefGoogle Scholar
  25. 25.
    Pantic M, Valster R, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Proceeding of IEEE international conference multimedia and expo, pp. 317–321Google Scholar
  26. 26.
    Pantic M, Valster M, Rademaker R, Maat L (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specific expression. In Proceeding of 3rd IEEE workshop on CVPR for human communication behavior analysis, pp. 94–101Google Scholar
  27. 27.
    Rahulmathavan Y, Phan RCW, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4:83–92CrossRefGoogle Scholar
  28. 28.
    Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369Google Scholar
  29. 29.
    Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact 2014:1–13CrossRefGoogle Scholar
  30. 30.
    Samal A, Iyenger PA (1994) Automatic recognition of human face and facial expressions: a survey. Pattern Recogn 25:65–77CrossRefGoogle Scholar
  31. 31.
    Schels M, Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequence utilizing GMM Supervectors. In Proceeding of the 2010 20th international conference on pattern recognition, pp 4251–4254Google Scholar
  32. 32.
    Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816CrossRefGoogle Scholar
  33. 33.
    Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713CrossRefGoogle Scholar
  34. 34.
    Soyel H, Demirel H (2012) Localized discriminative scale invariant feature transform based facial expression recognition. Comput Electr Eng 38:1299–1309CrossRefGoogle Scholar
  35. 35.
    Sung J, Kin D (2009) Real-time facial expression recognition using STAAM and layered GDA classifiers. Image Vis Comput 27:1313–1325CrossRefGoogle Scholar
  36. 36.
    Tian Y-L, Kanade T, Cohn JF (2005) Handbook of face recognition. Springer, Berlin, pp 247–275CrossRefGoogle Scholar
  37. 37.
    Uddin M, Lee J, Kim T (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55:2216–2224CrossRefGoogle Scholar
  38. 38.
    Uhls YT, Michikyan M, Morris J, Garcia D, Small GW, Zgourou E, Greenfield PM (2014) Five days at outdoor education camp without screen improves pattern skills with nonverbal emotion cues. Comput Hum Behav 39:387–392CrossRefGoogle Scholar
  39. 39.
    Valster MF, Mehu M, Jiang B, Pantic M, Scherer K (2012) Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern B Cybern 42:966–979CrossRefGoogle Scholar
  40. 40.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154CrossRefGoogle Scholar
  41. 41.
    Wiskott L, Fellous JM, Krüger N (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19:775–779CrossRefGoogle Scholar
  42. 42.
    Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31:210–227CrossRefGoogle Scholar
  43. 43.
    Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508CrossRefGoogle Scholar
  44. 44.
    Zafeirius S, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10:1528–1540CrossRefGoogle Scholar
  45. 45.
    Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31:39–58CrossRefGoogle Scholar
  46. 46.
    Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761CrossRefGoogle Scholar
  47. 47.
    Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915–928CrossRefGoogle Scholar
  48. 48.
    Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11:9573–9588MathSciNetCrossRefGoogle Scholar
  49. 49.
    Zhi R, Flierl M, Ruan Q, Kleijn WB (2011) Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans Syst Man Cybern B Cybern 41:38–52CrossRefGoogle Scholar
  50. 50.
    Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class AdaBoost. Stat Its Interface 2:349–360MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Deepak Ghimire
    • 1
  • Joonwhoan Lee
    • 2
    Email author
  • Ze-Nian Li
    • 3
  • Sunghwan Jeong
    • 1
  1. 1.Korea Electronics Technology InstituteJeonju-siRepublic of Korea
  2. 2.Division of Computer EngineeringChonbuk National UniversityJeonju-siRepublic of Korea
  3. 3.School of Computing ScienceSimon Fraser UniversityBurnabyCanada

Personalised recommendations