Human vision inspired feature extraction for facial expression recognition

  • Hamid Sadeghi
  • Abolghasem-A. RaieEmail author


Facial expression is a powerful way for human emotional communications. According to various applications, automatic facial expression recognition becomes an interesting problem for researchers in different areas. An automatic facial expression recognition system recognizes the expressed emotion in input facial image using several processing stages. Feature extraction is a vital step in facial expression recognition. One of the most widely used techniques for feature extraction in machine vision is utilizing Gabor filter which is sensitive to lines at various orientations. The disadvantage of Gabor filters is their computational cost and large feature vector length. Inspiring the human vision system and stimulations of complex cells, in this paper, firstly, facial image is convolved with Gabor filters. Then, the achieved convolution matrices are properly coded based on the maximum and minimum responses. Finally, the feature vector is obtained by calculating the histogram of these codes. The length of achieved histogram for 16 and 8 Gabor filters are 240 and 56, respectively, which is considerably less than keeping all Gabor responses. The proposed method is (person-independently) evaluated on four facial expression recognition datasets including CK+, SFEW, MMI, and RAF-DB. The experimental results show that the proposed method outperforms existing image texture descriptors in facial expression recognition in both controlled and uncontrolled images.


Facial expression recognition Complex cell Human vision system Gabor filter Image texture description 



This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.


  1. 1.
    Acevedo D, Negri P, Buemi ME, Fernández FG, Mejail M (2017) A simple geometric-based descriptor for facial expression recognition. In: Automatic face & gesture recognition (FG 2017), 2017 12th IEEE international conference on. IEEE, pp 802–808Google Scholar
  2. 2.
    Bashyal S, Venayagamoorthy GK (2008) Recognition of facial expression using Gabor wavelets and learning vector quantization. Eng Appl Artif Intell 21:1056–1064CrossRefGoogle Scholar
  3. 3.
    Ben X, Jia X, Yan R, Zhang X, Meng W (2018) Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn Lett 107:50–58CrossRefGoogle Scholar
  4. 4.
    Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):1–27. Software available at CrossRefGoogle Scholar
  5. 5.
    Cruz EAS, Jung CR, Franco CHE (2017) Facial expression recognition using temporal POEM features. Pattern Recogn Lett, In PressGoogle Scholar
  6. 6.
    Daugmen J (1988) Complete discrete 2d Gabor transforms by neutral networks for image analysis and compression. IEEE Trans Acoust Speech Signal Process 36(7):1169–1179CrossRefGoogle Scholar
  7. 7.
    Dhall A, Ramana Murthy OV, Goecke R, Joshi J, Gedeon T (2015) Video and image based emotion recognition challenges in the wild: Emotiw 2015. In: Proceedings of the 2015 ACM on int. conf. multimodal interaction. ACM, pp 423–426Google Scholar
  8. 8.
    Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38(3):518–531CrossRefGoogle Scholar
  9. 9.
    Duan Y, Lu J, Feng J, Zhou J (2018) Context-aware local binary feature learning for face recognition. IEEE Trans Pattern Anal Mach Intell 40(5):1139–1153CrossRefGoogle Scholar
  10. 10.
    Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129CrossRefGoogle Scholar
  11. 11.
    Eleftheriadis S, Rudovic O, Pantic M (2015) Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans Image Process 24(1):189–204MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Frijda NH, Tcherkassof A (1997) Facial expressions as modes of action readiness. In: the psychology of facial expression, 2nd edn, pp 78–102Google Scholar
  13. 13.
    Gabbiani F, Cox SJ (2017) Mathematics for neuroscientists. Chapter 23: models of simple and complex cells. Academic, New YorkzbMATHGoogle Scholar
  14. 14.
    Ghimire D, Lee J, Li ZN, Jeong S (2017) Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76(6):7921–7946CrossRefGoogle Scholar
  15. 15.
    Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821CrossRefGoogle Scholar
  16. 16.
    Girard JM, Cohn JF, Mahoor MH, Mavadati SM, Hammal Z, Rosenwald DP (2014) Nonverbal social withdrawal in depression: evidence from manual and automatic analysis. Image Vis Comput 32(10):641–647CrossRefGoogle Scholar
  17. 17.
    Gu W, Xiang C, Venkatesh YV, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45:80–91CrossRefGoogle Scholar
  18. 18.
    Guo M, Hou X, Ma Y, Wu X (2017) Facial expression recognition using ELBP based on covariance matrix transform in KLT. Multimed Tools Appl 76(2):2995–3010CrossRefGoogle Scholar
  19. 19.
    Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12CrossRefGoogle Scholar
  20. 20.
    Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2016) Effective semantic features for facial expressions recognition using SVM. Multimed Tools Appl 75(11):6663–6682CrossRefGoogle Scholar
  21. 21.
    Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Multimed Tools Appl 73(1):309–326CrossRefGoogle Scholar
  22. 22.
    Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41(6):765–781CrossRefGoogle Scholar
  23. 23.
    Irani R, Nasrollahi K, Simon MO, Corneanu CA, Escalera S, Bahnsen C, Lundtoft DH, Moeslund TB, Pedersen TL, Klitgaard M-L et al (2015) Spatiotemporal analysis of rgb-dt facial images for multimodal pain level recognition. In: CVPR WorkshopsGoogle Scholar
  24. 24.
    Jia X, Ben X, Yuan H, Kpalma K, Meng W (2018) Macro-to-micro transformation model for micro-expression recognition. J Comput Sci-Neth 25:289–297CrossRefGoogle Scholar
  25. 25.
    Joshi J, Dhall A, Goecke R, Breakspear M, Parker G (2012) Neural-net classification for spatio-temporal descriptor based depression analysis. In: Pattern recognition (ICPR), 2012 21st international conference on. IEEE, pp 2634–2638Google Scholar
  26. 26.
    Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. ISVC, pp 368–377Google Scholar
  27. 27.
    Kandel ER, Schwartz JH, Jessell TM, Siegelbaum SA, Hudspeth AJ (2013) Principles of neural science, Fifth edn. McGraw-hill, New YorkGoogle Scholar
  28. 28.
    Kazemi V, Josephine S (2014) One millisecond face alignment with an ensemble of regression trees. In: Computer vision and pattern recognition (CVPR), 2014 IEEE conference on. IEEEGoogle Scholar
  29. 29.
    Kobayashi H, Hara F (1992) Recognition of six basic facial expression and their strength by neural network. In: Proc. int’l workshop robot and human comm, pp 381–386Google Scholar
  30. 30.
    Kobayashi H, Hara F (1992) Recognition of mixed facial expression by neural network. In: Proc. int’l workshop robot and human comm, pp 387–391Google Scholar
  31. 31.
    Lei Y, Zhao X, Wang G, Yu K, Guo W (2017) A novel approach for cirrhosis recognition via improved LBP algorithm and dictionary learning. Biomed Signal Proces 38:281–292CrossRefGoogle Scholar
  32. 32.
    Levenson RW, Ekman P, Friesen WV (1990) Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology 27(4):363–384CrossRefGoogle Scholar
  33. 33.
    Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861Google Scholar
  34. 34.
    Li R, Tian J, Chua MCH (2018) Facial expression classification using salient pattern driven integrated geometric and textual features. Multimed Tools Appl:1–13Google Scholar
  35. 35.
    Linton O (2017) Probability, statistics and econometrics. Chapter 12: hypothesis testing. Academic, LondonGoogle Scholar
  36. 36.
    Liu M, Shan S, Wang R, Chen X (2014) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: IEEE Conf. Comp. Vis. Patt. Rec., pp 1749–1756Google Scholar
  37. 37.
    Liu Z, Shan L, Weihong D (2017) Boosting-POOF: boosting part based one vs one feature for facial expression recognition in the wild. In: Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE international conference on. IEEE, pp 967–972Google Scholar
  38. 38.
    Lu J, Liong VE, Zhou X, Zhou J (2015) Learning compact binary face descriptor for face recognition. IEEE Trans Pattern Anal Mach Intell 37(10):2041–2056CrossRefGoogle Scholar
  39. 39.
    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the third international workshop on CVPR for human communicative behavior analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101Google Scholar
  40. 40.
    Lucey P, Cohn JF, Matthews I, Lucey S, Sridharan S, Howlett J, Prkachin KM (2011) Automatically detecting pain in video through facial action units. IEEE Trans Syst Man Cybern B Cybern 41(3):664–674CrossRefGoogle Scholar
  41. 41.
    Majumdern A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recogn 47(3):1282–1293CrossRefGoogle Scholar
  42. 42.
    Moreno P, Bernardino A, Santos-Victor J Gabor parameter selection for local feature detection. In: Iberian conference on pattern recognition and image analysis 2005 Jun 7. Springer, Berlin, pp 11–19Google Scholar
  43. 43.
    Niedenthal PM (2007) Embodying emotion. Science 116:1002–1005CrossRefGoogle Scholar
  44. 44.
    Nigam S, Singh R, Misra AK (2018) Efficient facial expression recognition using histogram of oriented gradients in wavelet domain. Multimed Tools Appl:1–23Google Scholar
  45. 45.
    Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distribution. Pattern Recogn 29(1):51–59CrossRefGoogle Scholar
  46. 46.
    Pai N-S, Chang S-P (2011) An embedded system for real-time facial expression recognition based on the extension theory. Comput Math Appl 61:2101–2106CrossRefzbMATHGoogle Scholar
  47. 47.
    Palestra G, Pettinicchio A, Coco MD, Carcagnì P, Leo M, Distante C (2015) Improved performance in facial expression recognition using 32 geometric features. In: Image Analysis and Processing, vol 9280. Springer, pp 518–528Google Scholar
  48. 48.
    Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo (ICME)Google Scholar
  49. 49.
    Rivera AR, Castillo JR, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Roger Highfield RW, Jenkins R (2009) How your looks betray your personality. New ScientistGoogle Scholar
  51. 51.
    Sadeghi H, Raie AA (2013) Semi-dynamic facial expression recognition based on masked displacement image. In: International symposium on artificial intelligence and signal processing. Springer, Cham, pp 100–108Google Scholar
  52. 52.
    Scherer S, Stratou G, Mahmoud M, Boberg J, Gratch J, Rizzo A, Morency L-P (2013) Automatic behavior descriptors for psychological disorder analysis. In: Automatic face and gesture recognition (FG), 2013 10th IEEE international conference and workshops on. IEEE, pp 1–8Google Scholar
  53. 53.
    Shams L, Von Der Malsburg C (2002) The role of complex cells in object recognition. Vis Res 42(22):2547–2554CrossRefGoogle Scholar
  54. 54.
    Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816CrossRefGoogle Scholar
  55. 55.
    Shu C, Ding X, Fang C (2011) Histogram of the oriented gradient for face recognition. Tsinghua Sci Technol 16(2):216–224CrossRefGoogle Scholar
  56. 56.
    Tian Y, Kanade T, Cohn J (2005) Handbook of face recognition. Springer (Chapter 11. Facial Expression Analysis).Google Scholar
  57. 57.
    Valstar MF, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd intern. Workshop on EMOTION (satellite of LREC): corpora for research on emotion and affect, Valletta, Malta. ELRAGoogle Scholar
  58. 58.
    Vu NS, Caplier A (2012) Enhanced patterns of oriented edge magnitudes for face recognition and image matching. IEEE Trans Image Process 21(3):1352–1365MathSciNetCrossRefzbMATHGoogle Scholar
  59. 59.
    Xiao R, Zhao Q, Zhang D, Shi P (2011) Facial expression recognition on multiple manifolds. Pattern Recogn 44:107–116CrossRefzbMATHGoogle Scholar
  60. 60.
    Yang B, Cao JM, Jiang DP, Lv JD (2017) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tools Appl:1–23Google Scholar
  61. 61.
    Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229CrossRefGoogle Scholar
  62. 62.
    Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928CrossRefGoogle Scholar
  63. 63.
    Zhao G, Pietikäinen M (2009) Boosted multi-resolution spatiotemporal descriptors for facial. Pattern Recogn Lett 13:1117–1127CrossRefGoogle Scholar
  64. 64.
    Zong Y et al (2016) Emotion recognition in the wild via sparse transductive transfer linear discriminant analysis. J Multimodal User In 10(2):163–172CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Electrical Engineering DepartmentAmirkabir University of TechnologyTehranIran

Personalised recommendations