Skip to main content
SpringerLink
Log in

Novel directional patterns and a Generalized Supervised Dimension Reduction System (GSDRS) for facial emotion recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents two novel directional patterns, a Maximum Response-based Directional Texture Pattern (MRDTP) and a Maximum Response-based Directional Number Pattern (MRDNP), for recognizing the facial emotions in constrained as well as unconstrained situations. The intensity information obtained from the maximum of the edge responses, after applying eight Kirsch masks, is used for the calculation of facial features in MRDTP. In MRDNP, instead of intensity information, the direction number of the maximum response is used. After dividing MRDNP and MRDTP code images into grids, feature vectors are created from the concatenated histograms obtained from the grids. This paper also proposes an effective Generalized Supervised Dimension Reduction System (GSDRS) and uses Extreme Learning Machine with Radial Basis Function (ELM-RBF) classifier for rapid and efficient classification of emotions. Both the proposed patterns are more effective than the existing ones in removing random noise and providing good structural information using prominent edges which help to achieve high classification accuracy when tested with seven datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014) Gabor wavelet transform based facial expression recognition using PCA and LBP. In 2014 22nd Signal Processing and Communications Applications Conference (SIU), IEEE, pp 2265–2268

  2. Agarwal S, Santra B, Mukherjee DP (2016) Anubhav: recognizing emotions through facial expression. Vis Comput. https://doi.org/10.1007/s00371-016-1323-z

  3. Ahmed F, Hossain E (2013) Automated facial expression recognition using gradient-based ternary texture patterns. Chin J Eng. https://doi.org/10.1155/2013/831747

  4. Ahmed F, Kabir MH (2012) Directional ternary pattern (DTP) for facial expression recognition. In IEEE International Conference on Consumer Electronics, pp 265–266

  5. Aifanti, N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In Proc. 11th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Desenzano, Italy, April 12–14

  6. Anisetti M, Bellandi V (2009) Emotional state inference using face related features. In New directions in intelligent interactive multimedia systems and services-2. Springer Berlin Heidelberg, pp 401–411

  7. Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1859–1866

  8. Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. In Computer Vision and Pattern Recognition Workshop, CVPRW'03, pp 53–53

  9. Baudat G, Anouar F (2000) Generalized discriminant analysis using a kernel approach. Neural Comput 12(10):2385–2404

    Article  Google Scholar 

  10. Berretti S, Amor BB, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected key points. Vis Comput 27(11):1021–1036

    Article  Google Scholar 

  11. Bhat FA, Wani MA (2016) Elastic bunch graph matching based face recognition under varying lighting, pose, and expression conditions. IAES International Journal of Artificial Intelligence (IJ-AI) 3(4):177–182

  12. Bourbakis N, Esposito A, Kavraki D (2011) Extracting and associating meta-features for understanding people’s emotional behaviour: Face and speech. Cogn Comput 3(3):436–448

    Article  Google Scholar 

  13. Calder AJ, Burton AM, Miller P, Young AW, Akamatsu S (2001) A principal component analysis of facial expressions. Vis Res 41(9):1179–1208

    Article  Google Scholar 

  14. Castillo JA, Rivera AR, Chae O ( 2012) Facial expression recognition based on local sign directional pattern. In 19th IEEE International Conference on Image Processing, pp 2613–2616

  15. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3):27

    Google Scholar 

  16. Chen J, Chen Z, Chi Z, Fu H (2014) Facial expression recognition based on facial components detection and hog features. In International Workshops on Electrical and Computer Engineering Subfields, pp 884–888

  17. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn:20273–20229

  18. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In Proc. Int. Conf Comput. Vis. Workshops, pp 2106–2112

  19. Dhall, A, Asthana A, Goecke R, Gedeon T (2011) Emotion recognition using PHOG and LPQ features. In Automatic Face & Gesture Recognition and Workshops (FG 2011), IEEE International Conference, pp 878–883

  20. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting Large, Richly Annotated Facial Expression Databases from Movies. IEEE MultiMedia 19:34–41

    Article  Google Scholar 

  21. Dhall A, Goecke R, Joshi J, Sikka K, Gedeon T (2014) Emotion recognition in the wild challenge 2014: baseline, data and protocol, ACM ICMI 2014

  22. Ekman P (2004) Emotional and conversational nonverbal signals. In Language, knowledge, and representation, Springer Netherlands, pp 39–50

  23. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124

    Article  Google Scholar 

  24. Eleftheriadis S, Rudovic O, Pantic M (2015) Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans Image Process 24(1):189–204

    Article  MathSciNet  Google Scholar 

  25. Ghimire D, Lee J, Li ZN, Jeong S (2016) Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools and Applications 15:1–26

    Google Scholar 

  26. Gupta SK (1998) Peak decomposition using Pearson type VII function. J Appl Crystallogr 31(3):474–476

    Article  Google Scholar 

  27. Haghighat M, Zonouz S, Abdel-Mottaleb M (2015) CloudID: Trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916

    Article  Google Scholar 

  28. Hamester D, Barros P, Wermter S (2015) Face expression recognition with a 2-channel Convolutional Neural Network. In 2015 International Joint Conference on Neural Networks (IJCNN), pp 1–8

  29. Hao XL, Tian M (2017) Deep belief network based on double weber local descriptor in micro-expression recognition. In Advanced Multimedia and Ubiquitous Engineering May 22. Springer, Singapore, pp 419–425

  30. Huang GB, Siew CK (2005) Extreme learning machine with randomly assigned RBF kernels. Int J Inf Technol 11(1):16–24

    Google Scholar 

  31. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. Part B: IEEE Transactions on Systems, Man, and Cybernetics 42(2):513–529

    Google Scholar 

  32. Iosifidis A, Tefas A, Pitas I (2015) On the kernel extreme learning machine classifier. Pattern Recogn Lett 54:11–17

    Article  Google Scholar 

  33. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  34. Jolliffe I (2002) Principal component analysis. Wiley, New York

    MATH  Google Scholar 

  35. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. Proceedings of Fourth IEEE International Conference in Automatic Face and Gesture Recognition, pp 46–53

  36. Kim Y, Lee H, Provost EM (2013) Deep learning for robust feature generation in audiovisual emotion recognition. In 2013 I.E. International Conference on Acoustics, Speech and Signal Processing, pp 3687–3691

  37. Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition, arXiv preprint arXiv:1703.07140

  38. Kirsch RA (1971) Computer determination of the constituent structure of biological images. Comput Biomed Res 4(3):315–328

    Article  Google Scholar 

  39. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  40. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101

  41. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. Third IEEE International Conference on Automatic Face and Gesture Recognition, pp 200–205

  42. Mavadati SM, Mahoor MH, Bartlett K, Trinh P (2012) Automatic detection of non-posed facial action units. In Image Processing (ICIP), 19th IEEE International Conference on 2012 Sep 30 IEEE, pp 1817–1820

  43. Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) Disfa: A spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160

    Article  Google Scholar 

  44. Mavadati M, Sanger P, Mahoor MH (2016) Extended DISFA Dataset: Investigating Posed and Spontaneous Facial Expressions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 1–8

  45. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In Applications of Computer Vision (WACV), 2016 I.E. Winter Conference, pp 1–10

  46. Ojansivu V, Heikkilä J (2008) Blur insensitive texture classification using local phase quantization. In International conference on image and signal processing, pp 236–243

  47. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Multimedia and Expo, ICME 2005. IEEE International Conference on Jul 6 IEEE, pp 5

  48. Rahulamathavan Y, Phan RC, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92

    Article  Google Scholar 

  49. Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  MathSciNet  MATH  Google Scholar 

  50. Rivera AR, Castillo JA, Chae O (2012) Recognition of face expressions using local principal texture pattern. In 19th IEEE International Conference on Image Processing, pp 2609–2612

  51. Rivera AR, Rojas J, Chae O (2012) Local gaussian directional pattern for face recognition. In Pattern Recognition (ICPR), 21st International Conference, pp 1000–1003

  52. Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100

    Article  Google Scholar 

  53. Shan C, Gong S, McOwan PW (2005) Appearance manifold of facial expression. In Computer Vision in Human-Computer Interaction, pp 221–230

  54. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  55. Suja P, Tripathi S, Deepthy J (2014) Emotion recognition from facial expressions using frequency domain techniques. In Advances in signal processing and intelligent recognition systems, pp 299–310

  56. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  MATH  Google Scholar 

  57. Tao J, Tan T (2005) Affective computing: A review. In Affective computing and intelligent interaction, pp 981–995

  58. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  59. Üstün B, Melssen WJ, Buydens LMC (2006) Facilitating the application of Support Vector Regression by using a universal Pearson VII function based kernel. Chemom Intell Lab Syst 81(1):29–40

    Article  Google Scholar 

  60. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect May 21, p 65

  61. Valstar MF, Pantic M (2012) Fully automatic recognition of the temporal phases of facial actions. IEEE Transactions on Systems, Man, and Cybernetics 42(1):28–43

    Article  Google Scholar 

  62. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  63. Vo A, Ly NQ (2015) Facial expression recognition using pyramid local phase quantization descriptor. In Knowledge and Systems Engineering, Springer International Publishing, pp 105–115

  64. Wang H, Huang H, Makedon F (2014) Emotion detection via discriminant laplacian embedding. Univ Access Inf Soc 13(1):23–31

    Article  Google Scholar 

  65. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition, cognitive computation, pp 1–4

  66. Wu T, Bartlett MS, Movellan JR (2010) Facial expression recognition using gabor motion energy filters. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp 42–47

  67. Xie S, Hu H (2017) Facial expression recognition with FRR-CNN. Electron Lett 53(4):235–237

    Article  Google Scholar 

  68. Yang S, Bhanu B (2012) Understanding discrete facial expressions in video using an emotion avatar image. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 42(4):980–992

    Article  Google Scholar 

  69. Yang J, Zhang D, Frangi AF, Yang JY (2004) Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–137

    Article  Google Scholar 

  70. Yang J, Frangi AF, Yang JY, Zhang D, Jin Z (2005) KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition. IEEE Trans Pattern Anal Mach Intell 27(2):230–244

    Article  Google Scholar 

  71. Zhang B, Zhang L, Zhang D, Shen L (2010) Directional binary code with application to PolyU near-infrared face database. Pattern Recogn Lett 31(14):2337–2344

    Article  Google Scholar 

  72. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

  73. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multiorientation fusion histogram. Math Probl Eng. https://doi.org/10.1155/2017/7206041

  74. Zia MS, Jaffar MA (2015) An adaptive training based on classification system for patterns in facial expressions using SURF descriptor templates. Multimedia Tools and Applications 74(11):3881–3899

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CSE Department, Regional Campus, Anna University, Tirunelveli, 627007, India

    A. Sherly Alphonse & Dejey Dharma

Authors
  1. A. Sherly Alphonse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dejey Dharma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Sherly Alphonse.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alphonse, A.S., Dharma, D. Novel directional patterns and a Generalized Supervised Dimension Reduction System (GSDRS) for facial emotion recognition. Multimed Tools Appl 77, 9455–9488 (2018). https://doi.org/10.1007/s11042-017-5141-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5141-8

Keywords

Search

Navigation