Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition

Abstract

Facial expression recognition (FER) is an essential part of effective human–computer interaction and serves as a helpful medium for children and patients who have problems with communication. However, most of the previous studies focus on building a FER model based on supervised and unsupervised approaches. This paper is focused on a semi-supervised deep belief network (DBN) approach to predict the facial expressions from the CK+, Oulu CASIA, MMI, and JAFFE datasets. To achieve accurate classification of the facial expressions, a gravitational search algorithm (GSA) is applied to optimize some parameters in the DBN network. The Histogram oriented gradients (HOG) and 2D-Discrete Wavelet Transform (2D-DWT) are used for feature extraction from the lip, cheek, brow, eye, and furrow patches. The unwanted information present in the image is eliminated using a feature selection approach. The feature extraction is done by the Kernel-principal component analysis to obtain higher-order correlations between input variables and detect non-linear components. The HOG features extracted from the lip patch provides the best performance for accurate facial expression classification. Finally, a comparative analysis to compare the proposed model with different machine learning techniques based on the evaluation criteria. The results demonstrate that DBN-GSA based classifier is more accurate than the rest of the classifiers.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Abbreviations

DBN:

Deep belief network

FER:

Facial expression recognition

GSA:

Gravitational search algorithm

HOG:

Histogram oriented gradients

2D-DWT:

2D-Discrete Wavelet transform

DR:

Dimensionality reduction

SVM:

Support-vector machines

DBTR-BR:

Deep convolutional neural network, discrete Beamlet transform regularization-blind restoration

MSCNN-PHRNN:

Multi-signal convolutional neural network part-based hierarchical bidirectional neural network

CK+:

Cohn Kanade

MDSTFN:

Multi-channel deep spatial–temporal feature neural network

MMI:

Man–machine interaction

PHRNN:

Part-based hierarchical bidirectional neural network

MSCNN:

Multi-signal convolutional neural network

Oulu-CASIA:

Oulu-Chinese Academy of Science Institute of Automation

SMO:

Spider monkey optimization

DCT:

Discrete Cosine transform

DISFA+:

Extended denver intensity of spontaneous facial actions

EEG:

Electroencephalography

RBM:

Restricted Boltzmann machine

GO:

Gradient and orientation

LPF:

Low pass filter

HPF:

High pass filter

PCA:

Principal component analysis

(DR):

Dimensionality reduction

(CD):

Hinton contrast divergence

(LLE):

Locally linear embedding

(t-SNE):

t-Distributed stochastic neighbor embedding

(ROC):

Receiver operating characteristic curve

(AUC):

Area under the ROC curve

References

  1. Cheng F, Yu J, Xiong H (2010) Facial expression recognition in JAFFE dataset based on Gaussian process classification. IEEE Trans Neural Netw 21(10):1685–1690

    Article  Google Scholar 

  2. Dahmouni A, Aharrane N, Moutaouakil KE, Satori K (2016) A new hybrid face recognition system via local gradient probabilistic pattern (LGPP) and 2D-DWT. In: Advances in intelligent systems and computing Europe and MENA cooperation advances in information and communication technologies, pp 269–278

  3. Fan X, Tjahjadi T (2019) Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J Vis Commun Image Represent 65:102659

    Article  Google Scholar 

  4. Guha T, Yang Z, Grossman RB, Narayanan SS (2018) A computational study of expressive facial dynamics in children with autism. IEEE Trans Affect Comput 9(1):14–20

    Article  Google Scholar 

  5. Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  6. Hinton GE, Osindero S, Teh YW (2006) A Fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    MathSciNet  Article  Google Scholar 

  7. Kurup AR, Ajith M, Ramón MM (2019) Semi-supervised facial expression recognition using reduced spatial features and deep belief networks. Neurocomputing 367:188–197

    Article  Google Scholar 

  8. Lanihun O, Tiddeman B, Tuci E, Shaw P (2015) Improving active vision system categorization capability through histogram of oriented gradients. In: Dixon C, Tuyls K (eds) Towards autonomous robotic systems, TAROS 2015, vol 9287. Lecture Notes in Computer Science. Springer, Cham

    Google Scholar 

  9. Liu Q, Liu H (2020) Criminal psychological emotion recognition based on deep learning and EEG signals. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05024-0

    Article  Google Scholar 

  10. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE

  11. Lv Y, Feng Z, Xu C (2014) Facial expression recognition via deep learning. In: 2014 international conference on smart computing

  12. Ly TS, Do NT, Kim SH, Yang HJ, Lee GS (2019) A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition. Image Vis Comput 92:103817

    Article  Google Scholar 

  13. McDuff D, Kaliouby RE, Picard RW (2012) Crowdsourcing facial responses to online videos. IEEE Trans Affect Comput 3(4):456–468

    Article  Google Scholar 

  14. McDuff D, Kaliouby RE, Cohn JF, Picard RW (2015) Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. IEEE Trans Affect Comput 6(3):223–235

    Article  Google Scholar 

  15. Minaee S, Abdolrashidi A (2019) Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019

  16. Palm G, Glodek M (2013) Towards emotion recognition in human computer interaction. In: Neural nets and surroundings. Springer, Berlin, Heidelberg, pp 323–336

    Google Scholar 

  17. Pantic M, Rothkrantz LJM (2003) Toward an affect-sensitive multimodal human-computer interaction. Proc IEEE 91(9):1370–1390

    Article  Google Scholar 

  18. Rashedi E, Nezamabadi-Pour H, Saryazdi S (2009) GSA: a gravitational search algorithm. Inf Sci 179(13):2232–2248

    Article  Google Scholar 

  19. Reddy GV, Savarni CD, Mukherjee S (2020) Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features. Cognit Syst Res 62:23–34

    Article  Google Scholar 

  20. Rejeesh MR (2019) Interest point based face recognition using adaptive neuro fuzzy inference system. Multimed Tools Appl 78(16):22691–22710

    Article  Google Scholar 

  21. Schölkopf B, Smola A, Müller KR (1997) Kernel principal component analysis In: Lecture notes in computer science artificial neural networks—ICANN97, pp 583–588

  22. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  23. Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recognit Lett 119:49–61

    Article  Google Scholar 

  24. Sundararaj V (2016) An efficient threshold prediction scheme for wavelet based ECG signal noise reduction using variable step size firefly algorithm. Int J Intell Eng Syst 9(3):117–126

    Google Scholar 

  25. Sundararaj V (2019) Optimised denoising scheme via opposition-based self-adaptive learning PSO algorithm for wavelet-based ECG signal noise reduction. Int J Biomed Eng Technol 31(4):325

    Article  Google Scholar 

  26. Sundararaj V, Muthukumar S, Kumar RS (2018) An optimal cluster formation based energy efficient dynamic scheduling hybrid MAC protocol for heavy traffic load in wireless sensor networks. Comput Secur 77:277–288

    Article  Google Scholar 

  27. Tian Y, Chen S (2012) Understanding effects of image resolution for facial expression analysis. J Comput Vis Image Process

  28. Tieleman T, Hinton G (2009) Using fast weights to improve persistent contrastive divergence. In: Proceedings of the 26th annual international conference on machine learning—ICML 09

  29. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of the 3rd international workshop on EMOTION (satellite of LREC): corpora for research on emotion and affect

  30. Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimedia Tools Appl. https://doi.org/10.1007/s11042-020-08901-x

    Article  Google Scholar 

  31. Vinu S (2019) Optimal task assignment in mobile cloud computing by queue based ant-bee algorithm. Wirel Pers Commun 104(1):173–197

    Article  Google Scholar 

  32. Wu H, Liu Y, Liu Y, Liu S (2019) Efficient facial expression recognition via convolution neural network and infrared imaging technology. Infrared Phys Technol 102:103031

    Article  Google Scholar 

  33. Xie W, Jia X, Shen L, Yang M (2019) Sparse deep feature learning for facial expression recognition. Pattern Recognit 96:106966

    Article  Google Scholar 

  34. Yang S, Kafai M, An L, Bhanu B (2014) Zapping index: using smile to measure advertisement zapping likelihood. IEEE Trans Affect Comput 5(4):432–444

    Article  Google Scholar 

  35. Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  36. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

    MathSciNet  Article  Google Scholar 

  37. Zhao Y, Ju YF (2018) A family of efficient appearance models based on histogram of oriented gradients (HOG), color histogram and their fusion for human pose estimation. In: Proceedings of the fifth Euro-China conference on intelligent data analysis and applications advances in intelligent systems and computing, pp 842–850

  38. Zhao XM, Shi XG, Zhang SQ (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):347–355

    Article  Google Scholar 

Download references

Acknowledgment

The Authors extend their warm thanks to Deanship of Scientific Research (DSR), King Saud University, Riyadh, Saudi Arabia for permitting us to carry out the research and also for acknowledging the financial aid from DSR through the Project Group No. RG-1441-343.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Wael Mohammad Alenazy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Alenazy, W.M., Alqahtani, A.S. Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. J Ambient Intell Human Comput (2020). https://doi.org/10.1007/s12652-020-02235-0

Download citation

Keywords

  • Facial expression recognition
  • Deep belief network
  • Semi-supervised approach
  • Gravitational search algorithm
  • Histogram oriented gradients