To address the variabilities of the number and position of extracted feature points for the traditional scale-invariant feature transform (SIFT) method, an improved SIFT algorithm is proposed for robust emotion recognition. Specifically, shape decomposition is first performed on the detected facial images by defining a weight vector. Then, a feature point constraint algorithm is developed to determine the optimum position of the feature points that can effectively represent the expression change regions. On this basis, the SIFT descriptors are applied to extract the regional gradient information as feature parameters. Finally, the support vector machine classifier combined with the principal component analysis method is used to reduce the feature dimensions and facial expression recognition. Experiments have been performed under different conditions, i.e., varied illuminations, face poses and facial moisture levels, using 15 participants. In the cases of frontal face and 5-degree face rotation views, the average recognition accuracies are 98.52% and 94.47% (no additional light sources), as well as 96.97% and 95.40% (two additional light sources), respectively. In addition, as an effective supplement to the problem of changes in illumination, the average recognition ratios are 96.23% and 96.20% under dry and wet face conditions, respectively. The experimental results reveal the robust performance of the proposed method in facial expression recognition.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141
Mohammadi Z, Frounchi J, Amiri M (2017) Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl 28(8):1985–1990
Aspinall P, Mavros P, Coyne R, Roe J (2015) The urban brain: analysing outdoor physical activity with mobile EEG. Br J Sports Med 49(4):272–276
Tantawi MM, Revett K, Salem A-B, Tolba MF (2015) A wavelet feature extraction method for electrocardiogram (ECG)-based biometric recognition. Signal Image Video Process 9(6):1271–1280
Chen Y, Yang Z, Gong H, Wang SJ (2018) Recognition of sketching from surface electromyography. Neural Comput Appl 30(9):2725–2737
Gruebler A, Suzuki K (2010) Measurement of distal EMG signals using a wearable device for reading facial expressions. In: 2010 annual international conference of the IEEE engineering in medicine and biology. IEEE, pp 4594–4597
Di Rienzo M, Rizzo F, Parati G, Brambilla G, Ferratini M, Castiglioni P (2005) MagIC system: a new textile-based wearable device for biological signal monitoring. Applicability in daily life and clinical setting. In: Conference proceedings IEEE engineering in medicine and biology society, pp 7167–7169
Einighammer H, Gilenko M (2009) Method and device for recognition of natural skin during contact-free biometric identification of a person. Google Patents
Giannakakis G, Pediaditis M, Manousos D, Kazantzaki E, Chiarugi F, Simos PG, Marias K, Tsiknakis M (2017) Stress and anxiety detection using facial cues from videos. Signal Process Control 31:89–101
Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1–2):160–187
Wilson PI, Fernandez J (2006) Facial feature detection using Haar classifiers. J Comput Sci Coll 21(4):127–133
Zhang H, Wu QJ, Chow TW, Zhao M (2012) A two-dimensional neighborhood preserving projection for appearance-based face recognition. Pattern Recognit 45(5):1866–1876
Saeed S, Mahmood MK, Khan YD (2018) An exposition of facial expression recognition techniques. Neural Comput Appl 29(9):425–443
Gross R, Brajovic V (2003) An image preprocessing algorithm for illumination invariant face recognition. In: International conference on audio-and video-based biometric person authentication. Springer, Berlin, pp 10–18
Dizdaroğlu B, Ataer-Cansizoglu E, Kalpathy-Cramer J, Keck K, Chiang MF, Erdogmus D (2014) Structure-based level set method for automatic retinal vasculature segmentation. EURASIP J Image Video Process 2014(1):39
Xu F, Wang Z (2017) A facial expression recognition method based on cubic spline interpolation and HOG features. In: 2017 IEEE international conference on robotics and biomimetics (ROBIO). IEEE, pp 1300–1305
Wang L, Li R, Wang K (2013) Automatic facial expression recognition using SVM based on AAMs. In: 2013 5th international conference on intelligent human-machine systems and cybernetics. IEEE, pp 330–333
Wang X, Liu X, Lu L, Shen Z (2014) A new facial expression recognition method based on geometric alignment and l bp features. In: 2014 IEEE 17th international conference on computational science and engineering. IEEE, pp 1734–1737
Luo Y, Zhang L, Chen Y, Jiang W (2017) Facial expression recognition algorithm based on reverse co-salient regions (RCSR) features. In: 2017 4th international conference on information science and control engineering (ICISCE). IEEE, pp 326–329
Cornejo JYR, Pedrini H (2018) Emotion recognition from occluded facial expressions using weber local descriptor. In: 2018 25th international conference on systems, signals and image processing (IWSSIP). IEEE, pp 1–5
Zhang H, Ji Y, Huang W, Liu L (2018) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3579-x
Zhang H, Cao X, Ho JK, Chow TW (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531
Chaki J, Dey N, Shi F, Sherratt RS (2019) Pattern mining approaches used in sensor-based biometric recognition: a review. IEEE Sens J 19(10):3569–3580
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628
Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203
Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, pp 558–565
Uçar A, Demir Y, Güzeliş C (2016) A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering. Neural Comput Appl 27(1):131–142
Lowe DG (1999) Object recognition from local scale-invariant features. In: iccv, vol 2, pp 1150–1157
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154
Gower JC (1975) Generalized procrustes analysis. Psychometrika 40(1):33–51
Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 1034–1041
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73
Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52
Rosipal R, Girolami M, Trejo LJ, Cichocki A (2001) Kernel PCA for feature extraction and de-noising in nonlinear regression. Neural Comput Appl 10(3):231–243
Yildirim A, Halici U (2013) Analysis of dimension reduction by PCA and AdaBoost on spelling paradigm EEG data. In: 2013 6th international conference on biomedical engineering and informatics. IEEE, pp 192–196
Zhang C, Shao Y, Tan J, Deng N (2013) Mixed-norm linear support vector machine. Neural Comput Appl 23(7–8):2159–2166
Zhang J, Wang S (2016) A fast leave-one-out cross-validation for SVM-like family. Neural Comput Appl 27(6):1717–1730
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol 2. Montreal, Canada, pp 1137–1145
Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The Japanese female facial expression (JAFFE) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16
Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580). IEEE, pp 46–53
Lu Y, Zhang C, Zhou B-Y, Gao X-P, Lv Z (2018) A dual model approach to EOG-based human activity recognition. Biomed Signal Process Control 45:50–57
Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit 30(7):1145–1159
Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45(2):171–186
Friesen E, Ekman P (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto, CA
Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: Face and gesture 2011. IEEE, pp 314–321
Dharavath K, Talukdar F, Laskar R, Dey N (2017) Face recognition under dry and wet face conditions. In: Dey N, Santhi V (eds) Intelligent techniques in signal processing for multimedia security. Springer, Berlin, pp 253–271
Munasinghe M (2018) Facial expression recognition using facial landmarks and random forest classifier. In: 2018 IEEE/ACIS 17th international conference on computer and information science (ICIS). IEEE, pp 423–427
Alsubari A, Satange D, Ramteke R (2017) Facial expression recognition using wavelet transform and local binary pattern. In: 2017 2nd international conference for convergence in technology (I2CT). IEEE, pp 338–342
The authors would like to thank the volunteers who participated in this study and anonymous reviewers for comments on a draft of this article. The research work is supported by the Open fund for Discipline Construction, Institute of Physical Science and Information Technology, Anhui University; the Anhui Provincial Natural Science Research Project of Colleges and Universities Fund under Grant KJ2018A0008, KJ2017A012; the Natural Science Foundation of Anhui Province under Grant 1908085MF203; and the Open fund for Shaanxi Key Laboratory of Network and System Security under Grant NSSOF1900101.
Conflict of interest
Author Yong Shi declares that he has no conflict of interest. Author Chao Zhang declares that he has no conflict of interest. Author Ning Bi declares that he has no conflict of interest. Author Zhao Lv declares that he has no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Shi, Y., Lv, Z., Bi, N. et al. An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations. Neural Comput & Applic 32, 9267–9281 (2020). https://doi.org/10.1007/s00521-019-04437-w
- Expression recognition
- Feature point position
- Scale-invariant feature transform (SIFT) descriptors
- Principal component analysis (PCA)
- Support vector machine (SVM)