Abstract
Human facial expression recognition has been treated as a multi-class classification problem in the field of artificial intelligence. The main difficulty lies in how to distinguish the different categories of expression features. In this paper, we identify common facial expressions by fusing multiple weak classifiers. It compensates for the disadvantage of single classifier in weak generalization ability and low recognition rate for different datasets and different environments. This paper integrates the prediction results of each classifier through improved weighted mean value method and proposes an expression feature extraction method based on keypoint detection. Classifier fusion methods enable each classifier to perform at its best in order to improve overall expression recognition. Keypoint detection is used to improve the model’s attention on the expression features. Convolution neural network is selected as the model for feature extraction and classification, and the model structure is adjusted. Experiments show that the recognition accuracy of this method used on datasets FER 2013 and CK+ are 70.7% and 95.4% respectively, which are better than that of a single classifier, which shows that the keypoint extraction feature and classifier fusion method used in this paper have a good effect on facial expression recognition.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bera S, Roy SK (2020) Fuzzy rough soft set and its application to lattice. Granular Comput 5(2):217–223
Dange AD, Momin B (2019) The cnn and dpm based approach for multiple object detection in images. In: 2019 International conference on intelligent computing and control systems (ICCS), IEEE, pp 1106–1109
Egrioglu E, Yolcu U, Bas E (2019) Intuitionistic high-order fuzzy time series forecasting method based on pi-sigma artificial neural networks trained by artificial bee colony. Granular Comput 4(4):639–654
Ejegwa PA (2020) Generalized triparametric correlation coefficient for pythagorean fuzzy sets with application to mcdm problems. Granular Comput (1–2)
Ekman P, Friesen W (1978) A technique for the measurement of facial actions. Rivista DI Psichiatria 47(2):126–138
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:170404861
Kido S, Hirano Y, Hashimoto N (2018) Detection and classification of lung abnormalities by use of convolutional neural network (cnn) and regions with cnn features (r-cnn). In: 2018 International workshop on advanced image technology (IWAIT), IEEE, pp 1–4
Kong Q, Zhang X, Xu W (2019) Operation properties and algebraic properties of multi-covering rough sets. Granular Comput 4(3):377–390
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105
Lecun Y, Bottou L (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Liang M, Mi J, Feng T (2019) Optimal granulation selection for multi-label data based on multi-granulation rough sets. Granular Comput 4(3):323–335
Lin JT (1997) Granular computing. Announcement of the BISC Special Interest Group on Granular Computing
Mehrabian A, Russell JA (1974) An approach to environmental psychology. The MIT Press, Chicago
Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), IEEE, pp 1–10
Pawlak Z (1982) Rough sets. Int J Comput Inform Sci 11(5):341–356
Shi D, Zhang X (2019) Probabilistic decision making based on rough sets in interval-valued fuzzy information systems. Granular Comput 4(3):391–405
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556
Tsai HH, Chang YC (2018) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput 22(13):4389–4405
Wu F, Yan S, Smith JS, Zhang B (2019) Deep multiple classifier fusion for traffic scene recognition. Granular Comput pp 1–12
Xu LL, Zhang SM, Zhao JL (2019) Expression recognition algorithm for parallel convolutional neural network. J Image Graph 24:227–236
Yang B, Cao JM, Jiang DP, Lv JD (2018) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tool Appl 77:20477–20499
Yao J (2005a) Information granulation and granular relationships. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 326–329
Yao JT, Vasilakos AV, Pedrycz W (2013) Granular computing: perspectives and challenges. IEEE Trans Cybern 43(6):1977–1989
Yao MZ, Huang GW (2020) Facial expression recognition based on convolutional neural network. Comput Knowl Techol 16(16):19–23
Yao Y (2005b) Perspectives of granular computing. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 85–90
Yurtkan K, Demirel H (2014) Entropy-based feature selection for improved 3d facial expression recognition. SIViP 8(2):267–277
Zadeh LA (1965) Fuzzy sets. Inf. Control 8(3):338–353
Zadeh LA (1979) Fuzzy sets and information granularity. Adv Fuzzy Set Theory Appl 11:3–18
Zadeh LA (1996) Key roles of information granulation and fuzzy logic in human reasoning, concept formulation and computing with words. In: Proceedings of IEEE 5th international fuzzy systems, IEEE, vol 1, pp 1–1
Zadeh LA (1998) Some reflections on soft computing, granular computing and their roles in the conception, design and utilization of information/intelligent systems. Soft Comput 2(1):23–25
Zhang W, Wang X, Yang X, Chen X, Wang P (2019) Neighborhood attribute reduction for imbalanced data. Granular Comput 4(3):301–311
Zhao H, Liu H (2020) Multiple classifiers fusion and cnn feature extraction for handwritten digits recognition. Granular Comput 5(3):411–418
Acknowledgements
This work is supported by ‘Chenguang Program’ supported by Shanghai Education Development Fo-undation and Shanghai Municipal Education Commission under grant number 18CG54. Furthermore, this work is also sponsored by Project funded by China Postdoctoral Science Foundation under Grant Number 2019M651576, National Natural Science Foundation of China (CN) under Grant Number 61602296, Natural Science Foundation of Shanghai (CN) Under Grant Number 16ZR1414500. The authors would like to thank their supports.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, C., Zhu, C. Multiple classifiers fusion for facial expression recognition. Granul. Comput. 7, 171–181 (2022). https://doi.org/10.1007/s41066-021-00258-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41066-021-00258-2