Multiple classifiers fusion for facial expression recognition

Abstract

Human facial expression recognition has been treated as a multi-class classification problem in the field of artificial intelligence. The main difficulty lies in how to distinguish the different categories of expression features. In this paper, we identify common facial expressions by fusing multiple weak classifiers. It compensates for the disadvantage of single classifier in weak generalization ability and low recognition rate for different datasets and different environments. This paper integrates the prediction results of each classifier through improved weighted mean value method and proposes an expression feature extraction method based on keypoint detection. Classifier fusion methods enable each classifier to perform at its best in order to improve overall expression recognition. Keypoint detection is used to improve the model’s attention on the expression features. Convolution neural network is selected as the model for feature extraction and classification, and the model structure is adjusted. Experiments show that the recognition accuracy of this method used on datasets FER 2013 and CK+ are 70.7% and 95.4% respectively, which are better than that of a single classifier, which shows that the keypoint extraction feature and classifier fusion method used in this paper have a good effect on facial expression recognition.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Bera S, Roy SK (2020) Fuzzy rough soft set and its application to lattice. Granular Comput 5(2):217–223

    Article  Google Scholar 

  2. Dange AD, Momin B (2019) The cnn and dpm based approach for multiple object detection in images. In: 2019 International conference on intelligent computing and control systems (ICCS), IEEE, pp 1106–1109

  3. Egrioglu E, Yolcu U, Bas E (2019) Intuitionistic high-order fuzzy time series forecasting method based on pi-sigma artificial neural networks trained by artificial bee colony. Granular Comput 4(4):639–654

    Article  Google Scholar 

  4. Ejegwa PA (2020) Generalized triparametric correlation coefficient for pythagorean fuzzy sets with application to mcdm problems. Granular Comput (1–2)

  5. Ekman P, Friesen W (1978) A technique for the measurement of facial actions. Rivista DI Psichiatria 47(2):126–138

    Google Scholar 

  6. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:170404861

  7. Kido S, Hirano Y, Hashimoto N (2018) Detection and classification of lung abnormalities by use of convolutional neural network (cnn) and regions with cnn features (r-cnn). In: 2018 International workshop on advanced image technology (IWAIT), IEEE, pp 1–4

  8. Kong Q, Zhang X, Xu W (2019) Operation properties and algebraic properties of multi-covering rough sets. Granular Comput 4(3):377–390

    Article  Google Scholar 

  9. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  10. Lecun Y, Bottou L (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  11. Liang M, Mi J, Feng T (2019) Optimal granulation selection for multi-label data based on multi-granulation rough sets. Granular Comput 4(3):323–335

    Article  Google Scholar 

  12. Lin JT (1997) Granular computing. Announcement of the BISC Special Interest Group on Granular Computing

  13. Mehrabian A, Russell JA (1974) An approach to environmental psychology. The MIT Press, Chicago

    Google Scholar 

  14. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), IEEE, pp 1–10

  15. Pawlak Z (1982) Rough sets. Int J Comput Inform Sci 11(5):341–356

    Article  Google Scholar 

  16. Shi D, Zhang X (2019) Probabilistic decision making based on rough sets in interval-valued fuzzy information systems. Granular Comput 4(3):391–405

    Article  Google Scholar 

  17. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:14091556

  18. Tsai HH, Chang YC (2018) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput 22(13):4389–4405

    Article  Google Scholar 

  19. Wu F, Yan S, Smith JS, Zhang B (2019) Deep multiple classifier fusion for traffic scene recognition. Granular Comput pp 1–12

  20. Xu LL, Zhang SM, Zhao JL (2019) Expression recognition algorithm for parallel convolutional neural network. J Image Graph 24:227–236

    Google Scholar 

  21. Yang B, Cao JM, Jiang DP, Lv JD (2018) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tool Appl 77:20477–20499

    Article  Google Scholar 

  22. Yao J (2005a) Information granulation and granular relationships. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 326–329

  23. Yao JT, Vasilakos AV, Pedrycz W (2013) Granular computing: perspectives and challenges. IEEE Trans Cybern 43(6):1977–1989

    Article  Google Scholar 

  24. Yao MZ, Huang GW (2020) Facial expression recognition based on convolutional neural network. Comput Knowl Techol 16(16):19–23

    Google Scholar 

  25. Yao Y (2005b) Perspectives of granular computing. In: 2005 IEEE international conference on granular computing, IEEE, vol 1, pp 85–90

  26. Yurtkan K, Demirel H (2014) Entropy-based feature selection for improved 3d facial expression recognition. SIViP 8(2):267–277

    Article  Google Scholar 

  27. Zadeh LA (1965) Fuzzy sets. Inf. Control 8(3):338–353

    MathSciNet  Article  Google Scholar 

  28. Zadeh LA (1979) Fuzzy sets and information granularity. Adv Fuzzy Set Theory Appl 11:3–18

    MathSciNet  Google Scholar 

  29. Zadeh LA (1996) Key roles of information granulation and fuzzy logic in human reasoning, concept formulation and computing with words. In: Proceedings of IEEE 5th international fuzzy systems, IEEE, vol 1, pp 1–1

  30. Zadeh LA (1998) Some reflections on soft computing, granular computing and their roles in the conception, design and utilization of information/intelligent systems. Soft Comput 2(1):23–25

    Article  Google Scholar 

  31. Zhang W, Wang X, Yang X, Chen X, Wang P (2019) Neighborhood attribute reduction for imbalanced data. Granular Comput 4(3):301–311

    Article  Google Scholar 

  32. Zhao H, Liu H (2020) Multiple classifiers fusion and cnn feature extraction for handwritten digits recognition. Granular Comput 5(3):411–418

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by ‘Chenguang Program’ supported by Shanghai Education Development Fo-undation and Shanghai Municipal Education Commission under grant number 18CG54. Furthermore, this work is also sponsored by Project funded by China Postdoctoral Science Foundation under Grant Number 2019M651576, National Natural Science Foundation of China (CN) under Grant Number 61602296, Natural Science Foundation of Shanghai (CN) Under Grant Number 16ZR1414500. The authors would like to thank their supports.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Changming Zhu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Zhu, C. Multiple classifiers fusion for facial expression recognition. Granul. Comput. (2021). https://doi.org/10.1007/s41066-021-00258-2

Download citation

Keywords

  • Facial expression recognition
  • Classifier fusion
  • Convolution neural network