Advertisement

Weakly supervised facial expression recognition via transferred DAL-CNN and active incremental learning

  • Ying Xu
  • Jian Liu
  • Yikui ZhaiEmail author
  • Junying Gan
  • Junying Zeng
  • He Cao
  • Fabio Scotti
  • Vincenzo Piuri
  • Ruggero Donida Labati
Focus
First Online:
  • 2 Downloads

Abstract

In recent years, facial expression recognition (FER) has becoming a growing topic in computer vision with promising applications on virtual reality and human–robot interaction. Due to the influence of illumination, individual differences, attitude variation, etc., facial expression recognition with robust accuracy in complex environment is still an unsolved problem. Meanwhile, with the wide use of social communication, massive data are uploaded to the Internet; the effective utilization of those data is still a challenge due to noisy label phenomenon in the study of FER. To resolve the above-mentioned problems, firstly, a double active layer-based CNN is established to recognize the facial expression with high accuracy by learning robust and discriminative features from the data, which could enhance the robustness of network. Secondly, an active incremental learning method was utilized to tackle the problem of using Internet data. During the training phase, a two-stage transfer learning method is explored to transfer the relative information from face recognition to FER task to alleviate the inadequate training data in deep convolution network. Besides, in order to make better use of facial expression data from Web site and further improve the FER accuracy, Unconstrained Facial Expression Database from Web site database is built in this paper. Extensive experiments performed on two public facial expression recognition databases FER 2013 and SFEW 2.0 have demonstrated that the proposed scheme outperforms the state-of-the-art methods, which could achieve 67.08% and 51.90%, respectively.

Keywords

Deep convolutional neural network Facial expression recognition Two-stage transfer learning Weakly supervised active incremental learning 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

This work is supported by National Natural Science Foundation of China (No. 61771347); Characteristic Innovation Project of Guangdong Province (No. 2017KTSCX181); Young Innovative Talents Project of Guangdong Province (2017KQNCX206); Jiangmen Science and Technology Project ([2017] No. 268); 2017 Guangdong Science and Technology Plan Project (No. 2017A010101019); Youth Foundation of Wuyi University (No. 2015zk11); the Opening Project of GuangDong Province Key Laboratory of Information Security Technology (Grant No. 2017B030314131); the 2018 Opening Project of GuangDong Province Key Laboratory of Digital Signal and Image Processing.

Compliance with ethical standards

Conflict of interest

All authors declare that he/she has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

References

  1. Ahmed A (2016) Facial expression recognition using Gabor wavelet and artificial neural networks. Dissertation Sudan University of Science and TechnologyGoogle Scholar
  2. Alsamirat MA, Al-Alem F, Al-Ayoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimed Tools Appl 78(3):3649–3688CrossRefGoogle Scholar
  3. Burkert P, Trier F, Afzal M (2015) Dexpression: deep convolutional neural network for expression recognition. arXiv preprintGoogle Scholar
  4. Chen YS (2019) Performance identification in large-scale class data from advanced facets of computational intelligence and soft computing techniques. Int J High Perform Comput Netw 13(3):283–293CrossRefGoogle Scholar
  5. Chen J, Liu X, Tu P (2012) Person-specific expression recognition with transfer learning. In: Proceedings of IEEE international conference on image processing, pp 2621–2624Google Scholar
  6. Ding H, Zhou S, Chellappa R (2017) Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: Proceedings of the 12th IEEE international conference on automatic face and gesture recognition, pp 118–126Google Scholar
  7. El-Latif AA, Abd-El-Atty B, Hossain MS, Rahman MA, Alamri A, Gupta BB (2018) Efficient quantum information hiding for remote medical image sharing. IEEE Access 6:21075–21083CrossRefGoogle Scholar
  8. Frénay B, Verleysen M (2014) Classification in the presence of label noise: a survey. IEEE Trans Neural Netw Learn Syst 25(5):845–869CrossRefGoogle Scholar
  9. Hai T, Le H, Thuy N (2015) Facial expression classification using artificial neural network and K-nearest neighbor. Int J Inf Technol Comput Sci 7(3):27–32Google Scholar
  10. Jing W, Jiang T, Zhang X, Zhu L (2019) The optimization of speech recognition based on convolutional neural network. IJHPCN 13(2):222–231CrossRefGoogle Scholar
  11. Kumar P, Happy S, Routray A (2017) A real-time robust facial expression recognition system using HOG features. In: Proceedings of international conference on computing, analytics and security trends, pp 289–293Google Scholar
  12. Kung S (2016) Facial expression recognition using optical flow and 3D HMM and human action recognition using cuboid and topic models. Oakland University, OaklandGoogle Scholar
  13. Lecun Y, Boser B, Denker J (2008) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551CrossRefGoogle Scholar
  14. Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of ACM on international conference on multimodal interaction, 2015, pp 503–510Google Scholar
  15. Li J, Ban X, Yang G, Li Y, Wang Y (2019) Real-time human action recognition using depth motion maps and convolutional neural networks. Int J High Perform Comput Netw 13(3):312–320CrossRefGoogle Scholar
  16. Liu P, Han S, Meng Z (2014) Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1805–1812Google Scholar
  17. Liu M, Li S, Shan S (2015) AU-inspired deep networks for facial expression feature learning. Neurocomputing 159(2):126–136CrossRefGoogle Scholar
  18. Liu Y, Cao Y, Li Y (2016a) Facial expression recognition with PCA and LBP features extracting from active facial patches. In: Proceedings of IEEE international conference on real-time computing and robotics (RCAR), pp 368–373Google Scholar
  19. Liu S, Chen X, Fan D (2016b) 3D smiling facial expression recognition based on SVM. In: Proceedings of IEEE international conference on mechatronics and automation, pp 1661–1666Google Scholar
  20. Mohammadi M, Fatemizadeh E, Mahoor M (2014) PCA-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092CrossRefGoogle Scholar
  21. Mollahosseini A, Chan D, Mahoor MH (2015) Going deeper in facial expression recognition using deep neural networks. In: IEEE winter conference on applications of computer vision (WACV), pp 1–10.  https://doi.org/10.1109/WACV.2016.7477450
  22. Ng H, Nguyen V, Vonikakis V (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of ACM international conference on multimodal interaction, 2015, pp 443–449Google Scholar
  23. Pan S, Yang Q (2010) A survey on transfer learning. Proc IEEE Trans Knowl Data Eng 22(10):1345–1359CrossRefGoogle Scholar
  24. Sukhbaatar S, Fergus R (2015) Learning from noisy labels with deep neural networks. In: 3rd International conference on learning representations (ICLR) Workshop Track Proceedings, San Diego, CA, USA, May 7–9, 2015. https://dblp.org/rec/bib/journals/corr/SukhbaatarF14
  25. Sumeet S, Singh S, Saini R (2016) Hardware accelerator for facial expression classification using linear SVM. In: Advances in signal processing and intelligent recognition systems, pp 39–50Google Scholar
  26. Sun N, Chen Z, Day R (2016) Facial expression recognition using digitalized facial features based on active shape model. In: Proceedings of international conference on computer science engineering and applications, pp 39–46Google Scholar
  27. Viola P, Jones M (2004) Robust real-time face detection. In: Eighth IEEE international conference on IEEE Xplore, p 747Google Scholar
  28. Wang X, Liu A, Zhang S (2015) New facial expression recognition based on FSVM and KNN. Int J Light Electron Opt 126(21):3132–3134CrossRefGoogle Scholar
  29. Wang F, Xiang X, Liu C (2017) Transferring face verification nets to pain and expression regression. arXiv preprintGoogle Scholar
  30. Wu X, He R, Sun Z (2015) A lightened CNN for deep face representation. Comput Sci 111–118. https://arxiv.org/abs/1511.02683
  31. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 435–442Google Scholar
  32. Yu K, Wang Z, Zhuo L et al (2010) Harvesting web images for realistic facial expression recognition. In: International conference on digital image computing: techniques and applications, pp 516–521Google Scholar
  33. Zhai Y, Liu J, Zeng J et al (2017) Deep convolutional neural network for facial expression recognition. In: International conference on image and graphics, pp 211–223Google Scholar
  34. Zhang X (2015) Facial expression analysis via transfer learning. Dissertation University of DenverGoogle Scholar
  35. Zhang Y, Zong Y, Wang H (2008) Facial expression recognition based on C-means and K-nearest neighbor algorithms. Caai Trans Intell Syst 3(1):57–61 (in Chinese) Google Scholar
  36. Zhang B, Liu G, Xie G (2017) Facial expression recognition using LBP and LPQ based on Gabor wavelet transform. In: Proceedings of IEEE international conference on computer and communications, pp 365–369Google Scholar
  37. Zhao Q, Zhang D, Lu H (2005) Supervised LLE in ICA space for facial expression recognition. In: Proceedings of international conference on neural networks and brain, pp 1970–1975Google Scholar
  38. Zhou Y, Xu T, Zheng W et al (2017) Classification and recognition approaches of tomato main organs based on DCNN. Trans Chin Soc Agric Eng 33(15):219–226Google Scholar
  39. Zhou Z, Shin J, Gurudu S et al (2018) AFT*: integrating active learning and transfer learning to reduce annotation efforts. arXiv preprint 2018Google Scholar
  40. Zhu R, Zhang T, Zhao Q (2015) A transfer learning approach to cross-database facial expression recognition. In: International conference on biometrics, pp 293–298Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Intelligent ManufacturingWuyi UniversityJiangmenChina
  2. 2.Dipartimento di InformaticaUniversita’ Degli Studi di MilanoMilanItaly

Personalised recommendations

Cite article

Buy options