Neural Computing and Applications

, Volume 31, Issue 12, pp 8583–8592 | Cite as

A hybrid framework for smile detection in class imbalance scenarios

  • Thanh Vo
  • Trang Nguyen
  • C. T. LeEmail author
Original Article


In this study, we consider the problem of smile detection in both an imbalanced data scenario, in which the number of smile images is in the minority compared with the number of neutral images, and a balanced data scenario. We first propose a smile detection model using a convolutional neural network (SD-CNN) to improve the performance in the balanced data scenario, and then a hybrid deep learning framework (HF-SD) that uses a modification of the SD-CNN model to learn and then extracts the features from dataset. These extracted features are then used to train an extreme gradient boosting approach to handle the imbalanced problem. An experiment shows that the proposed model has impressive discriminative ability for smile detection, in both balanced and imbalanced data scenarios, compared with existing approaches. HF-SD yields an accuracy of 93.6% and outperforms the state-of-the-art approaches for the original GENK14K database in the balanced data scenario. The results of the second experiment show that HF-SD also achieves better AUCs (area under the receiver operating characteristic curve) compared with the state-of-the-art methods for smile detection in an imbalanced data scenario with different balancing ratios.


Smile detection Deep learning Imbalanced scenarios Machine vision 


Compliance with ethical standards

Conflict of interest

The authors declare no conflict of interest.


  1. 1.
    Ali M, Le HS, Khan M, Nguyen TT (2018) Segmentation of dental X-ray images in medical imaging using neutrosophic orthogonal matrices. Expert Syst Appl 91:434–441CrossRefGoogle Scholar
  2. 2.
    An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363CrossRefGoogle Scholar
  3. 3.
    Bianco S, Celona L, Schettini R (2016) Robust smile detection using convolutional neural networks. J Electron Imaging 25(6):063002CrossRefGoogle Scholar
  4. 4.
    Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183CrossRefGoogle Scholar
  5. 5.
    Chen T, Guestrin T (2016) XGBoost: a scalable tree boosting system. In: KDD, pp 785–794Google Scholar
  6. 6.
    Chen Y, Zou W, Tang Y, Li X, Xu C, Komodakis N (2018) SCOM: spatiotemporal constrained optimization for salient object detection. IEEE Trans Image Process 27(7):3345–3357MathSciNetCrossRefGoogle Scholar
  7. 7.
    Dinh VS, Le TBC, Do PT (2017) Facial smile detection using convolutional neural networks. In: KSE’17, pp 136–141Google Scholar
  8. 8.
    Galar M, Fernández A, Tartas EB, Sola HB, Herrera F (2012) A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern Part C 42(4):463–484CrossRefGoogle Scholar
  9. 9.
    Gao Y, Liu H, Wu P, Wang C (2016) A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing 174:1077–1086CrossRefGoogle Scholar
  10. 10.
    Han D, Liu Q, Fan W (2018) A new image classification method using CNN transfer learning and web data augmentation. Expert Syst Appl 95:43–56CrossRefGoogle Scholar
  11. 11.
    He H, Garcia EA (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284CrossRefGoogle Scholar
  12. 12.
    Hu L, Ni Q (2018) IoT-driven automated object detection algorithm for urban surveillance systems in smart cities. IEEE Internet Things J 5(2):747–754CrossRefGoogle Scholar
  13. 13.
    Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML’15, pp 448–456Google Scholar
  14. 14.
    King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758Google Scholar
  15. 15.
    Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR)Google Scholar
  16. 16.
    Le A, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363CrossRefGoogle Scholar
  17. 17.
    Le HS, Pham HT (2017) Some novel hybrid forecast methods based on picture fuzzy clustering for weather nowcasting from satellite image sequences. Appl Intell 46(1):1–15MathSciNetCrossRefGoogle Scholar
  18. 18.
    Le HS, Tran MT, Fujita H, Dey N, Ashour AS, Vo TNN, Le QA, Chu DT (2018) Dental diagnosis from x-ray images: an expert system based on fuzzy computing. Biomed Signal Process Control 39:64–73CrossRefGoogle Scholar
  19. 19.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444CrossRefGoogle Scholar
  20. 20.
    Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: CVPR, pp 5325–5334Google Scholar
  21. 21.
    Liu M, Li S, Shan S, Chen X (2012) Enhancing expression recognition in the wild with unlabeled reference data. In: Asian conference on computer vision (ACCV), pp 577–588CrossRefGoogle Scholar
  22. 22.
    Lopes AT, Aguiar E, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628CrossRefGoogle Scholar
  23. 23.
    Parka BH, Oha SY, Kim IJ (2017) Face alignment using a deep neural network with local feature learning and recurrent regression. Expert Syst Appl 89:66–80CrossRefGoogle Scholar
  24. 24.
    Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2017) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832CrossRefGoogle Scholar
  25. 25.
    Raza M, Chen Z, Rehman SU, Wang P, Bao P (2018) Appearance based pedestrians’ head pose and body orientation estimation using deep learning. Neurocomputing 272:647–659CrossRefGoogle Scholar
  26. 26.
    Ren S, He K, Girshick RB, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149CrossRefGoogle Scholar
  27. 27.
    Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018MathSciNetCrossRefGoogle Scholar
  28. 28.
    Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436MathSciNetCrossRefGoogle Scholar
  29. 29.
    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLRGoogle Scholar
  30. 30.
    Vo T, Nguyen T, Le CT (2018) Race recognition using deep convolutional neural networks. Symmetry 10(11):564CrossRefGoogle Scholar
  31. 31.
    Wei Y, Xia W, Lin M, Huang J, Ni B, Dong J, Zhao Y, Yan S (2016) HCP: a flexible CNN framework for multi-label image classification. IEEE Trans Pattern Anal Mach Intell 38(9):1901–1907CrossRefGoogle Scholar
  32. 32.
    Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31:2106–2111CrossRefGoogle Scholar
  33. 33.
    Zhang Y, Zhang E, Chen W (2016) Deep neural network for halftone image classification based on sparse auto-encoder. Eng Appl Artif Intell 50:245–255CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute of Research and DevelopmentDuy Tan UniversityDa NangVietnam
  2. 2.Faculty of Information TechnologyHo Chi Minh City Open UniversityHo Chi Minh CityVietnam
  3. 3.Faculty of Information TechnologyHo Chi Minh City University of Technology (HUTECH)Ho Chi Minh CityVietnam

Personalised recommendations