RGB-D Sensor for Facial Expression Recognition in AAL Context

  • Andrea Caroppo
  • Alessandro Leone
  • Pietro Siciliano
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 457)


This paper investigates the use of a commercial and low-cost RGB-D sensor for real-time facial expression recognition in Ambient Assisted Living Context. Since head poses and light conditions could be very different in domestic environments, the methodology used was designed to handle such situations. The implemented framework is able to classify four different categories of facial expressions: (1) happy, (2) sad, (3) fear/surprise, and (4) disgust/anger. The classification is obtained through an hybrid-based approach, by combining appearance and geometric features. The HOG feature descriptor and a group of Action Units compose the feature vector that is given as input, in the classification step, to a group of Support Vector Machines. The robustness of the approach is highlighted by the results obtained: the average accuracy for fear/surprise is the lowest with 85.2%, while happy is the facial expression better recognized (93.6%). Sad and disgust/anger are the facial expression confused the most.


Facial expression recognition Kinect sensor HOG features Action units Emotion recognition Support vector machine (SVM) 


  1. 1.
    Hu, Y., Zeng, Z., Yin, L., Wei, X., Tu, J., Huang, T.S.: A study of non-frontal-view facial expressions recognition. In: 19th International Conference on Pattern Recognition, 2008 (ICPR 2008), IEEE, pp. 1–4 (2008)Google Scholar
  2. 2.
    Rudovic, O., Pantic, M., Patras, I.: Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1357–1369 (2013)CrossRefGoogle Scholar
  3. 3.
    Zheng, W.: Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Trans. Affect. Comput. 5(1), 71–85 (2014)CrossRefGoogle Scholar
  4. 4.
    Cament, L.A., Galdames, F.J., Bowyer, K.W., Perez, C.A.: Face recognition under pose variation with local Gabor features enhanced by active shape and statistical models. Pattern Recogn. 48(11), 3371–3384 (2015)CrossRefGoogle Scholar
  5. 5.
    Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30(10), 683–697 (2012)CrossRefGoogle Scholar
  6. 6.
    Malawski, F., Kwolek, B., Sako, S.: Using kinect for facial expression recognition under varying poses and illumination. In: International Conference on Active Media Technology, pp. 395–406, Springer International Publishing (2014)Google Scholar
  7. 7.
    Andò, B., Siciliano, P., Marletta, V., Monteriù, A.: Ambient Assisted Living. (2015)Google Scholar
  8. 8.
    Chang, Y., Hu, C., Feris, R., Turk, M.: Manifold based analysis of facial expression. Image Vis. Comput. 24(6), 605–614 (2006)CrossRefGoogle Scholar
  9. 9.
    Shbib, R., Zhou, S.: Facial expression analysis using active shape model. Int J Signal Process., Image Process. Pattern Recognit. 8(1), 9–22 (2015)Google Scholar
  10. 10.
    Cheon, Y., Kim, D.: Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recognit. 42(7), 1340–1350 (2009)CrossRefMATHGoogle Scholar
  11. 11.
    Chen, Y., Hua, C., Bai, R.: Regression-based active appearance model initialization for facial feature tracking with missing frames. Pattern Recognit. Lett. 38, 113–119 (2014)CrossRefGoogle Scholar
  12. 12.
    Soyel, H., Demirel, H.: Facial expression recognition based on discriminative scale invariant feature transform. Electron. Lett. 46(5), 343–345 (2010)CrossRefGoogle Scholar
  13. 13.
    Gu, W., Xiang, C., Venkatesh, Y.V., Huang, D., Lin, H.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)CrossRefGoogle Scholar
  14. 14.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6): (2007)Google Scholar
  15. 15.
    Dahmane, M., Meunier, J.: Emotion recognition using dynamic grid-based HoG features. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 884–888, IEEE (2011)Google Scholar
  16. 16.
    Jack, R.E., Garrod, O.G., Schyns, P.G.: Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Curr. Biol. 24(2), 187–192 (2014)CrossRefGoogle Scholar
  17. 17.
    Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)CrossRefGoogle Scholar
  18. 18.
  19. 19.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  20. 20.
    Yu, X., Huang, J., Zhang, S., Yan, W., Metaxas, D.N.: Pose-free facial landmark fitting via optimized part mixtures and cascaded deformable shape model. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1944–1951 (2013)Google Scholar
  21. 21.
    Dalal, N., & Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005 (CVPR 2005), vol. 1, pp. 886–893, IEEE (2005)Google Scholar
  22. 22.
    Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System (FACS). A Technique for the Measurement of Facial Action. Consulting, Palo Alto, 22 (1978)Google Scholar
  23. 23.
    Knerr, S., Personnaz, L., Dreyfus, G.: Single-layer learning revisited: a stepwise procedure for building and training a neural network. In: Neurocomputing, pp. 41–50. Springer, Berlin (1990)Google Scholar
  24. 24.
    Hsu, C.W., Chang, C.C., Lin, C.J.: A Practical Guide to Support Vector Classification. (2003)Google Scholar
  25. 25.
    Aly, S., Trubanova, A., Abbott, L., White, S., Youssef, A.: VT-KFER: a Kinect-based RGBD+ time dataset for spontaneous and non-spontaneous facial expression recognition. In: 2015 International Conference on Biometrics (ICB), pp. 90–97. IEEE (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Andrea Caroppo
    • 1
  • Alessandro Leone
    • 1
  • Pietro Siciliano
    • 1
  1. 1.National Research Council of Italy, Institute for Microelectronics and MicrosystemsLecceItaly

Personalised recommendations