Using Kinect for real-time emotion recognition via facial expressions

  • Qi-rong Mao
  • Xin-yu Pan
  • Yong-zhao Zhan
  • Xiang-jun Shen


Emotion recognition via facial expressions (ERFE) has attracted a great deal of interest with recent advances in artificial intelligence and pattern recognition. Most studies are based on 2D images, and their performance is usually computationally expensive. In this paper, we propose a real-time emotion recognition approach based on both 2D and 3D facial expression features captured by Kinect sensors. To capture the deformation of the 3D mesh during facial expression, we combine the features of animation units (AUs) and feature point positions (FPPs) tracked by Kinect. A fusion algorithm based on improved emotional profiles (IEPs) and maximum confidence is proposed to recognize emotions with these real-time facial expression features. Experiments on both an emotion dataset and a real-time video show the superior performance of our method.

Key words

Kinect, Emotion recognition Facial expression Real-time classification Fusion algorithm Support vector machine (SVM) 

CLC number



Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ahlberg, J., 2001. Candide-3—an Updated Parameterised Face. Technical Report.Google Scholar
  2. Breidt, M., Biilthoff, H.H., Curio, C., 2011. Robust semantic analysis by synthesis of 3D facial motion. Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition and Workshops, p.713–719. [doi:10.1109/FG.2011.5771336]Google Scholar
  3. Cao, C., Weng, Y.L., Zhou, S., et al., 2014. FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual. Comput. Graph., 20(3):413–425. [doi:10.1109/TVCG.2013.249]CrossRefGoogle Scholar
  4. Chang, C.C., Lin, C.J., 2011a. LIBSVM: a Library for Support Vector Machines. Available from Scholar
  5. Chang, C.C., Lin, C.J., 2011b. LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol., 2(3):27. [doi:10.1145/1961189.1961199]CrossRefGoogle Scholar
  6. Cosker, D., Krumhuber, E., Hilton, A., 2011. A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. Proc. IEEE Int. Conf. on Computer Vision, p.2296–2303. [doi:10.1109/ICCV.2011.6126510]Google Scholar
  7. Ekman, P., 1993. Facial expression and emotion. Am. Psychol., 48(4):384–392. [doi:10.1037/0003-066X.48.4.384]CrossRefGoogle Scholar
  8. Ekman, P., Friesen, W.V., 1978. Facial action coding system: a technique for the measurement of facial movement. Palo Alto.Google Scholar
  9. Fasel, B., Luettin, J., 2003. Automatic facial expression analysis: a survey. Patt. Recog., 36(1):259–275. [doi:10. 1016/S0031-3203(02)00052-3]CrossRefMATHGoogle Scholar
  10. Hg, R.I., Jasek, P., Rofidal, C., et al., 2012. An RGB-D database using Microsoft’s Kinect for windows for face detection. Proc. 8th Int. Conf. on Signal Image Technology and Internet Based Systems, p.42–46. [doi:10. 1109/SITIS.2012.17]Google Scholar
  11. Li, B.Y., Mian, A.S., Liu, W.Q., et al., 2013. Using Kinect for face recognition under varying poses, expressions, illumination and disguise. Proc. IEEE Workshop on Applications of Computer Vision, p.186–192. [doi:10.1109/WACV.2013.6475017]Google Scholar
  12. Li, D.X., Sun, C., Hu, F.Q., et al., 2013. Real-time performance-driven facial animation with 3ds Max and Kinect. Proc. 3rd Int. Conf. on Consumer Electronics, Communications and Networks, p.473–476. [doi:10. 1109/CECNet.2013.6703372]Google Scholar
  13. Ma, X.H., Tan, Y.Q., Zheng, G.M., 2013. A fast classification scheme and its application to face recognition. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 14(7):561–572. [doi:10.1631/jzus.CIDE1309]CrossRefGoogle Scholar
  14. Mao, Q.R., Zhao, X.L., Huang, Z.W., et al., 2013. Speakerindependent speech emotion recognition by fusion of functional and accompanying paralanguage features. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 14(7):573–582. [doi:10.1631/jzus.CIDE1310]CrossRefGoogle Scholar
  15. Nicolaou, M.A., Gunes, H., Pantic, M., 2011. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. Affect. Comput., 2(2):92–105. [doi:10.1109/T-AFFC. 2011.9]CrossRefGoogle Scholar
  16. Savran, A., Alyüz, N., Dibeklioğlu, H., et al., 2008. Bosphorus database for 3D face analysis. Proc. 1st European Workshop on Biometrics and Identity Management, p.47–56. [doi:10.1007/978-3-540-89991-4_6]CrossRefGoogle Scholar
  17. Seddik, B., Maamatou, H., Gazzah, S., et al., 2013. Unsupervised facial expressions recognition and avatar reconstruction from Kinect. Proc. 10th Int. Multi-conf. on Systems, Signals & Devices, p.1–6. [doi:10.1109/SSD.2013.6564032]Google Scholar
  18. Stratou, G., Ghosh, A., Debevec, P., et al., 2011. Effect of illumination on automatic expression recognition: a novel 3D relightable facial database. Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition and Workshops, p.611–618. [doi:10.1109/FG.2011.5771467]Google Scholar
  19. van den Hurk, Y., 2012. Gender Classification with Visual and Depth Images. MS Thesis, Tilburg University, the Netherlands.Google Scholar
  20. Vinciarelli, A., Pantic, M., Heylen, D., et al., 2012. Bridging the gap between social animal and unsocial machine: a survey of social signal processing. IEEE Trans. Affect. Comput., 3(1):69–87. [doi:10.1109/T-AFFC.2011.27]CrossRefGoogle Scholar
  21. Xu, S.B., Ma, G.H., Meng, W.L., et al., 2013. Statistical learning based facial animation. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 14(7):542–550. [doi:10. 1631/jzus.CIDE1307]CrossRefGoogle Scholar
  22. Zeng, Z., Pantic, M., Roisman, G.I., et al., 2009. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Patt. Anal. Mach. Intell., 31(1):39–58. [doi:10.1109/TPAMI.2008.52]CrossRefGoogle Scholar
  23. Zhu, X.X., Ramanan, D., 2012. Face detection, pose estimation, and landmark localization in the wild. Proc. IEEE Conf. on Computer Vision and Pattern Recognition, p.2879–2886. [doi:10.1109/CVPR.2012.6248014]Google Scholar

Copyright information

© Journal of Zhejiang University Science Editorial Office and Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Qi-rong Mao
    • 1
  • Xin-yu Pan
    • 1
  • Yong-zhao Zhan
    • 1
  • Xiang-jun Shen
    • 1
  1. 1.Department of Computer Science and Communication EngineeringJiangsu UniversityZhenjiangChina

Personalised recommendations