4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformations

  • Payam ZarbakhshEmail author
  • Hasan Demirel
Original Article


One of the main challenges in dynamic facial expression recognition is how to capture temporal information in the system. In this study, a novel approach based on time series analysis is adapted for this problem. The proposed dynamic facial expression recognition system comprises four phases: head pose correction and normalization, feature extraction, feature selection and classification. Head pose detection and correction is the first phase to realign locations of the facial landmarks. A comprehensive set of geometric deformations including point, distance and angle deformations are extracted from the key points. The concept of facial action unit analysis is interlocked with this phase to identify related key points from the landmarks. A set of multimodal time series are then constructed from the extracted deformations by applying a sliding window to characterize the dynamics of mean deformations in a window. In the third phase, a feature selection method based on neighborhood component analysis is applied on the peak value of deformation features to select useful features and discard irrelevant ones. Finally, adaptive cost dynamic time warping is utilized to recognize six prototypic expressions from multimodal time series of selected features. Experimental results on BU-4DFE data set confirm that proposed algorithm is efficient in dynamic facial expression recognition compared with state of the art.


4D facial expression recognition Geometric deformation Temporal analysis Multimodal time series Adaptive cost dynamic time warping 



  1. 1.
    Amor, B.B., Drira, H., Berretti, S., Daoudi, M., Srivastava, A.: 4-D facial expression recognition by learning geometric deformations. IEEE Trans. Cybern. 44(12), 2443–2457 (2014)CrossRefGoogle Scholar
  2. 2.
    An, F., Liu, Z.: Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM. Vis. Comput. (2019). Google Scholar
  3. 3.
    Arcoverde Neto, E.N., Duarte, R.M., Barreto, R.M., Magalhães, J.P., Bastos, C.C.M., Ren, T.I., Cavalcanti, G.D.C.: Enhanced real-time head pose estimation system for mobile device. Integr. Comput.-Aided Eng. 21(3), 281–293 (2014)CrossRefGoogle Scholar
  4. 4.
    Berndt, D.J., Clifford, J.: Finding patterns in time series: a dynamic programming approach. In: Advances in Knowledge Discovery and Data Mining, pp. 229–248 (1996)Google Scholar
  5. 5.
    Berretti, S., del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis. Comput. 29(12), 1333–1350 (2013)CrossRefGoogle Scholar
  6. 6.
    Bolourchi, P., Demirel, H., Uysal, S.: Target recognition in SAR images using radial Chebyshev moments. Signal Image Video Process. 11(6), 1033–1040 (2017)CrossRefGoogle Scholar
  7. 7.
    Carcagnì, P., Del Coco, M., Leo, M., Distante, C.: Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1), 645 (2015)CrossRefGoogle Scholar
  8. 8.
    Chen, J., Chen, Z., Chi, Z., Fu, H.: Dynamic texture and geometry features for facial expression recognition in video. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4967–4971. IEEE (2015)Google Scholar
  9. 9.
    Choi, J.Y.: Spatial pyramid face feature representation and weighted dissimilarity matching for improved face recognition. Vis. Comput. 34(11), 1535–1549 (2018)CrossRefGoogle Scholar
  10. 10.
    Derkach, D., Ruiz, A., Sukno, F.M.: Head pose estimation based on 3-D facial landmarks localization and regression. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), pp. 820–827. IEEE (2017)Google Scholar
  11. 11.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  12. 12.
    Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)CrossRefGoogle Scholar
  13. 13.
    Fang, T., Zhao, X., Shah, S.K., Kakadiaris, I.A.: 4D facial expression recognition. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1594–1601 (2011)Google Scholar
  14. 14.
    Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)CrossRefGoogle Scholar
  15. 15.
    Ghimire, D., Lee, J., Li, Z.N., Jeong, S.: Recognition of facial expressions based on salient geometric features and support vector machines. Multimed. Tools Appl. 76(6), 7921–7946 (2017)CrossRefGoogle Scholar
  16. 16.
    Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. (2018). Google Scholar
  17. 17.
    Goh, K.M., Ng, C.H., Lim, L.L., Sheikh, U.U.: Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. (2018). Google Scholar
  18. 18.
    Goldberger, J., Hinton, G.E., Roweis, S.T., Salakhutdinov, R.R.: Neighbourhood components analysis. In: Saul, L.K., Weiss, Y., Bottou, L. (eds.) Advances in Neural Information Processing Systems, vol. 17, pp. 513–520. MIT Press, Cambridge (2005)Google Scholar
  19. 19.
    Górecki, T., Łuczak, M.: Multivariate time series classification with parametric derivative dynamic time warping. Expert Syst. Appl. 42(5), 2305–2312 (2015)CrossRefGoogle Scholar
  20. 20.
    Guo, Y., Zhao, G., Pietikainen, M.: Dynamic facial expression recognition with atlas construction and sparse representation. IEEE Trans. Image Process. 25(5), 1977–1992 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Happy, S.L., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)CrossRefGoogle Scholar
  22. 22.
    Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2983–2991 (2015)Google Scholar
  23. 23.
    Kalsum, T., Anwar, S.M., Majid, M., Khan, B., Ali, S.M.: Emotion recognition from facial expressions using hybrid feature descriptors. IET Image Process. 12(6), 1004–1012 (2018)CrossRefGoogle Scholar
  24. 24.
    Keogh, E., Kasetty, S.: On the need for time series data mining benchmarks: a survey and empirical demonstration. Data Min. Knowl. Discov. 7(4), 349–371 (2003)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Kołakowska, A., Landowska, A., Szwoch, M., Szwoch, W., Wróbel, M.R.: Emotion Recognition and Its Applications, pp. 51–62. Springer, Berlin (2014)Google Scholar
  26. 26.
    Li, K., Jin, Y., Akram, M.W., Han, R., Chen, J.: Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis. Comput. (2019). Google Scholar
  27. 27.
    Li, W., Huang, D., Li, H., Wang, Y.: Automatic 4D facial expression recognition using dynamic geometrical image network. In: 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pp. 24–30. IEEE (2018)Google Scholar
  28. 28.
    Liang, D., Liang, H., Yu, Z., Zhang, Y.: Deep convolutional BiLSTM fusion network for facial expression recognition. Vis. Comput. (2019). Google Scholar
  29. 29.
    Lien, J.J.J., Kanade, T., Cohn, J.F., Li, C.C.: Detection, tracking, and classification of action units in facial expression. Robot. Auton. Syst. 31(3), 131–146 (2000)CrossRefGoogle Scholar
  30. 30.
    Lopes, A.T., de Aguiar, E., Souza, A.F.D., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)CrossRefGoogle Scholar
  31. 31.
    Reale, M., Zhang, X., Yin, L.: Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)Google Scholar
  32. 32.
    Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: 2011 IEEE International Conference on Face and Gesture, pp. 406–413. IEEE (2011)Google Scholar
  33. 33.
    Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Image Vis. Comput. 30(10), 762–773 (2012)CrossRefGoogle Scholar
  34. 34.
    Shao, J., Gori, I., Wan, S., Aggarwal, J.: 3D dynamic facial expression recognition using low-resolution videos. Pattern Recognit. Lett. 65, 157–162 (2015)CrossRefGoogle Scholar
  35. 35.
    Sun, Y., Chen, X., Rosato, M., Yin, L.: Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 40(3), 461–474 (2010)CrossRefGoogle Scholar
  36. 36.
    Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)CrossRefGoogle Scholar
  37. 37.
    Valstar, M.F., Pantic, M.: Fully automatic recognition of the temporal phases of facial actions. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(1), 28–43 (2012)CrossRefGoogle Scholar
  38. 38.
    Wan, Y., Chen, X.L., Shi, Y.: Adaptive cost dynamic time warping distance in time series analysis for classification. J. Comput. Appl. Math. 319, 514–520 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., Kissler, J.: Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE 12(5), e0177,239 (2017)CrossRefGoogle Scholar
  40. 40.
    Xue, M., Mian, A., Liu, W., Li, L.: Automatic 4D facial expression recognition using DCT features. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 199–206. IEEE (2015)Google Scholar
  41. 41.
    Yang, W., Wang, K., Zuo, W.: Neighborhood component feature selection for high-dimensional data. J. Comput. 7(1), 31–37 (2012)Google Scholar
  42. 42.
    Yao, Y., Huang, D., Yang, X., Wang, Y., Chen, L.: Texture and geometry scattering representation-based facial expression recognition in 2D+3D videos. ACM Trans. Multimed. Comput. Commun. Appl. 14(1s), 18:1–18:23 (2018)CrossRefGoogle Scholar
  43. 43.
    Yeganli, S.F., Demirel, H., Yu, R.: Noise removal from mr images via iterative regularization based on higher-order singular value decomposition. Signal Image Video Process. 11(8), 1477–1484 (2017)CrossRefGoogle Scholar
  44. 44.
    Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, pp. 1–6. IEEE (2008)Google Scholar
  45. 45.
    Yurtkan, K., Demirel, H.: Feature selection for improved 3D facial expression recognition. Pattern Recognit. Lett. 38, 26–33 (2014)CrossRefGoogle Scholar
  46. 46.
    Zarbakhsh, P., Demirel, H.: Low-rank sparse coding and region of interest pooling for dynamic 3D facial expression recognition. Signal Image Video Process. 12(8), 1611–1618 (2018)CrossRefGoogle Scholar
  47. 47.
    Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans. Multimed. 18(12), 2528–2536 (2016)CrossRefGoogle Scholar
  48. 48.
    Zhao, J., Mao, X., Zhang, J.: Learning deep facial expression features from image and optical flow sequences using 3D CNN. Vis. Comput. 34(10), 1461–1475 (2018)CrossRefGoogle Scholar
  49. 49.
    Zhao, J., Wang, S.H., Liu, X., Liu, Y., Chen, Y.Q.: Early diagnosis of cirrhosis via automatic location and geometric description of liver capsule. Vis. Comput. 34(12), 1677–1689 (2018)CrossRefGoogle Scholar
  50. 50.
    Zhen, Q., Huang, D., Drira, H., Amor, B.B., Wang, Y., Daoudi, M.: Magnifying subtle facial motions for effective 4D expression recognition. IEEE Trans. Affect. Comput. (2017). Google Scholar
  51. 51.
    Zhen, Q., Huang, D., Wang, Y., Chen, L.: Muscular movement model-based automatic 3D/4D facial expression recognition. IEEE Trans. Multimed. 18(7), 1438–1450 (2016)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Electrical and Electronic Engineering DepartmentEastern Mediterranean UniversityFamagustaTurkey
  2. 2.Faculty of EngineeringCyprus International UniversityNicosiaTurkey

Personalised recommendations