Skip to main content
SpringerLink
Log in

Facial Micro-expression Recognition Using Enhanced Temporal Feature-Wise Model

  • Conference paper
  • First Online:
Cyberspace Data and Intelligence, and Cyber-Living, Syndrome, and Health (CyberDI 2019, CyberLife 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1138))

Abstract

Automatic facial micro-expression recognition is challenging for the subtlety and transience in facial motion, and limited databases. Most researches focus on handcrafted techniques for facial micro-expression analysis on two-dimensional images. However, spatiotemporal facial feature representation is a critical issue for facial micro-expression recognition due to its short duration and subtle facial movement. To deeply extract the appearance characteristics and facial changes effectively from facial image sequences, a feature-wise deep learning model was proposed by applying temporal Convolutional Neural Network (3D-CNN) and Long Short-Term Memory (LSTM) to enhance temporal feature learning. There are two stages involved: (1) The CNN was extended to convolute along spatio and temporal simultaneously, to better represent the facial texture and motion. (2) The feature vector obtained by 3D-CNN was fed into LSTM for temporal enrichment. It was demonstrated that the proposed model achieved promising good performance on CASME II and SMIC databases on person-independent and cross-database experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Mehrabian, A.: Communication without words. Communication Theory, pp. 193-200 (2008)

    Chapter  Google Scholar 

  2. Haggard, E.A., Isaacs, K.S.: Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy, pp. 154–165. Springer US (1966). http://doi.org/10.1007/978-1-4684-6045-2_14

    Chapter  Google Scholar 

  3. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Article  Google Scholar 

  4. Frank, M.G., Herbasz, M., Sinuk, K., et al.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton, New York, New York City (2009)

    Google Scholar 

  5. Ojala, T., Harwood I.: A comparative study of texture measures with classification based on feature distributions. Pattern Recogn. 29(1), 51–59 (1996)

    Article  Google Scholar 

  6. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  7. Pfister, T., Zhao, G., Pietikainen, M.: Recognising spontaneous facial micro-expressions. In: International Conference on Computer Vision, pp.1449–1456. IEEE Computer Society (2011)

    Google Scholar 

  8. Li, X., Pfister, T., Huang, X., et al.: A spontaneous micro-expression database: inducement, collection and baseline. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6. IEEE Computer Society (2013)

    Google Scholar 

  9. Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H.: LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9003, pp. 525–537. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16865-4_34

    Chapter  Google Scholar 

  10. Shreve, M., Godavarthy, S., Goldgof, D., et al.: Macro- and micro-expression spotting in long videos using spotio-temporal strain. In: The Ninth IEEE International Conference on Automatic Face and Gesture Recognition (2011)

    Google Scholar 

  11. Liong, S., See, J., Wong, K., Phan, R.C.-W.: Less is more: Micro-expression recognition from video using apex frame. Sig. Process Image Commun. 62, 82–92 (2018)

    Google Scholar 

  12. Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 1-1 (2016)

    Google Scholar 

  13. Kahou, S.E., Michalski, V., Konda, K., et al.: Recurrent neural networks for emotion recognition in video. In: Acm International Conference on Multimodal Interaction, pp.467-474. ACM (2015)

    Google Scholar 

  14. Byeon, Y.H., Kwak, K.C.: Facial expression recognition using 3D convolutional neural network. Int. J. Adv. Comput. Sci. Appl. 5(12) (2014)

    Google Scholar 

  15. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3D convolutional neural networks. arXiv preprint arXiv:1705.07871 (2017)

  16. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 445–450. ACM (2016)

    Google Scholar 

  17. Zhu, G., Zhang, L., Shen, P., et al.: Multimodal gesture recognition using 3D convolution and convolutional LSTM. IEEE Access PP(99), 1–1 (2017)

    Google Scholar 

  18. Tran, D., Bourdev, L., Fergus, R., et al.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE International Conference on Computer Vision, pp. 4489–4497 (2014)

    Google Scholar 

  19. Ji, S., Xu, W., Yang, M., et al.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221-231 (2013)

    Article  Google Scholar 

  20. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: International Conference on Machine Learning, DBLP, pp. 807–814 (2010)

    Google Scholar 

  21. Irsoy, O., Cardie, C.: Deep recursive neural networks for compositionality in language. In: Advances in neural information processing systems, pp. 2096–2104 (2014)

    Google Scholar 

  22. Yan, W.J., Li, X., Wang, S.J, et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. Plos One 9(1), e86041 (2014)

    Article  Google Scholar 

  23. Zhang, S., Feng, B., Chen, Z., Huang, X.: Micro-expression recognition by aggregating local spatio-temporal patterns. In: Amsaleg, L., Guðmundsson, G.Þ., Gurrin, C., Jónsson, B.Þ., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 638–648. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51811-4_52

    Chapter  Google Scholar 

  24. Le Ngo, A.C, See, J., Phan, C.W.R.: Sparsity in Dynamics of Spontaneous Subtle Emotion: Analysis & Application. IEEE Trans. Affective Comput. 8(3), 396–411 (2017)

    Article  Google Scholar 

  25. Li, X., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2018)

    Article  Google Scholar 

  26. Mayya, V., Pai, R.M., Pai, M.M.M.: Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences. In: 2016 International Conference on. IEEE Advances in Computing, Communications and Informatics (ICACCI), pp. 699–703 (2016)

    Google Scholar 

  27. Kim, D.H., Baddar, W.J., Ro, Y.M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 382–386. ACM (2016)

    Google Scholar 

  28. Le Ngo, A.C., Phan, R.C.-W., See, J.: Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 33–48. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_3

    Chapter  Google Scholar 

  29. Liong, S.-T., See, J., Phan, R.C.-W., Le Ngo, A.C., Oh, Y.-H., Wong, K.: Subtle expression recognition using optical strain weighted features. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9009, pp. 644–657. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16631-5_47

    Chapter  Google Scholar 

  30. Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2258–2263. IEEE (2016).

    Google Scholar 

  31. Zong, Y., Huang, X., Zheng, W., et al.: Learning a target sample re-generator for cross-database micro-expression recognition. arXiv preprint arXiv:1707.08645 (2017)

Download references

Acknowledgments

This work was supported by the National Research and Development Major Project (2017YFD0400100), the National Natural Science Foundation of China (61673052), the Fundamental Research Fund for the Central Universities of China (2302018FRF-TP-18-014A2), and the grant from Chinese Scholarship Council (CSC).

Author information

Authors and Affiliations

  1. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, People’s Republic of China

    Ruicong Zhi, Mengyi Liu, Hairui Xu & Ming Wan

  2. Beijing Key Laboratory of Knowledge Engineering for Materials Science, Beijing, 100083, People’s Republic of China

    Ruicong Zhi, Mengyi Liu, Hairui Xu & Ming Wan

Authors
  1. Ruicong Zhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mengyi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hairui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruicong Zhi .

Editor information

Editors and Affiliations

  1. University of Science and Technology, Beijing, China

    Huansheng Ning

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhi, R., Liu, M., Xu, H., Wan, M. (2019). Facial Micro-expression Recognition Using Enhanced Temporal Feature-Wise Model. In: Ning, H. (eds) Cyberspace Data and Intelligence, and Cyber-Living, Syndrome, and Health. CyberDI CyberLife 2019 2019. Communications in Computer and Information Science, vol 1138. Springer, Singapore. https://doi.org/10.1007/978-981-15-1925-3_22

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-1925-3_22

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-1924-6

  • Online ISBN: 978-981-15-1925-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation