Improving Micro-expression Recognition Accuracy Using Twofold Feature Extraction

  • Madhumita A. Takalkar
  • Haimin Zhang
  • Min XuEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11295)


Micro-expressions are generated involuntarily on a person’s face and are usually a manifestation of repressed feelings of the person. Micro-expressions are characterised by short duration, involuntariness and low intensity. Because of these characteristics, micro-expressions are difficult to perceive and interpret correctly, and they are profoundly challenging to identify and categorise automatically.

Previous work for micro-expression recognition has used hand-crafted features like LBP-TOP, Gabor filter, HOG and optical flow. Recent work also has demonstrated the possible use of deep learning for micro-expression recognition. This paper is the first work to explore the use of hand-craft feature descriptor and deep feature descriptor for micro-expression recognition task. The aim is to use the hand-craft and deep learning feature descriptor to extract features and integrate them together to construct a large feature vector to describe a video. Through experiments on CASME, CASME II and CASME+2 databases, we demonstrate our proposed method can achieve promising results for micro-expression recognition accuracy with larger training samples.


Micro-expression recognition Deep learning Local binary pattern-three orthogonal planes (LBP-TOP) Convolutional neural network (CNN) Small training data Data augmentation 


  1. 1.
    Bernstein, D.M., Loftus, E.F.: How to tell if a particular memory is true or false. Perspect. Psychol. Sci. 4(4), 370–374 (2009)CrossRefGoogle Scholar
  2. 2.
    Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)CrossRefGoogle Scholar
  3. 3.
    Happy, S., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. (2017)Google Scholar
  4. 4.
    Huang, X., Wang, S.J., Zhao, G., Piteikainen, M.: Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1–9 (2015)Google Scholar
  5. 5.
    Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  6. 6.
    Kim, D.H., Baddar, W.J., Ro, Y.M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations, pp. 382–386. ACM (2016)Google Scholar
  7. 7.
    Li, X., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. (2017)Google Scholar
  8. 8.
    Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2016)CrossRefGoogle Scholar
  9. 9.
    Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: BMVC, vol. 1 (2015)Google Scholar
  10. 10.
    Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2258–2263. IEEE (2016)Google Scholar
  11. 11.
    Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)CrossRefGoogle Scholar
  12. 12.
    Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1449–1456. IEEE (2011)Google Scholar
  13. 13.
    Takalkar, M., Xu, M., Wu, Q., et al.: A survey: facial micro-expression recognition. Multimed. Tools Appl. 77, 19301 (2018). Scholar
  14. 14.
    Takalkar, M.A., Xu, M.: Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7. IEEE (2017)Google Scholar
  15. 15.
    Vasconcellos, S.J.L., Salvador-Silva, R., Gauer, V., Gauer, G.J.C.: Psychopathic traits in adolescents and recognition of emotion in facial expressions. Psicologia: Reflexão e. Crítica 27(4), 768–774 (2014)Google Scholar
  16. 16.
    Wang, S.J., Yan, W.J., Li, X., Zhao, G., Fu, X.: Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd International Conference on Pattern Recognition (ICPR), pp. 4678–4683. IEEE (2014)Google Scholar
  17. 17.
    Wang, Y., et al.: Effective recognition of facial micro-expressions with video motion magnification. Multimed. Tools Appl. 76(20), 21665–21690 (2017)CrossRefGoogle Scholar
  18. 18.
    Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 254–267 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Electrical and Data Engineering, Faculty of Engineering and Information TechnologyUniversity of Technology SydneyUltimoAustralia

Personalised recommendations