Advertisement

Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Micro-expression recognition: an updated review of current trends, challenges and solutions

  • 504 Accesses

  • 3 Citations

Abstract

Micro-expression (ME) recognition has attracted numerous interests within the computer vision circle in different contexts particularly, localization, magnification, and recognition. Challenges in these areas remain relevant due to the nature of ME’s split-second transition with minute intensity levels. In this paper, a comprehensive state-of-the-art analysis of ME recognition and detection challenges are provided. Contemporary solutions are categorized into low-level, mid-level, and high-level solutions with a review of their characteristics and performances. This paper also provides possible extensions to basic methods, highlight, and predict emerging trends. A thorough analysis of mainstream ME datasets is also provided by elucidating each of their advantages and limitations. This survey gives readers an understanding of ME recognition and an appreciation of future research direction in ME recognition systems.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. 1.

    Zhang, M., Fu, Q., Chen, Y.-H., Fu, X.: Emotional context influences micro-expression recognition. PLoS ONE 9(4), e95018 (2014)

  2. 2.

    Takalkar, M., Xu, M., Wu, Q., Chaczko, Z.: A survey: facial micro-expression recognition. Multimed. Tools Appl. 77, 301 (2017)

  3. 3.

    Ekman, P.: Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000(1), 205–221 (2003)

  4. 4.

    Ekman, P.: Micro expression training tool (METT) and subtle expression training tool (SETT). Paul Ekman Company, San Francisco (2003)

  5. 5.

    Li, X., Xiaopeng, H., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikainen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect Comput. PP(99), 1–1 (2017)

  6. 6.

    Wen-Jing, Y., Wu, Q., Yong-Jin, L., Su-Jing, W., Fu, X.: CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 22–26 April 2013, pp. 1–7

  7. 7.

    Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Fu, X.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

  8. 8.

    Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 22–26 April 2013, pp. 1–6

  9. 9.

    Ekman, P., Friesen, W.V.: Facial Action Coding System, vol. 1. Consulting Psychologists Press, Washington (1978)

  10. 10.

    Tian, Y.I., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)

  11. 11.

    Frank, M., Herbasz, M., Sinuk, K., Keller, A., Nolan, C.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City (2009)

  12. 12.

    Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., Fu, X.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)

  13. 13.

    Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect Comput. 8(2), 254–267 (2017)

  14. 14.

    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

  15. 15.

    Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)

  16. 16.

    Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., Sarkar, S.: Automatic expression spotting in videos. Image Vis. Comput. 32(8), 476–486 (2014)

  17. 17.

    Qu, F., Wang, S.J., Yan, W.J., Li, H., Wu, S., Fu, X.: CAS(ME)2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect Comput. PP(99), 1–1 (2017). https://doi.org/10.1109/taffc.2017.2654440

  18. 18.

    Yan, W.-J., Wang, S.-J., Chen, Y.-H., Zhao, G., Fu, X.: Quantifying micro-expressions with constraint local model and local binary pattern. In: Cham 2015. Computer Vision—ECCV 2014 Workshops, pp. 296–305. Springer

  19. 19.

    Porter, S., Brinke, L.T.: Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychol. Sci. 19(5), 508–514 (2008). https://doi.org/10.1111/j.1467-9280.2008.02116.x

  20. 20.

    Yan, W.-J., Wang, S.-J., Liu, Y.-J., Wu, Q., Fu, X.: For micro-expression recognition: database and suggestions. Neurocomputing 136, 82–87 (2014)

  21. 21.

    Wang, Y., See, J., Oh, Y.-H., Phan, R.C.-W., Rahulamathavan, Y., Ling, H.-C., Tan, S.-W., Li, X.: Effective recognition of facial micro-expressions with video motion magnification. Multimed. Tools Appl. 76(20), 21665–21690 (2017)

  22. 22.

    Ngo, A.C.L., Oh, Y.H., Phan, R.C.W., See, J.: Eulerian emotion magnification for subtle expression recognition. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 20–25 March 2016, pp. 1243–1247

  23. 23.

    Le Ngo, A.C., Phan, R.C.-W., See, J.: Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Cham 2015. Computer Vision—ACCV 2014, pp. 33–48. Springer

  24. 24.

    Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009), 3–3 Dec 2009, pp. 1–6

  25. 25.

    Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro- and micro-expression spotting in long videos using spatio-temporal strain. Face Gesture 2011(21–25), 51–56 (2011)

  26. 26.

    Warren, G., Schertler, E., Bull, P.: Detecting deception from emotional and unemotional cues. J. Nonverbal Behav. 33(1), 59–69 (2009)

  27. 27.

    Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect Comput. PP(99), 1–1 (2016)

  28. 28.

    Yu, Z., Liu, Q., Liu, G.: Deeper cascaded peak-piloted network for weak expression recognition. Vis. Comput. 1–9 (2017)

  29. 29.

    Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Trans. Inf. Syst. E96.D(1), 81–92 (2013)

  30. 30.

    Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

  31. 31.

    Sariyanidi, E., Gunes, H., Cavallaro, A.: Robust registration of dynamic facial sequences. IEEE Trans. Image Process. 26(4), 1708–1722 (2017)

  32. 32.

    Wang, Y., See, J., Phan, R.C.W., Oh, Y.-H.: Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS ONE 10(5), e0124674 (2015)

  33. 33.

    Le Ngo, A.C., See, J., Phan, R.C.W.: Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Trans. Affect Comput. 8(3), 396–411 (2017)

  34. 34.

    Liong, S.-T., See, J., Phan, R.C.-W., Le Ngo, A.C., Oh, Y.-H., Wong, K.: Subtle expression recognition using optical strain weighted features. In: Cham 2015. Computer Vision—ACCV 2014 Workshops, pp. 644–657. Springer

  35. 35.

    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 2001, vol. 511, pp. I-511–I-518

  36. 36.

    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, 16–21 June 2012, pp. 2879–2886

  37. 37.

    Chen, D., Ren, S., Wei, Y., Cao, X., Sun, J.: Joint cascade face detection and alignment. In: Cham 2014. Computer Vision—ECCV 2014, pp. 109–122. Springer

  38. 38.

    Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y.: Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 16(5), 555–559 (2003)

  39. 39.

    Liao, S., Jain, A.K., Li, S.Z.: A fast and accurate unconstrained face detector. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 211–223 (2016)

  40. 40.

    Ranjan, R., Patel, V.M., Chellappa, R.: A deep pyramid deformable part model for face detection. In: IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), 8–11 Sept 2015, pp. 1–8

  41. 41.

    Yang, S., Luo, P., Loy, C.C., Tang, X.: From facial parts responses to face detection: a deep learning approach. In: IEEE International Conference on Computer Vision (ICCV), 7–13 Dec 2015, pp. 3676–3684

  42. 42.

    Li, H., Lin, Z., Shen, X., Brandt, J., Hua, G.: A convolutional neural network cascade for face detection. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 June 2015, pp. 5325–5334 (2015)

  43. 43.

    Ranjan, R., Patel, V.M., Chellappa, R.: HyperFace: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1–1 (2017)

  44. 44.

    Yan, J., Zhang, X., Lei, Z., Li, S.Z.: Face detection by structural models. Image Vis. Comput. 32(10), 790–799 (2014)

  45. 45.

    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

  46. 46.

    Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: IEEE International Conference on Computer Vision (ICCV) 2011, pp. 1449–1456. IEEE

  47. 47.

    Wang, S.-J., Yan, W.-J., Zhao, G., Fu, X., Zhou, C.-G.: Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) Computer Vision—ECCV 2014 Workshops: Zurich, Switzerland, September 6–7 and 12, 2014, Proceedings, Part I. pp. 325–338. Springer, Cham (2015)

  48. 48.

    Ruiz-Hernandez, J.A., Pietikäinen, M.: Encoding local binary patterns using the re-parametrization of the second order Gaussian jet. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 22–26 April 2013, pp. 1–6

  49. 49.

    Wang, S.J., Yan, W.J., Li, X., Zhao, G., Zhou, C.G., Fu, X., Yang, M., Tao, J.: Micro-expression recognition using color spaces. IEEE Trans. Image Process. 24(12), 6034–6047 (2015)

  50. 50.

    Wang, S.J., Yan, W.J., Li, X., Zhao, G., Fu, X.: Micro-expression recognition using dynamic textures on tensor independent color space. In: 22nd International Conference on Pattern Recognition, 24–28 Aug 2014, pp. 4678–4683

  51. 51.

    Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M.: Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175, 564–578 (2016)

  52. 52.

    Huang, X.H., Wang, S.J., Liu, X., Zhao, G., Feng, X., Pietikainen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect Comput. PP(99), 1–1 (2017)

  53. 53.

    Xia, Z., Feng, X., Peng, J., Peng, X., Zhao, G.: Spontaneous micro-expression spotting via geometric deformation modeling. Comput. Vis. Image Underst. 147, 87–94 (2016)

  54. 54.

    Huang, X., Wang, S.J., Zhao, G., Piteikäinen, M.: Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: IEEE International Conference on Computer Vision Workshop (ICCVW), 7–13 Dec 2015, pp. 1–9

  55. 55.

    Liu, Q., Deng, J., Tao, D.: Dual sparse constrained cascade regression for robust face alignment. IEEE Trans. Image Process. 25(2), 700–712 (2016)

  56. 56.

    Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7–12 June 2015, pp. 3659–3667

  57. 57.

    Xiong, X., Torre, F.D.L.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, 23–28 June 2013, pp. 532–539

  58. 58.

    Gines Hidalgo, Z.C., Tomas, S., Shih-En, W., Hanbyul, J., Yaser, S.: OpenPose Library. https://github.com/CMU-Perceptual-Computing-Lab/openpose (2017). 2018

  59. 59.

    Simon, T., Joo, H., Matthews, I., Sheikh, Y.: Hand keypoint detection in single images using multiview bootstrapping. In: CVPR 2017, p. 4

  60. 60.

    Deng, W., Fang, Y., Xu, Z., Hu, J.: Facial landmark localization by enhanced convolutional neural network. Neurocomputing 273, 222–229 (2018)

  61. 61.

    Bian, P., Xie, Z., Jin, Y.: Multi-task feature learning-based improved supervised descent method for facial landmark detection. SIViP 12(1), 17–24 (2018)

  62. 62.

    Pan, W., Qin, K., Chen, Y.: An adaptable-multilayer fractional fourier transform approach for image registration. IEEE Trans. Pattern Anal. Mach. Intell. 31(3), 400–414 (2009)

  63. 63.

    Kumar, S., Azartash, H., Biswas, M., Nguyen, T.: Real-time affine global motion estimation using phase correlation and its application for digital image stabilization. IEEE Trans. Image Process. 20(12), 3406–3418 (2011)

  64. 64.

    Xiong, X., Qin, K.: Linearly estimating all parameters of affine motion using radon transform. IEEE Trans. Image Process. 23(10), 4311–4321 (2014)

  65. 65.

    Lucas, B.D.: Generalized image matching by the method of differences. (1985)

  66. 66.

    Baker, S., Matthews, I.: Lucas–Kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)

  67. 67.

    Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: Robust and efficient parametric face alignment. In: International Conference on Computer Vision, 6–13 Nov 2011, pp. 1847–1854

  68. 68.

    Evangelidis, G.D., Psarakis, E.Z.: Parametric image alignment using enhanced correlation coefficient maximization. IEEE Trans. Pattern Anal. Mach. Intell. 30(10), 1858–1865 (2008)

  69. 69.

    Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: IEEE International Conference on Computer Vision Workshops (ICCV Workshops), 6–13 Nov 2011, pp. 868–875

  70. 70.

    Gunn, S.R.: Support vector machines for classfication and regression. ISIS Tech. Rep. 14(1), 5–16 (1998)

  71. 71.

    Guo, Y., Tian, Y., Gao, X., Zhang, X.: Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. In: 2014 International Joint Conference on Neural Networks (IJCNN), 6–11 July 2014, pp. 3473–3479

  72. 72.

    Happy, S.L., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect Comput. PP(99), 1–1 (2017)

  73. 73.

    Ben, X., Zhang, P., Yan, R., Yang, M., Ge, G.: Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput. Appl. 27(8), 2629–2646 (2016)

  74. 74.

    Wang, S.-J., Chen, H.-L., Yan, W.-J., Chen, Y.-H., Fu, X.: Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process. Lett. 39(1), 25–43 (2014)

  75. 75.

    Adegun, I.P., Vadapalli, H.B.: Automatic recognition of micro-expressions using local binary patterns on three orthogonal planes and extreme learning machine. In: Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech), 30 Nov 2016–2 Dec 2016, pp. 1–5

  76. 76.

    Guo, Y., Xue, C., Wang, Y., Yu, M.: Micro-expression recognition based on CBP-TOP feature with ELM. Opt. Int. J. Light Electron Opt. 126(23), 4446–4451 (2015)

  77. 77.

    Izenman, A.J.: Linear discriminant analysis. In: Casella, G., Fienberg, S., Olkin, I. (eds.) Modern Multivariate Statistical Techniques, pp. 237–280. Springer, Cham (2013)

  78. 78.

    Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)

  79. 79.

    Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), 4–8 Dec 2016, pp. 2258–2263

  80. 80.

    Kim, D.H., Baddar, W., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect Comput. 66(99), 1–1 (2017)

  81. 81.

    Takalkar, M.A., Xu, M.: Image based facial micro-expression recognition using deep learning on small datasets. In: International Conference on Digital Image Computing: Techniques and Applications (DICTA), 29 Nov 2017–1 Dec 2017, pp. 1–7

  82. 82.

    Lim, C.H., Goh, K.M.: Fuzzy qualitative approach for micro-expression recognition. In: Asic-Pasific Signal and Information Processing Association Annual Submit and Conference Kuala Lumpur, 12nd–15th Dec 2017

  83. 83.

    Lim, C.H., Risnumawan, A., Chan, C.S.: A scene image is nonmutually exclusive—a fuzzy qualitative scene understanding. IEEE Trans. Fuzzy Syst. 22(6), 1541–1556 (2014)

  84. 84.

    Liu, H., Coghill, G.M., Barnes, D.P.: Fuzzy qualitative trigonometry. Int. J. Approx. Reason. 51(1), 71–88 (2009)

  85. 85.

    Shen, Q., Leitch, R.: Fuzzy qualitative simulation. IEEE Trans. Syst. Man Cybern. 23(4), 1038–1061 (1993)

  86. 86.

    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

  87. 87.

    House, C., Meyer, R.: Preprocessing and descriptor features for facial micro-expression recognition (2015)

  88. 88.

    Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H.: LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition. In: Cham 2015. Computer Vision—ACCV 2014, pp. 525–537. Springer

  89. 89.

    Mayya, V., Pai, R.M., Pai, M.M.M.: Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences. In: International Conference on Advances in Computing, Communications and Informatics (ICACCI), 21–24 Sept 2016, pp. 699–703

  90. 90.

    Lee, S.H., Kim, H., Ro, Y.M., Plataniotis, K.N.: Using color texture sparsity for facial expression recognition. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 22–26 April 2013, pp. 1–6

  91. 91.

    Liong, S.-T., Phan, R.C.-W., See, J., Oh, Y.-H., Wong, K.: Optical strain based recognition of subtle emotions. In: International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) 2014, pp. 180–184. IEEE

  92. 92.

    Liong, S.-T., See, J., Phan, R.C.W., Oh, Y.-H., Cat Le Ngo, A., Wong, K., Tan, S.-W.: Spontaneous subtle expression detection and recognition based on facial strain. Signal Process. Image Commun. 47, 170–182 (2016)

  93. 93.

    Ben, X., Jia, X., Yan, R., Zhang, X., Meng, W.: Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn. Lett. 107, 50–58 (2017)

  94. 94.

    Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: IEEE Conference on Computer Vision and Pattern Recognition, 20–25 June 2009, pp. 1932–1939

  95. 95.

    Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

  96. 96.

    Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect Comput. 7(4), 299–310 (2016)

  97. 97.

    Oh, Y.H., Ngo, A.C.L., See, J., Liong, S.T., Phan, R.C.W., Ling, H.C.: Monogenic Riesz wavelet representation for micro-expression recognition. In: IEEE International Conference on Digital Signal Processing (DSP), 21–24 July 2015, pp. 1237–1241

  98. 98.

    Zheng, H.: Micro-expression recognition based on 2D Gabor filter and sparse representation. J. Phys. Conf. Ser. 787(1), 012013 (2017)

  99. 99.

    Chengjun, L., Wechsler, H.: Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition. IEEE Trans. Image Process. 11(4), 467–476 (2002)

  100. 100.

    Kamarol, S.K.A., Jaward, M.H., Parkkinen, J., Parthiban, R.: Spatiotemporal feature extraction for facial expression recognition. IET Image Process. 10, 534 (2016)

  101. 101.

    Lu, Z., Luo, Z., Zheng, H., Chen, J., Li, W.: A Delaunay-based temporal coding model for micro-expression recognition. In: Cham 2015. Computer Vision—ACCV 2014 Workshops, pp. 698–711. Springer

  102. 102.

    Sariyanidi, E., Gunes, H., Cavallaro, A.: Learning bases of activity for facial expression recognition. IEEE Trans. Image Process. 26(4), 1965–1978 (2017)

  103. 103.

    He, J., Hu, J.-F., Lu, X., Zheng, W.-S.: Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognit. 66, 44–52 (2017)

  104. 104.

    Sikka, K., Wu, T., Susskind, J., Bartlett, M.: Exploring bag of words architectures in the facial expression domain. In: Berlin, Heidelberg 2012. Computer Vision—ECCV 2012. Workshops and Demonstrations, pp. 250–259. Springer, Berlin

  105. 105.

    Zheng, H., Geng, X., Yang, Z.: A Relaxed K-SVD algorithm for spontaneous micro-expression recognition. In: Cham 2016. PRICAI 2016: Trends in Artificial Intelligence, pp. 692–699. Springer

  106. 106.

    Zheng, H., Zhu, J., Yang, Z., Jin, Z.: Effective micro-expression recognition using relaxed K-SVD algorithm. Int. J. Mach. Learn. Cybern. 8(6), 2043–2049 (2017)

  107. 107.

    Pei, L., Ye, M., Zhao, X., Dou, Y., Bao, J.: Action recognition by learning temporal slowness invariant features. Vis. Comput. 32(11), 1395–1404 (2016)

  108. 108.

    Wu, H., Zou, B., Zhao, Y.-Q., Guo, J.: Scene text detection using adaptive color reduction, adjacent character model and hybrid verification strategy. Vis. Comput. 33(1), 113–126 (2017)

  109. 109.

    Bi, L., Kim, J., Kumar, A., Fulham, M., Feng, D.: Stacked fully convolutional networks with multi-channel learning: application to medical image segmentation. Vis. Comput. 33(6), 1061–1071 (2017)

  110. 110.

    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the Devil in the Details: Delving Deep into Convolutional Nets. CoRR abs/1405.3531 (2014)

  111. 111.

    Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)

  112. 112.

    Hong, X., Tran, T.-K., Zhao, G.: Micro-Expression Spotting: A Benchmark. CoRR abs/1710.02820 (2017)

  113. 113.

    Tran, T.-K., Hong, X., Zhao, G.: Sliding window based micro-expression spotting: a benchmark. In: Cham 2017. Advanced Concepts for Intelligent Vision Systems, pp. 542–553. Springer

  114. 114.

    Wu, Q., Shen, X., Fu, X.: The machine knows what you are hiding: an automatic micro-expression recognition system. In: Berlin, Heidelberg 2011. Affective Computing and Intelligent Interaction, pp. 152–162. Springer, Berlin

  115. 115.

    Moilanen, A., Zhao, G., Pietikäinen, M.: Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: 22nd International Conference on Pattern Recognition, 24–28 Aug 2014, pp. 1722–1727

  116. 116.

    Davison, A.K., Yap, M.H., Lansley, C.: Micro-facial movement detection using individualised baselines and histogram-based descriptors. In: IEEE International Conference on Systems, Man, and Cybernetics, 9–12 Oct 2015, pp. 1864–1869

  117. 117.

    Davison, A.K., Lansley, C., Ng, C.-C., Tan, K., Yap, M.H.: Objective micro-facial movement detection using FACS-based regions and baseline evaluation. CoRR abs/1612.05038 (2016)

  118. 118.

    Lu, H., Kpalma, K., Ronsin, J.: Micro-expression detection using integral projections. J WSCG 25(2), 87–96 (2017)

  119. 119.

    Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S.: Towards macro- and micro-expression spotting in video using strain patterns. In: Workshop on Applications of Computer Vision (WACV), 7–8 Dec 2009, pp. 1–6

  120. 120.

    Wang, S.-J., Wu, S., Qian, X., Li, J., Fu, X.: A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing 230, 382–389 (2017)

  121. 121.

    Li, X., Yu, J., Zhan, S.: Spontaneous facial micro-expression detection based on deep learning. In: IEEE 13th International Conference on Signal Processing (ICSP), 6–10 Nov 2016, pp. 1130–1134

  122. 122.

    Duque, C., Alata, O., Emonet, R., Legrand, A.-C., Konik, H.: Micro-expression spotting using the Riesz pyramid. In: WACV 2018 (2018)

  123. 123.

    Wang, S.-J., Yan, W.-J., Zhao, G., Fu, X., Zhou, C.-G.: Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In: Cham 2015. Computer Vision—ECCV 2014 Workshops, pp. 325–338. Springer

  124. 124.

    Parkinson, C., Walker, T.T., Memmi, S., Wheatley, T.: Emotions are understood from biological motion across remote cultures. Emotion 17(3), 459–477 (2017)

Download references

Acknowledgements

This work is supported by Ministry of Higher Education (MOHE), Malaysia, under Fundamental Research Grant Scheme (FRGS) (Reference code: FRGS/1/2016/TK04/TARUC/02/1).

Author information

Correspondence to Kam Meng Goh.

Ethics declarations

Conflict of interest

The author declares that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Goh, K.M., Ng, C.H., Lim, L.L. et al. Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis Comput 36, 445–468 (2020). https://doi.org/10.1007/s00371-018-1607-6

Download citation

Keywords

  • Classification
  • Dataset
  • Feature extraction
  • Micro-expression
  • Pre-processing
  • Spotting