Anubhav: recognizing emotions through facial expression


We present a computer vision-based system named Anubhav (a Hindi word meaning feeling) which recognizes emotional facial expressions from streaming face videos. Our system runs at a speed of 10 frames per second (fps) on a 3.2-GHz desktop and at 3 fps on an Android mobile device. Using entropy and correlation-based analysis, we show that some particular salient regions of face image carry major expression-related information compared with other face regions. We also show that spatially close features within a salient face region carry correlated information regarding expression. Therefore, only a few features from each salient face region are enough for expression representation. Extraction of only a few features considerably saves response time. Exploitation of expression information from spatial as well as temporal dimensions gives good recognition accuracy. We have done extensive experiments on two publicly available data sets and also on live video streams. The recognition accuracies on benchmark CK\(+\) data set and on live video stream by our system are at least 13 and 20 % better, respectively, compared to competing approaches.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 199

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13


  1. 1.


  1. 1. Accessed on 14 Sept 2016

  2. 2.

    Aldebaran: Accessed on 26 March 2016

  3. 3.

    Agarwal, S., Chatterjee, M., Mukherjee, D.P.: Recognizing facial expressions using a novel shape motion descriptor. In: Indian Conference on Vision, Graphics and Image Processing. ACM (2012). doi:10.1145/2425333.2425362

  4. 4.

    Agarwal, S., Mukherjee, D.P.: Decoding mixed emotions from expression map of face images. In: International Conference and Workshops on Automatic Face and Gesture Recognition. IEEE (2013). doi:10.1109/FG.2013.6553731

  5. 5.

    Agarwal, S., Umer, S., Mukherjee, D.P.: Mp-feg: media player controlled by facial expressions and gestures. In: National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics. IEEE (2015)

  6. 6.

    Aifanti, N., Papachristou, C., Delopoulos, A.: The mug facial expression database. In: International Workshop on Image Analysis for Multimedia Interactive Services, pp. 76–84 (2010)

  7. 7.

    An, K.H., Chung, M.J.: Cognitive face analysis system for future interactive tv. IEEE Trans. Consum. Electron. 55(4), 2271–2279 (2009)

  8. 8.

    Anand, B., Navathe, B., Velusamy, S., Kannan, H., Sharma, A., Gopalakrishnan, V.: Beyond touch: natural interactions using facial expressions. In: Consumer Communications and Networking Conference, pp. 255–259. IEEE (2012). doi:10.1109/CCNC.2012.6181097

  9. 9.

    Bacivarov, I., Corcoran, P., Ionita, M.: Smart cameras: 2d affine models for determining subject facial expressions. IEEE Trans. Consum. Electron. 56(2), 289–297 (2010)

  10. 10.

    Baltrusaitis, T., McDuff, D., Banda, N., Mahmoud, M., El Kaliouby, R., Robinson, P., Picard, R.: Real-time inference of mental states from facial expressions and upper body gestures. In: International Conference on Automatic Face Gesture Recognition and Workshops, pp. 909–914. IEEE (2011). doi:10.1109/FG.2011.5771372

  11. 11.

    Bejani, M., Gharavian, D., Charkari, N.M.: Audiovisual emotion recognition using anova feature selection method and multi-classifier neural networks. Neural Comput. Appl. 24(2), 399–412 (2014)

  12. 12.

    Berretti, S., Amor, B.B., Daoudi, M., Del Bimbo, A.: 3d facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021–1036 (2011)

  13. 13.

    Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3d face scans. Vis. Comput. 29(12), 1333–1350 (2013)

  14. 14.

    Boughrara, H., Chtourou, M., Amar, C.B., Chen, L.: Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed. Tools Appl. 75(2), 709–731 (2016)

  15. 15.

    Bradski, G.: The opencv library. Dr. Dobb’s J. 25(11), 120–126 (2000)

  16. 16.

    Chew, S.W., Rana, R., Lucey, P., Lucey, S., Sridharan, S.: Sparse temporal representations for facial expression recognition. In: Yo-sung, H. (ed.) Advances in Image and Video Technology, pp. 311–322. Springer (2012)

  17. 17.

    Danelakis, A., Theoharis, T., Pratikakis, I.: A spatio-temporal wavelet-based descriptor for dynamic 3d facial expression retrieval and recognition. Vis. Comput. 1–11 (2016). doi:10.1007/s00371-016-1243-y

  18. 18.

    Darwin, C.: The Expression of the Emotions in Man and Animals, anniversary edn. Harper Perennial (1872/2009).

  19. 19.

    Duchenne, G.B.: Mecanisme de la Physionomie Humaine, ou analyse electro-physiologique de I’expression des passions. Jules Renouard, Paris (1862)

  20. 20.

    Ekman, P., Friesen, W.V.: The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press Inc., San Francisco (1978)

  21. 21.

    Erden, F., Cetin, A.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014)

  22. 22.

    Esau, N., Wetzel, E., Kleinjohann, L., Kleinjohann, B.: Real-time facial expression recognition using a fuzzy emotion model. In: International Fuzzy Systems Conference, pp. 1–6. IEEE (2007)

  23. 23.

    Hou, X.N., Ding, S.H., Ma, L.Z., Wang, C.J., Li, J.L., Huang, F.Y.: Similarity metric learning for face verification using sigmoid decision function. Vis. Comput. 32(4), 479–490 (2016)

  24. 24.

    Hsu, F.S., Lin, W.Y., Tsai, T.W.: Facial expression recognition using bag of distances. Multimed. Tools Appl. 73(1), 309–326 (2014)

  25. 25.

    Jeong, J.W., Lee, D.H.: Inferring search intents from remote control movement patterns: a new content search method for smart tv. IEEE Trans. Consum. Electron. 60(1), 92–98 (2014)

  26. 26.

    Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection uing sparse appearance descriptors in space-time video volumes. In: International Conference on Automatic Face and Gesture Recognition and Workshops, pp. 314–321. IEEE (2011). doi:10.1109/FG.2011.5771416

  27. 27.

    Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: International Conference on Automatic Face and Gesture Recognition, pp. 46–53. IEEE (2000). doi:10.1109/AFGR.2000.840611

  28. 28.

    Khan, R.A., Meyer, A., Konik, H., Bouakaz, S.: Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit. Lett. 34(10), 1159–1168 (2013)

  29. 29.

    Kotsia, I., Pitas, I.: Real time facial expression recognition from image sequences using support vector machines. In: Visual Communications and Image Processing 2005, pp. 59,602E–59,602E. International Society for Optics and Photonics (2005)

  30. 30.

    Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6(1), 159–169 (2012)

  31. 31.

    Li, H., Buenaposada, J.M., Baumela, L.: Real-time facial expression recognition with illumination-corrected image sequences. In: International Conference on Automatic Face & Gesture Recognition, pp. 1–6. IEEE (2008). doi:10.1109/AFGR.2008.4813328

  32. 32.

    Lian, S., Hu, W., Wang, K.: Automatic user state recognition for hand gesture based low-cost television control system. IEEE Trans. Consum. Electron. 60(1), 107–115 (2014)

  33. 33.

    Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: International Conference on Automatic Face Gesture Recognition and Workshops, pp. 298–305 (2011). doi:10.1109/FG.2011.5771414

  34. 34.

    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Computer Vision and Pattern Recognition, pp. 94–101 (2010)

  35. 35.

    Maeda, A., Kobayashi, M.: Trace select: acquiring remote targets with gesture guides generated from glyphs. IEEE Trans. Consum. Electron. 60(3), 453–460 (2014)

  36. 36.

    Rashid, M., Abu-Bakar, S., Mokji, M.: Human emotion recognition from videos using spatio-temporal and audio features. Vis. Comput. 29(12), 1269–1275 (2013)

  37. 37.

    Sanchez, A., Ruiz, J.V., Moreno, A.B., Montemayor, A.S., Hernndez, J., Pantrigo, J.J.: Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74, 1272–1282 (2011)

  38. 38.

    Shan, C., Gong, S., McOwan, P.W.: Dynamic facial expression recognition using a bayesian temporal manifold model. In: British Machine Vision Conference, pp. 297–306. Citeseer (2006)

  39. 39.

    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009). doi:10.1016/j.imavis.2008.08.005

  40. 40.

    Suk, M., Prabhakaran, B.: Real-time mobile facial expression recognition system—a case study, pp. 132–137. IEEE (2014)

  41. 41.

    de la Torre, F., Chu, W.S., Xiong, X., Vicente, F., Ding, X., Cohn, J.: Intraface. In: International Conference and Workshops on Automatic Face and Gesture Recognition, vol. 1, pp. 1–8. IEEE (2015). doi:10.1109/FG.2015.7163082

  42. 42.

    Truong, A., Boujut, H., Zaharia, T.: Laban descriptors for gesture recognition and emotional analysis. Vis. Comput. 32(1), 83–98 (2016)

  43. 43.

    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

  44. 44.

    Wang, J., Yin, L.: Static topographic modeling for facial expression recognition and analysis. Comput. Vis. Image Underst. 108(1), 19–34 (2007)

  45. 45.

    Wang, Z., Miao, Z., Wu, Q.J., Wan, Y., Tang, Z.: Low-resolution face recognition: a review. Vis. Comput. 30(4), 359–386 (2014)

  46. 46.

    Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 2106–2111 (2009). doi:10.1109/TPAMI.2009.42

  47. 47.

    Xiao, R., Zhao, Q., Zhang, D., Shi, P.: Facial expression recognition on multiple manifolds. Pattern Recognit. 44(1), 107–116 (2011)

  48. 48.

    Xiong, X., Torre, F.: Supervised descent method and its applications to face alignment. In: Conference on Computer Vision and Pattern Recognition, pp. 532–539. IEEE (2013)

  49. 49.

    Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86,041 (2014)

  50. 50.

    Zhang, X., Yin, L., Hipp, D., Gerhardstein, P.: Evaluation of perceptual biases in facial expression recognition by humans and machines. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., McMahan, R.,Jerald, J., Zhang, H., Drucker, S. M., Kambhamettu, C., ElChoubassi, M., Deng, Z., Carlson, M. (eds.) Advances in Visual Computing, pp. 809–819. Springer Switzerland (2014)

  51. 51.

    Zhao, G., Pietikinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

Download references


The work of Swapna Agarwal was supported by Department of Science and Technology, Government of India Project No. SR/WOS-A/ET-53/2012(G).

Author information

Correspondence to Swapna Agarwal.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 11948 KB)

Supplementary material 1 (avi 11948 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Agarwal, S., Santra, B. & Mukherjee, D.P. Anubhav: recognizing emotions through facial expression. Vis Comput 34, 177–191 (2018) doi:10.1007/s00371-016-1323-z

Download citation


  • Anubhav
  • Facial expression recognition
  • Entropy
  • Salient face region
  • Local binary pattern