Signal, Image and Video Processing

, Volume 12, Issue 6, pp 1141–1148 | Cite as

An efficient approach for facial action unit intensity detection using distance metric learning based on cosine similarity

  • Neeru Rathee
  • Dinesh Ganotra
Original Paper


Emotions of human beings are largely represented by facial expressions. Facial expressions, simple as well as complex, are well decoded by facial action units. Any facial expression can be detected and analyzed if facial action units are decoded well. In the presented work, an attempt has been made to detect facial action unit intensity by mapping the features based on their cosine similarity. Distance metric learning based on cosine similarity maps the data by learning a metric that measures orientation rather than magnitude. The motivation behind using cosine similarity is that change in facial expressions can be better represented by changes in orientation as compared to the magnitude. The features are applied to support vector machine for classification of various intensities of action units. Experimental results on the popularly accepted database such as DISFA database and UNBC McMaster shoulder pain database confirm the efficacy of the proposed approach.


Facial feature descriptors Support vector machine Distance metric learning Cosine similarity Facial action units intensity detection 


  1. 1.
    Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek, C., Fasel, I.R., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. J. Multimed. 1(6), 22–35 (2006)Google Scholar
  2. 2.
    Bellet, A., Habrard, A., Sebban, M.: A survey on metric learning for feature vectors and structured data. CoRR abs/1306.6709 (2013).
  3. 3.
    Bingol, D., Çelik, T., Omlin, C.W., Vadapalli, H.B.: Facial action unit intensity estimation using rotation invariant features and regression analysis. In: 2014 IEEE International Conference on Image Processing, ICIP 2014, Paris, France, pp. 1381–1385 (2014).
  4. 4.
    Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE Trans. Pattern Anal. Mach. Intell. 11(6), 567–585 (1989)CrossRefzbMATHGoogle Scholar
  5. 5.
    Buciu, I., Kotropoulos, C., Pitas, I.: Comparison of ICA approaches for facial expression recognition. Signal, Image Video Process. 3(4), 345 (2008). CrossRefzbMATHGoogle Scholar
  6. 6.
    Chen, J., Takiguchi, T.: Ariki Y (2017) Rotation-reversal invariant hog cascade for facial expression recognition. Signal, Image Video Process. 11(8), 1485–1492 (2017). CrossRefGoogle Scholar
  7. 7.
    Ekman, P., Friesen, W.V.: Measuring facial movement. Environ. Psychol. Nonverbal Behav. 1(1), 56–75 (1976)CrossRefGoogle Scholar
  8. 8.
    Hammal, Z., Kunz, M.: Pain monitoring: a dynamic and context-sensitive system. Pattern Recognit. 45(4), 1265–1280 (2012)CrossRefGoogle Scholar
  9. 9.
    Kaltwang, S., Rudovic, O., Pantic, M.: Continuous pain intensity estimation from facial expressions. In: Advances in Visual Computing, pp. 368–377. Springer, Berlin (2012)Google Scholar
  10. 10.
    Lajevardi, S.M.: Structural similarity classifier for facial expression recognition. Signal, Image Video Process. 8(6), 1103–1110 (2014). CrossRefGoogle Scholar
  11. 11.
    Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal, Image Video Process. 6(1), 159–169 (2012). CrossRefGoogle Scholar
  12. 12.
    Li, Y., Chen, J., Zhao, Y., Ji, Q.: Data-free prior model for facial action unit recognition. T. Affect. Comput. 4(2), 127–141 (2013).
  13. 13.
    Li, Y., Mavadati, S.M., Mahoor, M.H., Zhao, Y., Ji, Q.: Measuring the intensity of spontaneous facial action units with dynamic bayesian network. Pattern Recognit. (0), (2015).
  14. 14.
    Lucey, S., Ashraf, A.B., Cohn, J.F., Investigating spontaneous facial action recognition through AAM representations of the face. In: Delac,K., Grgic, M. (eds.), Face Recognition. I-Tech Education and Publishing, pp. 275–286 (2007)Google Scholar
  15. 15.
    Mahoor, M., Cadavid, S., Messinger, D., Cohn, J.: A framework for automated measurement of the intensity of non-posed facial action units. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2009. CVPR Workshops 2009, pp. 74–80 (2009).
  16. 16.
    Mavadati, S., Mahoor, M., Bartlett, K., Trinh, P., Cohn, J.: Disfa: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151–160 (2013). CrossRefGoogle Scholar
  17. 17.
    McCall, J.C., Trivedi, M.M.: Pose invariant affect analysis using thin-plate splines. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol. 3, pp. 958–964. IEEE, (2004)Google Scholar
  18. 18.
    Mehrabian, A.: Silent Messages: Implicit Communications of Emotions and Attitudes. Wadsworth Wadsworth Publishing Company, Belmont, California (1981)Google Scholar
  19. 19.
    Mlakar, U., Potočnik, B.: Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal, Image Video Process. 9(1), 245–253 (2015). CrossRefGoogle Scholar
  20. 20.
    Ojala T., Pietikäinen M., Mäenpää T.: (2000) Gray scale and rotation invariant texture classification with local binary patterns. In: Computer Vision - ECCV 2000. ECCV 2000. Lecture Notes in Computer Science, vol. 1842, pp. 404–420 Springer, Berlin, HeidelbergGoogle Scholar
  21. 21.
    Pantic, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man. Cybern. Part B: Cybern. 36(2), 433–449 (2006). CrossRefGoogle Scholar
  22. 22.
    Pantic, M., Rothkrantz, L.J.M.: An expert system for recognition of facial actions and their intensity. In: H.A. Kautz, B.W. Porter (eds.) AAAI/IAAI, pp. 1026–1033. AAAI Press, Palo Alto, Massachusetts (2000).
  23. 23.
    Rathee, N., Ganotra, D.: A novel approach for pain intensity detection based on facial feature deformations. J. Vis. Commun. Image Represent. 33, 247 – 254 (2015).
  24. 24.
    Rudovic, O., Pavlovic, V., Pantic, M.: Context-sensitive dynamic ordinal regression for intensity estimation of facial action units. Pattern Anal. Mach. Intell. IEEE Trans. 37(5), 944–958 (2015). CrossRefGoogle Scholar
  25. 25.
    Sandbach, G., Zafeiriou, S., Pantic, M.: Binary pattern analysis for 3D facial action unit detection (2012)Google Scholar
  26. 26.
    Savran, A., Sankur, B., Bilge, M.T.: Regression-based intensity estimation of facial action units. 3D Facial Behaviour Analysis and Understanding Image Vision Computing 30(10), 774–784 (2012).
  27. 27.
    Tian, L.Y., Kanade, T., Cohn, J.F.: Evaluation of gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity. In: Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition, pp. 229–234. Springer, Berlin (2002)Google Scholar
  28. 28.
    Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. Pattern Anal. Mach. Intell. IEEE Trans. 29(10), 1683–1699 (2007)CrossRefGoogle Scholar
  29. 29.
    Yan, H.: Kinship verification using neighborhood repulsed correlation metric learning. Regularization techniques for high-dimensional data analysis. Image Vision Comput. 60((Supplement C)), 91–97 (2017). CrossRefGoogle Scholar
  30. 30.
    Yang, L., Jin, R.: Distance metric learning: a comprehensive survey. Department of Computer Science and Engineering, Michigan State University (2006)Google Scholar
  31. 31.
    Yurtkan, K., Demirel, H.: Entropy-based feature selection for improved 3d facial expression recognition. Signal, Image Video Process. 8(2), 267–277 (2014). CrossRefGoogle Scholar
  32. 32.
    Zhang, Y., Zhang, L., Hossain, M.: Adaptive 3d facial action intensity estimation and emotion recognition. Expert Syst. Appl. 42(3), 1446–1464 (2015).

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Maharaja Surajmal Institute of Technology, (Affiliated to Guru Gobind Singh Indra Prastha University)New DelhiIndia
  2. 2.Indira Gandhi Delhi Technical University for Women, (Formerly Indira Gandhi Institute of Technology)Kashmere GateIndia

Personalised recommendations