Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow

  • Allen JosephEmail author
  • P. Geetha
Original Article


Detection of emotion using facial expression is a growing field of research. Facial expression detection is also helpful to identify the behavior of a person when a man interacts with the computer. In this work, facial expression recognition with respect to the changes in the facial geometry is proposed. First, the image is enhanced by means of discrete wavelet transform and fuzzy combination. Then, the facial geometry is found using the modified eyemap and mouthmap algorithm after finding the landmarks. Finally, the area and angle of the constructed triangles are found and classified using neural network with the help of tensorflow central processing unit version. Results show that the proposed algorithm is efficient in finding the facial emotion.


Emotion Eyemap Facial expression Facial geometry Mouthmap 



This work was funded by the Department of Science and Technology—Promotion of University Research and Scientific Excellence (DST-PURSE) Phase II Program, India.


  1. 1.
    Agarwal, S., Santra, B., Mukherjee, D.P.: Anubhav: recognizing emotions through facial expression. Vis. Comput. 34(2), 177–191 (2018)Google Scholar
  2. 2.
    Alugupally, N., Samal, A., Marx, D., Bhatia, S.: Analysis of landmarks in recognition of face expressions. Pattern Recognit. Image Anal. 21(4), 681–693 (2011)Google Scholar
  3. 3.
    Buciu, I., Kotropoulos, C., Pitas, I.: Comparison of ica approaches for facial expression recognition. Signal Image Video Process. 3(4), 345–361 (2009)zbMATHGoogle Scholar
  4. 4.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)zbMATHGoogle Scholar
  5. 5.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Washington (1978)Google Scholar
  6. 6.
    Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 1–16 (2018)Google Scholar
  7. 7.
    Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)Google Scholar
  8. 8.
    Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)Google Scholar
  9. 9.
    Huang, C.L., Huang, Y.M.: Facial expression recognition using model-based feature extraction and action parameters classification. J. Vis. Commun. Image Represent. 8(3), 278–290 (1997)Google Scholar
  10. 10.
    Ilbeygi, M., Shah-Hosseini, H.: A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng. Appl. Artif. Intell. 25(1), 130–146 (2012)Google Scholar
  11. 11.
    Jackway, P.T., Deriche, M.: Scale-space properties of the multiscale morphological dilation–erosion. IEEE Trans. Pattern Anal. Mach. Intell. 18(1), 38–51 (1996)Google Scholar
  12. 12.
    Jain, V., Mavridou, E., Crowley, J.L., Lux, A.: Facial expression analysis and the affect space. Pattern Recognit. Image Anal. 25(3), 430–436 (2015)Google Scholar
  13. 13.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings, pp. 46–53. IEEE (2000)Google Scholar
  14. 14.
    Karthigayan, M., Juhari, M.R.M., Nagarajan, R., Sugisaka, M., Yaacob, S., Mamat, M.R., Desa, H.: Development of a personified face emotion recognition technique using fitness function. Artif. Life Robot. 11(2), 197–203 (2007)Google Scholar
  15. 15.
    Kim, D.: Facial expression recognition using ASM-based post-processing technique. Pattern Recognit. Image Anal. 26(3), 576–581 (2016)Google Scholar
  16. 16.
    Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6(1), 159–169 (2012)Google Scholar
  17. 17.
    Lekdioui, K., Messoussi, R., Ruichek, Y., Chaabi, Y., Touahni, R.: Facial decomposition for expression recognition using texture/shape descriptors and SVM classifier. Signal Process. Image Commun. 58, 300–312 (2017)Google Scholar
  18. 18.
    Liu, N., Zhang, B., Zong, Y., Liu, L., Chen, J., Zhao, G., Zhu, L.: Super wide regression network for unsupervised cross-database facial expression recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1897–1901. IEEE (2018)Google Scholar
  19. 19.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)Google Scholar
  20. 20.
    Lundqvist, D., Flykt, A., Öhman, A.: The Karolinska Directed Emotional Faces-KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, ISBN 91-630-7164-9 (1998)Google Scholar
  21. 21.
    Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. Proceedings, pp. 200–205. IEEE (1998)Google Scholar
  22. 22.
    Mayer, C., Eggers, M., Radig, B.: Cross-database evaluation for facial expression recognition. Pattern Recognit. Image Anal. 24(1), 124–132 (2014)Google Scholar
  23. 23.
    Mlakar, U., Potočnik, B.: Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal Image Video Process. 9(1), 245–253 (2015)Google Scholar
  24. 24.
    Panda, S.P.: Image contrast enhancement in spatial domain using fuzzy logic based interpolation method. In: 2016 IEEE Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–4. IEEE (2016)Google Scholar
  25. 25.
    Ruiz-Garcia, A., Palade, V., Elshaw, M., Almakky, I.: Deep learning for illumination invariant facial expression recognition. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2018)Google Scholar
  26. 26.
    Silva, C., Schnitman, L., Oliveira, L.: Detection of facial landmarks using local-based information. In: The 19th Edition of the Brazilian Conference on Automation-CBA 2012, Campina Grande, PB, Brazil (oral presentation), September 3 (2012)Google Scholar
  27. 27.
    Sun, Z., Hu, Z., Wang, M., Zhao, S.: Individual-free representation-based classification for facial expression recognition. Signal Image Video Process. 11(4), 597–604 (2017)Google Scholar
  28. 28.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I–I. IEEE (2001)Google Scholar
  29. 29.
    Wong, J.J., Cho, S.Y.: A face emotion tree structure representation with probabilistic recursive neural network modeling. Neural Comput. Appl. 19(1), 33–54 (2010)Google Scholar
  30. 30.
    Yaddaden, Y., Adda, M., Bouzouane, A., Gaboury, S., Bouchard, B.: User action and facial expression recognition for error detection systemin an ambient assisted environment. Expert. Syst. Appl. 112, 173–189 (2018)Google Scholar
  31. 31.
    Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2177 (2018)Google Scholar
  32. 32.
    Yu, Z., Liu, Q., Liu, G.: Deeper cascaded peak-piloted network for weak expression recognition. Vis. Comput. 34(12), 1691–1699 (2018)Google Scholar
  33. 33.
    Zhao, G., Huang, X., Taini, M., Li, S.Z., PietikäInen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringCollege of Engineering, Guindy, Anna UniversityChennaiIndia

Personalised recommendations