Skip to main content
SpringerLink
Log in

A Novel Model for Emotion Detection from Facial Muscles Activity

  • Conference paper
  • First Online:
Robot 2019: Fourth Iberian Robotics Conference (ROBOT 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1093))

Included in the following conference series:

Abstract

Considering human’s emotion in different applications and systems has received substantial attention over the last three decades. The traditional approach for emotion detection is to first extract different features and then apply a classifier, like SVM, to find the true class. However, recently proposed Deep Learning based models outperform traditional machine learning approaches without requirement of a separate feature extraction phase.

This paper proposes a novel deep learning based facial emotion detection model, which uses facial muscles activities as raw input to recognize the type of the expressed emotion in the real time. To this end, we first use OpenFace to extract the activation values of the facial muscles, which are then presented to a Stacked Auto Encoder (SAE) as feature set. Afterward, the SAE returns the best combination of muscles in describing a particular emotion, these extracted features at the end are applied to a Softmax layer in order to fulfill multi classification task.

The proposed model has been applied to the CK+, MMI and RADVESS datasets and achieved respectively average accuracies of 95.63%, 95.58%, and 84.91% for emotion type detection in six classes, which outperforms state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://imotions.com/blog/facial-action-coding-system/.

References

  1. Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: OpenFace 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, pp. 59–66. IEEE (2018)

    Google Scholar 

  2. Brave, S., Nass, C., Hutchinson, K.: Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int. J. Hum.-Comput. Stud. 62(2), 161–178 (2005)

    Article  Google Scholar 

  3. Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., Karpouzis, K.: Multimodal emotion recognition from expressive faces, body gestures and speech. In: IFIP International Conference on Artificial Intelligence Applications and Innovations, pp. 375–388. Springer (2007)

    Google Scholar 

  4. Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the facial action coding system. In: The Handbook of Emotion Elicitation and Assessment, pp. 203–221 (2007)

    Google Scholar 

  5. De Gauquier, L., Cao, H.L., Gomez Esteban, P., De Beir, A., van de Sanden, S., Willems, K., Brengman, M., Vanderborght, B.: Humanoid robot pepper at a Belgian chocolate shop. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 373–373. ACM (2018)

    Google Scholar 

  6. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. In: Proceedings of the National Academy of Sciences, p. 201322355 (2014)

    Google Scholar 

  7. Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 467–474. ACM (2015)

    Google Scholar 

  8. Ekman, P.: Strong evidence for universals in facial expressions: a reply to Russell’s mistaken critique (1994)

    Google Scholar 

  9. Ekman, P.: Facial Action Coding System (FACS). A human face (2002)

    Google Scholar 

  10. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: first steps towards an automatic system. In: Tutorial and Research Workshop on Affective Dialogue Systems, pp. 36–48. Springer (2004)

    Google Scholar 

  11. Hasani, B., Mahoor, M.H.: Spatio-temporal facial expression recognition using convolutional neural networks and conditional random fields. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2017, pp. 790–795. IEEE (2017)

    Google Scholar 

  12. Ko, B.: A brief review of facial emotion recognition based on visual information. Sensors 18(2), 401 (2018)

    Article  Google Scholar 

  13. Liliana, D.Y., Basaruddin, T.: Review of automatic emotion recognition through facial expression analysis. In: 2018 International Conference on Electrical Engineering and Computer Science (ICECOS), pp. 231–236. IEEE (2018)

    Google Scholar 

  14. Liu, M., Li, S., Shan, S., Chen, X.: AU-aware deep networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)

    Google Scholar 

  15. Livingstone, S.R., Russo, F.A.: The Ryerson audio-visual database of emotional speech and song (RAVDESS): a dynamic, multimodal set of facial and vocal expressions in North American English. PloS One 13(5), e0196391 (2018)

    Article  Google Scholar 

  16. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  17. Mehrabian, A.: Nonverbal Communication. Routledge, Abingdon (2017)

    Book  Google Scholar 

  18. Miyakoshi, Y., Kato, S.: Facial emotion detection considering partial occlusion of face using Bayesian network. In: 2011 IEEE Symposium on Computers & Informatics (ISCI), pp. 96–101. IEEE (2011)

    Google Scholar 

  19. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), pp. 1–10. IEEE (2016)

    Google Scholar 

  20. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: 2005 IEEE International Conference on Multimedia and Expo, pp. 5–pp. IEEE (2005)

    Google Scholar 

  21. Pitaloka, D.A., Wulandari, A., Basaruddin, T., Liliana, D.Y.: Enhancing cnn with preprocessing stage in automatic emotion recognition. Proc. Comput. Sci. 116, 523–529 (2017)

    Article  Google Scholar 

  22. Pop, C.A., Simut, R., Pintea, S., Saldien, J., Rusu, A., David, D., Vanderfaeillie, J., Lefeber, D., Vanderborght, B.: Can the social robot probo help children with autism to identify situation-based emotions? A series of single case experiments. Int. J. Humanoid Rob. 10(03), 1350025 (2013)

    Article  Google Scholar 

  23. Schuller, B., Rigoll, G., Lang, M.: Hidden Markov model-based speech emotion recognition. In: Proceedings of 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2003, vol. 2, pp. II–1. IEEE (2003)

    Google Scholar 

  24. Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1683–1699 (2007)

    Article  Google Scholar 

  25. De la Torre, F., Cohn, J.F.: Facial expression analysis. In: Visual Analysis of Humans, pp. 377–409. Springer (2011)

    Google Scholar 

  26. Vydana, H.K., Kumar, P.P., Krishna, K.S.R., Vuppala, A.K.: Improved emotion recognition using GMM-UBMs. In: 2015 International Conference on Signal Processing and Communication Engineering Systems (SPACES), pp. 53–57. IEEE (2015)

    Google Scholar 

  27. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2016)

    Google Scholar 

  28. Xiaoxi, M., Weisi, L., Dongyan, H., Minghui, D., Li, H.: Facial emotion recognition. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 77–81. IEEE (2017)

    Google Scholar 

  29. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)

    Article  Google Scholar 

  30. Zhang, L., Verma, B., Tjondronegoro, D., Chandran, V.: Facial expression analysis under partial occlusion: a survey. ACM Comput. Surv. (CSUR) 51(2), 25 (2018)

    Article  Google Scholar 

  31. Zhao, L., Wang, Z., Zhang, G.: Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and Gabor multiorientation fusion histogram. Math. Probl. Eng. 2017, 12 (2017)

    Google Scholar 

Download references

Acknowledgment

The work leading to these results has received funding from the European Commission 7th Framework Program as a part of the DREAM project, grant no. 611391 and the ICON project ROBO-CURE.

Author information

Authors and Affiliations

  1. Robotics and Multibody Mechanics Research Group, Vrije Universiteit Brussel and Flanders Make, Brussels, Belgium

    Elahe Bagheri, Pablo G. Esteban & Bram Vanderborgth

  2. Electrical Engineering Group, Chalmers University of Technology, Gothenburg, Sweden

    Azam Bagheri

Authors
  1. Elahe Bagheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azam Bagheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pablo G. Esteban
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bram Vanderborgth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elahe Bagheri .

Editor information

Editors and Affiliations

  1. School of Engineering, Polytechnic Institute of Porto, Porto, Portugal

    Manuel F. Silva

  2. Department of Electrical Engineering, Polytechnic Institute of Bragança, Bragança, Portugal

    José Luís Lima

  3. Faculty of Engineering, University of Porto, Porto, Portugal

    Luís Paulo Reis

  4. UPC, Universitat Politècnica de Catalunya, Barcelona, Spain

    Alberto Sanfeliu

  5. Centro Universitario de la Defensa (CUD), Zaragoza, Spain

    Danilo Tardioli

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bagheri, E., Bagheri, A., Esteban, P.G., Vanderborgth, B. (2020). A Novel Model for Emotion Detection from Facial Muscles Activity. In: Silva, M., Luís Lima, J., Reis, L., Sanfeliu, A., Tardioli, D. (eds) Robot 2019: Fourth Iberian Robotics Conference. ROBOT 2019. Advances in Intelligent Systems and Computing, vol 1093. Springer, Cham. https://doi.org/10.1007/978-3-030-36150-1_20

Download citation

Publish with us

Policies and ethics

Search

Navigation