Real-Time Emotional Recognition for Sociable Robotics Based on Deep Neural Networks Ensemble

  • Nadir Kamel BenamaraEmail author
  • Mikel Val-Calvo
  • José Ramón Álvarez-Sánchez
  • Alejandro Díaz-Morcillo
  • José Manuel Ferrández Vicente
  • Eduardo Fernández-Jover
  • Tarik Boudghene Stambouli
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11486)


Recognizing emotions in controlled conditions, based on facial expressions, has achieved high accuracies in the past years. This is still a challenging task for robots working in real-world scenarios due to different factors such as illumination, pose variation or occlusions. One of the next barriers of science is to give sociable robots the ability to fully engage in emotional interactions with users. In this paper a real-time emotion recognition system using a YOLO-based facial detection system and an ensemble CNN for sociable robots, is proposed. Experiments have been carried out on the most challenging database, FER 2013, giving a performance of 72.47% on test sets, achieving current standards.


Emotion recognition Sociable robotics Facial expression Human-machine interaction 



We want to acknowledge to Programa de Ayudas a Grupos de Excelencia de la Región de Murcia, from Fundación Séneca, Agencia de Ciencia y Tecnología de la Región de Murcia.


  1. 1.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPRV 2005), vol. 2, pp. 568–573, June 2005Google Scholar
  2. 2.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), San Diego, CA, USA, vol. 1, pp. 886–893. IEEE (2005)Google Scholar
  3. 3.
    Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. In: 2014 Canadian Conference on Computer and Robot Vision, pp. 98–103, May 2014Google Scholar
  4. 4.
    Diehl, J.J., Schmitt, L.M., Villano, M., Crowell, C.R.: The clinical use of robots for individuals with autism spectrum disorders: a critical review. Res. Autism Spectr. Disord. 6(1), 249–262 (2012)CrossRefGoogle Scholar
  5. 5.
    Ekman, P.: Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto (1976)Google Scholar
  6. 6.
    Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Robot. Auton. Syst. 42(3–4), 143–166 (2003)CrossRefGoogle Scholar
  7. 7.
    Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. arXiv:1307.0414 [cs, stat], July 2013
  8. 8.
    Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., Tao, D.: Deep neural networks with relativity learning for facial expression recognition. In: 2016 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pp. 1–6, July 2016Google Scholar
  9. 9.
    Itzcovich, I.: Yolo-face-detection (2018).
  10. 10.
    Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. 109(19), 7241–7244 (2012)CrossRefGoogle Scholar
  11. 11.
    Jeong, S., et al.: A social robot to mitigate stress, anxiety, and pain in hospital pediatric care. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts - HRI 2015, Extended Abstracts, Portland, Oregon, USA, pp. 103–104. ACM Press (2015)Google Scholar
  12. 12.
    Kim, B.K., Dong, S.Y., Roh, J., Kim, G., Lee, S.Y.: Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA, pp. 1499–1508. IEEE, June 2016Google Scholar
  13. 13.
    Li, S., Deng, W.: Deep facial expression recognition: a survey. arXiv:1804.08348 [cs], April 2018
  14. 14.
    Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 [cs], December 2013
  15. 15.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101, June 2010Google Scholar
  16. 16.
    Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10, March 2016Google Scholar
  17. 17.
    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)CrossRefGoogle Scholar
  18. 18.
    Pramerdorfer, C., Kampel, M.: Facial expression recognition using convolutional neural networks: state of the art. arXiv:1612.02903 [cs], December 2016
  19. 19.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 779–788. IEEE, June 2016Google Scholar
  20. 20.
    Scassellati, B., Admoni, H., Mataric, M.: Robots for use in autism research. Ann. Rev. Biomed. Eng. 14(1), 275–294 (2012)CrossRefGoogle Scholar
  21. 21.
    Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv:1412.6806 [cs], December 2014
  22. 22.
    Tang, Y.: Deep learning using linear support vector machines. arXiv:1306.0239 [cs, stat], June 2013
  23. 23.
    Tapus, A., Mataric, M.J., Scassellati, B.: The grand challenges in socially assistive robotics, p. 7Google Scholar
  24. 24.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, p. I. IEEE (2001)Google Scholar
  25. 25.
    Yang, S., Luo, P., Loy, C.C., Tang, X.: WIDER FACE: a face detection benchmark. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 5525–5533. IEEE, June 2016Google Scholar
  26. 26.
    Yu, R., et al.: Use of a therapeutic, socially assistive pet robot (PARO) in improving mood and stimulating social interaction and communication for people with dementia: study protocol for a randomized controlled trial. JMIR Res. Protoc. 4(2), e45 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nadir Kamel Benamara
    • 1
    Email author
  • Mikel Val-Calvo
    • 2
    • 3
  • José Ramón Álvarez-Sánchez
    • 2
  • Alejandro Díaz-Morcillo
    • 4
  • José Manuel Ferrández Vicente
    • 3
  • Eduardo Fernández-Jover
    • 5
  • Tarik Boudghene Stambouli
    • 1
  1. 1.Laboratoire Signaux et ImagesUniversité des Sciences et de la Technologie d’Oran Mohamed Boudiaf, USTO-MBOranAlgeria
  2. 2.Dpto. de Inteligencia ArtificialUniversidad Nacional de Educación a Distancia (UNED)MadridSpain
  3. 3.Dpto. Electrónica, Tecnología de Computadoras y ProyectosUniv. Politécnica de CartagenaCartagenaSpain
  4. 4.Dpto. Tecnologías de la Información y las ComunicacionesUniv. Politécnica de CartagenaCartagenaSpain
  5. 5.Instituto de BioingenieríaUniv. Miguel HernándezElcheSpain

Personalised recommendations