Neural Computing and Applications

, Volume 29, Issue 7, pp 359–373 | Cite as

A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots

  • Ariel Ruiz-Garcia
  • Mark Elshaw
  • Abdulrahman Altahhan
  • Vasile Palade
S.I. : EANN 2016


We have recently seen significant advancements in the development of robotic machines that are designed to assist people with their daily lives. Socially assistive robots are now able to perform a number of tasks autonomously and without human supervision. However, if these robots are to be accepted by human users, there is a need to focus on the form of human–robot interaction that is seen as acceptable by such users. In this paper, we extend our previous work, originally presented in Ruiz-Garcia et al. (in: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93, 2016., to provide emotion recognition from human facial expressions for application on a real-time robot. We expand on previous work by presenting a new hybrid deep learning emotion recognition model and preliminary results using this model on real-time emotion recognition performed by our humanoid robot. The hybrid emotion recognition model combines a Deep Convolutional Neural Network (CNN) for self-learnt feature extraction and a Support Vector Machine (SVM) for emotion classification. Compared to more complex approaches that use more layers in the convolutional model, this hybrid deep learning model produces state-of-the-art classification rate of \(96.26\%\), when tested on the Karolinska Directed Emotional Faces dataset (Lundqvist et al. in The Karolinska Directed Emotional Faces—KDEF, 1998), and offers similar performance on unseen data when tested on the Extended Cohn–Kanade dataset (Lucey et al. in: Proceedings of the third international workshop on CVPR for human communicative behaviour analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101, 2010). This architecture also takes advantage of batch normalisation (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift., 2015) for fast learning from a smaller number of training samples. A comparison between Gabor filters and CNN for feature extraction, and between SVM and multilayer perceptron for classification is also provided.


Deep Convolutional Neural Networks Emotion recognition Gabor filter Socially assistive robots Support Vector Machine 



We would like to thank the School of Computing, Electronics and Mathematics, at Coventry University for funding this research. We would like to also acknowledge the contribution of the Barry Gidden fund which partially funded this work. The authors would also like to acknowledge the invaluable contribution of Maria Charalambous, Kenny Ruiz, Ibrahim Alamakky and Danielle Labhardt in creating this paper.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Scassellati B, Admoni H, Mataric M (2012) Robots for use in autism research. Annu Rev Biomed Eng 14:275–294CrossRefGoogle Scholar
  2. 2.
    Fasola J, Mataric M (2013) A socially assistive robot exercise coach for the elderly. J Hum Robot Interact 2(2):3–32CrossRefGoogle Scholar
  3. 3.
    Chang W-L, Šabanovic S, Huber L (2013) Use of seal-like robot PARO in sensory group therapy for older adults with dementia. In: Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction. IEEE PressGoogle Scholar
  4. 4.
    Soler MV, Agüera-Ortiz L, Rodrguez JO, Rebolledo CM, Muoz AP, Prez IR, Ruiz EO, Snchez AB, Cano VH, Chilln LC, Ruiz SF (2015) Social robots in advanced dementia. Front Aging Neurosci 3:133. Google Scholar
  5. 5.
    Ruiz-Garcia A, Elshaw M, Altahhan A, Palade V (2016) Emotion recognition using facial expression images for a robotic companion. In: Engineering applications of neural networks: 17th international conference, EANN 2016, Aberdeen, UK, September 2–5, 2016, proceedings, pp 79–93. doi:
  6. 6.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift.
  7. 7.
    Lundqvist D, Flykt A, Öhman A (1998) The Karolinska Directed Emotional Faces—KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. ISBN 91-630-7164-9Google Scholar
  8. 8.
    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Kanade dataset (CK+): a complete expression dataset for action unit and emotion-specified expression. In: Proceedings of the third international workshop on CVPR for human communicative behavior analysis (CVPR4HB 2010), San Francisco, USA, pp 94–101Google Scholar
  9. 9.
    Robertson J (2014) Human rights vs. robot rights: forecasts from Japan. Crit Asian Stud 46(4):571–598. doi: CrossRefGoogle Scholar
  10. 10.
    Dahl T, Boulos M (2013) Robots in health and social care: a complementary technology to home care and telehealthcare? Robotics 3:1–21CrossRefGoogle Scholar
  11. 11.
    Castellano G, Paiva A, Kappas A, Nabais F, Aylett R, Barendregt W, Hastie H, Bull S (2013) Towards empathic virtual and robotic tutors. In: Lecture notes in computer science, pp 733–736Google Scholar
  12. 12.
    Toh LPE, Causo A, Tzuo PW, Chen IM, Yeo SH, Link S (2016) A review on the use of robots in education and young children. Sci Direct 19:148–163Google Scholar
  13. 13.
    Kory Westlund J, Gordon G, Spaulding S, Lee J, Plummer L, Martinez M, Das M, Breazeal C (2015) Learning a second language with a socially assistive robot. In: The 1st international conference on social robots in therapy and education, 2015, Almere, The Netherlands.
  14. 14.
    Affectiva (2016).
  15. 15.
    Kim ES, Berkovits LD, Bernier EP, Leyzberg D, Shic F, Paul R, Scassellati B (2013) Social robots as embedded reinforcers of social behavior in children with autism. J Autism Dev Disord 43(5):1038–1049CrossRefGoogle Scholar
  16. 16.
    Rabbitt SM, Kazdin AE, Hong JH (2015) Acceptability of robot-assisted therapy for disruptive behavior problems in children. Arch Sci Psychol 3(1):101–110.
  17. 17.
    KSERA (2016) Knowledgeable service robots for aging.
  18. 18.
    GeriJoy (2016) Care and companionship for seniors—GeriJoy.
  19. 19.
    Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A (2013) The influence of empathy in humanrobot relations. Int J Hum Comput Stud 71(3):250–260CrossRefGoogle Scholar
  20. 20.
    Graaf MMA, Ben Allouch S, Dijk JAGM (2016) Long-term acceptance of social robots in domestic environments: insights from a user’s perspective. In: AAAIGoogle Scholar
  21. 21.
    Duffy BR (2006) Fundamental issues in social robotics. Int Rev Inf Eth 6:31–36Google Scholar
  22. 22.
    Boughrara H, Chtourou M, Ben Amar C, Chen L (2014) Facial expression recognition based on a MLP neural network using constructive training algorithm. Multimed Tools Appl 75:709–731CrossRefGoogle Scholar
  23. 23.
    Kahou S, Michalski V, Konda K, Memisevic R, Pal C (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction (ICMI ’15), pp 467–474Google Scholar
  24. 24.
    Lawrence S, Giles CL, Tsoi AC, Back AD (1997) Face recognition: a convolutional neural-network approach. IEEE Trans Neural Netw 8(1):98–113.
  25. 25.
    Gupta A, Garg M (2016) A human emotion recognition system using supervised self-organising maps. In: 2014 International conference on computing for sustainable global development (INDIACom), pp 654–659Google Scholar
  26. 26.
    Sarnarawickrame K, Mindya S (2013) Facial expression recognition using active shape models and support vector machines. In: 2013 International conference on advances in ICT for emerging regions (ICTer), pp 51–55Google Scholar
  27. 27.
    Sohail ASM, Bhattacharya P (2011) Classifying facial expressions using level set method based lip contour detection and multi-class support vector machines. Int J Pattern Recognit Artif Intell 25(6):835–862. doi: MathSciNetCrossRefGoogle Scholar
  28. 28.
    Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. CoRR abs/1408.3, 6.
  29. 29.
    Ahsan T, Jabid T, Chong U-P (2013) Facial expression recognition using local transitional pattern on gabor filtered facial images. IETE Tech Rev 30(12):47.;year=2013;volume=30;issue=1;spage=47;epage=52;aulast=Ahsan l slightly outperforms Gabor filters as a feature extraction method, at least for this particular dataset
  30. 30.
    Chelali FZ, Djeradi A (2015) Face recognition using MLP and RBF neural network with Gabor and discrete wavelet transform characterization: a comparative study. Math Probl Eng 2015:1–16.
  31. 31.
    Mehta N, Jadhav S (2016) Facial emotion recognition using log Gabor filter and PCA. In: Proceedings of international conference on computing communication control and automation (ICCUBEA), pp 1–5Google Scholar
  32. 32.
    Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE international conference on automatic face and gesture recognition (FG’00), Grenoble, France, pp 46–53Google Scholar
  33. 33.
    Paul PP, Monwar MM, Gavrilova ML, Wang PSP (2010) Rotation invariant multivew face detection using skin color regressive model and support vector regression. Int J Pattern Recognit Artif Intell 24(8):1261–1280. CrossRefGoogle Scholar
  34. 34.
    Khan SA, Hussain A, Usman M, Nazir M, Riaz N, Mirza AM (2014) Robust face recognition using computationally efficient features. J Intell Fuzzy Syst 27(6):3131–3143.<GotoISI>://WOS:000345981600037Google Scholar
  35. 35.
    Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4295–4304.
  36. 36.
    Hewahi NM, Baraka ARM (2012) Impact of ethnic group on human emotion recognition using backpropagation neural network. BRAIN. Broad Res Artif 20–27.
  37. 37.
    Khashman A (2009) Application of an emotional neural network to facial recognition. Neural Comput Appl 18(4):309–320CrossRefGoogle Scholar
  38. 38.
    Burkert P, Trier F, Afzal MZ, Dengel A, Liwicki M (2015) DeXpression: Deep Convolutional Neural Network for expression recognition. arXiv preprint 1–8.
  39. 39.
    Beaudry O, Roy-Charland A, Perron M, Cormier I, Tapp R (2014) Featural processing in recognition of emotional facial expressions. Cognit Emot 28(3):416–432.
  40. 40.
    Kuhn H, Tucker A (1951) Nonlinear programming. In: Proceedings of the second Berkeley symposium on mathematical statistics and probability, pp 481–492Google Scholar
  41. 41.
    Aldebaran (2016) Who is NAO?
  42. 42.
    Tan CC, Eswaran C (2010) Reconstruction and recognition of face and digit images using autoencoders. Neural Comput Appl 19(7):1069–1079CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2018

Authors and Affiliations

  1. 1.School of Computing, Electronics and Mathematics, Faculty of Engineering, Environment and ComputingCoventry UniversityCoventryUK

Personalised recommendations