Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits

  • 70 Accesses

  • 2 Citations


This paper focuses on the validation of a tool designed to improve affect recognition, a fundamental aspect of social cognition as it greatly affects the functionality and quality of life of patients with mental disorders. The presented tool facilitates the generation of multi-modal avatar-based therapies by mental health professionals in this important clinical domain. Moreover, the tool for remediation of social cognitive deficits may be customised to each patient’s impairment. This paper describes how the tool was assessed by therapists after viewing a video explaining its most relevant aspects. The participants were asked to fill in a questionnaire based on UTAUT2 for the study of the acceptance and use of this technology. In light of the results obtained from 41 therapists about their intention of use, the most important statement is that their interest for this kind of tools is high. Nonetheless, there are some factors that negatively affect their behavioural intention.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. Abirached B, Zhang Y, Aggarwal JK, Fernandes T, Carlos J, Orvalho V (2011) Improving communication skills of children with ASDs through interaction with virtual characters. In: IEEE 1st international conference on serious games and applications for health, IEEE, pp 3493–3502

  2. Adery LH, Ichinose M, Torregrossa LJ, Wade J, Nichols H, Bekele E, Bian D, Gizdic A, Granholm E, Sarkar N, Park S (2018) The acceptability and feasibility of a novel virtual reality based social skills training game for schizophrenia: preliminary findings. Psychiatry Res 270:496–502.

  3. Akalin N, Kiselev A, Kristoffersson A, Loutfi A (2017) An evaluation tool of the effect of robots in eldercare on the sense of safety and security. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan J-J, Eyssel F, He H (eds) International conference on social robotics. Springer, Cham, pp 628–637

  4. Barclay D, Higgins C, Thompson R (1995) The partial least squares (PLS) approach to casual modeling: personal computer adoption ans use as an illustration. Technol Stud 2:285–309

  5. Baron-Cohen S, Wheelwright S, Skinner R, Martin J, Clubley E (2001) The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians. J Autism Dev Disord 31(1):5–17.

  6. Beer JM, Fisk AD, Rogers WA (2009) Emotion recognition of virtual agents facial expressions: the effects of age and emotion intensity. Proc Hum Factors Ergon Soc 53(2):131–135.

  7. Brothers L (1990) The neural basis of primate social communication. Motiv Emot 14(2):81–91.

  8. Castillo JC, Castro-González Á, Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A, Salichs MA (2016) Software architecture for smart emotion recognition and regulation of the ageing adult. Cogn Comput 8(2):357–367.

  9. Cerezo E, Hupont I, Baldassarri S, Ballano S (2012) Emotional facial sensing and multimodal fusion in a continuous 2d affective space. J Ambient Intell Hum Comput 3(1):31–46.

  10. Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A (2014) Improvement of the elderly quality of life and care through smart emotion regulation. In: Pecchia L, Chen LL, Nugent C, Bravo J (eds) Ambient assisted living and daily activities. Springer, Cham, pp 348–355.

  11. Fernández-Caballero A, Martínez-Rodrigo A, Pastor JM, Castillo JC, Lozano-Monasor E, López MT, Zangr’oniz R, Latorre JM, Fernández-Sotos A (2016) Smart environment architecture for emotion detection and regulation. J Biomed Inform 64:55–73.

  12. Fernández-Caballero A, Navarro E, Fernández-Sotos P, González P, Ricarte JJ, Latorre JM, Rodriguez-Jimenez R (2017) Human-avatar symbiosis for the treatment of auditory verbal hallucinations in schizophrenia through virtual/augmented reality and brain–computer interfaces. Front Neuroinform 11:64.

  13. Fernández-Sotos P, Navarro E, Torio I, Dompablo M, Fernández-Caballero A, Rodriguez-Jimenez R (2018) Pharmacological interventions in social cognition deficits: a systematic mapping review. Psychiatry Res 270:57–67.

  14. Fernández-Sotos P, Torio I, Fernández-Caballero A, Navarro E, González P, Dompablo M, Rodriguez-Jimenez R (2019) Social cognition remediation interventions: a systematic mapping review. PLoS One 14:e0218720

  15. García AS, Molina JP, Martínez D, González P (2008) Enhancing collaborative manipulation through the use of feedback and awareness in CVEs. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. ACM, New York, p 32.

  16. García AS, Navarro E, Fernández-Caballero A, González P (2018) Towards the design of avatar-based therapies for enhancing facial affect recognition. In: Novais P, Jung JJ, Villarrubia G, Fernández-Caballero A, Navarro E, González P, Carneiro D, Pinto A, Campbell AT, Durães D (eds) Ambient intelligence—software and applications, 9th international symposium on ambient intelligence. Springer, Cham, pp 306–313

  17. Garrido MV, Lopes D, Prada M, Rodrigues D, Jerónimo R, Mourão RP (2016) The many faces of a face: comparing stills and videos of facial expressions in eight dimensions (SAVE database). Behav Res Methods 49(4):1343–1360.

  18. Gottesman II (1991) Schizophrenia genesis: the origins of madness. WH Freeman/Times Books/Henry Holt & Co, New York

  19. Green MF, Penn DL, Bentall R, Carpenter WT, Gaebel W, Gur RC, Kring AM, Park S, Silverstein SM, Heinssen R (2008) Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophr Bull 34(6):1211–1220.

  20. Hair JF Jr, Hult GTM, Ringle C, Sarstedt M (2014) A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publications, Thousand Oaks

  21. Henseler J, Ringle CM, Sarstedt M (2015) A new criterion for assessing discriminant validity in variance-based structural equation modeling. J Acad Mark Sci 43(1):115–135

  22. Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB (2013) Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord 43(1):34–44.

  23. Khalilzadeh J, Ozturk AB, Bilgihan A (2017) Security-related factors in extended UTAUT model for NFC based mobile payment in the restaurant industry. Comput Human Behav 70:460–474

  24. Krishnappa Babu PR, Lahiri U (2019) Classification approach for understanding implications of emotions using eye-gaze. J Ambient Intell Hum Comput.

  25. Lahera G, Ruiz-Murugarren S, Iglesias P, Ruiz-Bennasar C, Herrería E, Montes JM, Fernández-Liria A (2012) Social cognition and global functioning in bipolar disorder. J Nerv Menta Dis 200(2):135–141.

  26. Laumer S, Gubler F, Maier C, Weitzel T (2018) Job seekers’ acceptance of job recommender systems: results of an empirical study. In: Proceedings of the 51st Hawaii international conference on system sciences. Curran Associates, Inc., pp 3172–3181

  27. Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Hum Comput 8(4):567–578.

  28. Manis KT, Choi D (2018) The virtual reality hardware acceptance model (vr-ham): Extending and individuating the technology acceptance model (TAM) for virtual reality hardware. J Bus Res.

  29. Marcoulides GA, Saunders C (2006) PLS: a silver bullet? Manag Inf Syst Q 30(2):1

  30. McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex sdk: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, pp 3723–3726

  31. Mendoza-Palechor F, Menezes ML, Sant’Anna A, Ortiz-Barrios M, Samara A, Galway L (2018) Affective recognition from EEG signals: an integrated data-mining approach. J Ambient Intell Hum Comput.

  32. Oechslein O, Fleischmann M, Hess T (2014) An application of utaut2 on social recommender systems: incorporating social information for performance expectancy. In: 2014 47th Hawaii international conference on system sciences. IEEE, pp 3297–3306

  33. Oliver M, Teruel M, Molina J, Romero-Ayuso D, González P (2018) Ambient intelligence environment for home cognitive telerehabilitation. Sensors 18(11):3671

  34. Pinkham AE, Penn DL, Green MF, Buck B, Healey K, Harvey PD (2014) The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophr Bull 40(4):813–823.

  35. Pinkham AE, Penn DL, Green MF, Harvey PD (2016) Social cognition psychometric evaluation: results of the initial psychometric study. Schizophr Bull 42(2):494–504.

  36. Roark DA, Barrett SE, Spence MJ, Abdi H, O’Toole AJ (2003) Psychological and neural perspectives on the role of motion in face recognition. Behav Cogn Neurosci Rev 2(1):15–46.

  37. Rosenberg H, McDonald S, Dethier M, Kessels RP, Westbrook RF (2014) Facial emotion recognition deficits following moderate-severe traumatic brain injury (TBI): re-examining the valence effect and the role of emotion intensity. J Int Neuropsychol Soc 20(10):994–1003.

  38. Rus-Calafell M, Gutiérrez-Maldonado J, Ribas-Sabaté J (2014) A virtual reality-integrated program for improving social skills in patients with schizophrenia: a pilot study. J Behav Ther Exp Psychiatry 45(1):81–89.

  39. Samara A, Galway L, Bond R, Wang H (2019) Affective state detection via facial expression analysis within a human–computer interaction context. J Ambient Intell Hum Comput 10(6):2175–2184.

  40. Souto YM, Campo MV, Llenderrozas FD, Álvarez MR, Mateos R, Caballero AAG (2018) Randomized clinical trial with e-MotionalTraining® 1.0 for social cognition rehabilitation in schizophrenia. Front Psychiatry 26(9):40.

  41. Teruel MA, Navarro E, González P (2017) Exploiting awareness for the development of collaborative rehabilitation systems. Mobile Inf Syst.

  42. Tracy JL, Randles D, Steckler CM (2015) The nonverbal communication of emotions. Curr Opin Behav Sci 3:25–30.

  43. Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q 27:425–478

  44. Venkatesh V, Thong JY, Xu X (2012) Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 36:157–178

  45. Woods S, Walters M, Koay KL, Dautenhahn K (2006) Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: 9th ieee international workshop on advanced motion control, 2006. IEEE, pp 750–755

  46. Yang YJD, Allen T, Abdullahi SM, Pelphrey K, Volkmar F, Chapman S (2017) Brain responses to biological motion predict treatment outcome in young adults with autism receiving virtual reality social cognition training: preliminary findings. Behav Res Ther 93:55–66.

Download references


This work was partially supported by Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (AEI)/European Regional Development Fund (FEDER, UE) under EmoBioFeedback (DPI2016-80894-R), Vi-SMARt (TIN2016-79100-R) and HA-SYMBIOSIS (TIN2015-72931-EXP) Grants, and by Biomedical Research Networking Centre in Mental Health (CIBERSAM) of the Instituto de Salud Carlos III.

Author information

Correspondence to Antonio Fernández-Caballero.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 119430 KB)

Supplementary material 1 (mp4 119430 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

García, A.S., Fernández-Sotos, P., Fernández-Caballero, A. et al. Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits. J Ambient Intell Human Comput (2019) doi:10.1007/s12652-019-01418-8

Download citation


  • Social cognition
  • Affect recognition
  • Virtual reality
  • Avatar