This paper focuses on the validation of a tool designed to improve affect recognition, a fundamental aspect of social cognition as it greatly affects the functionality and quality of life of patients with mental disorders. The presented tool facilitates the generation of multi-modal avatar-based therapies by mental health professionals in this important clinical domain. Moreover, the tool for remediation of social cognitive deficits may be customised to each patient’s impairment. This paper describes how the tool was assessed by therapists after viewing a video explaining its most relevant aspects. The participants were asked to fill in a questionnaire based on UTAUT2 for the study of the acceptance and use of this technology. In light of the results obtained from 41 therapists about their intention of use, the most important statement is that their interest for this kind of tools is high. Nonetheless, there are some factors that negatively affect their behavioural intention.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Abirached B, Zhang Y, Aggarwal JK, Fernandes T, Carlos J, Orvalho V (2011) Improving communication skills of children with ASDs through interaction with virtual characters. In: IEEE 1st international conference on serious games and applications for health, IEEE, pp 3493–3502
Adery LH, Ichinose M, Torregrossa LJ, Wade J, Nichols H, Bekele E, Bian D, Gizdic A, Granholm E, Sarkar N, Park S (2018) The acceptability and feasibility of a novel virtual reality based social skills training game for schizophrenia: preliminary findings. Psychiatry Res 270:496–502. https://doi.org/10.1016/j.psychres.2018.10.014
Akalin N, Kiselev A, Kristoffersson A, Loutfi A (2017) An evaluation tool of the effect of robots in eldercare on the sense of safety and security. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan J-J, Eyssel F, He H (eds) International conference on social robotics. Springer, Cham, pp 628–637
Barclay D, Higgins C, Thompson R (1995) The partial least squares (PLS) approach to casual modeling: personal computer adoption ans use as an illustration. Technol Stud 2:285–309
Baron-Cohen S, Wheelwright S, Skinner R, Martin J, Clubley E (2001) The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians. J Autism Dev Disord 31(1):5–17. https://doi.org/10.1023/A:1005653411471
Beer JM, Fisk AD, Rogers WA (2009) Emotion recognition of virtual agents facial expressions: the effects of age and emotion intensity. Proc Hum Factors Ergon Soc 53(2):131–135. https://doi.org/10.1177/154193120905300205
Brothers L (1990) The neural basis of primate social communication. Motiv Emot 14(2):81–91. https://doi.org/10.1007/BF00991637
Castillo JC, Castro-González Á, Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A, Salichs MA (2016) Software architecture for smart emotion recognition and regulation of the ageing adult. Cogn Comput 8(2):357–367. https://doi.org/10.1007/s12559-016-9383-y
Cerezo E, Hupont I, Baldassarri S, Ballano S (2012) Emotional facial sensing and multimodal fusion in a continuous 2d affective space. J Ambient Intell Hum Comput 3(1):31–46. https://doi.org/10.1007/s12652-011-0087-6
Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A (2014) Improvement of the elderly quality of life and care through smart emotion regulation. In: Pecchia L, Chen LL, Nugent C, Bravo J (eds) Ambient assisted living and daily activities. Springer, Cham, pp 348–355. https://doi.org/10.1007/978-3-319-13105-4_50
Fernández-Caballero A, Martínez-Rodrigo A, Pastor JM, Castillo JC, Lozano-Monasor E, López MT, Zangr’oniz R, Latorre JM, Fernández-Sotos A (2016) Smart environment architecture for emotion detection and regulation. J Biomed Inform 64:55–73. https://doi.org/10.1016/j.jbi.2016.09.015
Fernández-Caballero A, Navarro E, Fernández-Sotos P, González P, Ricarte JJ, Latorre JM, Rodriguez-Jimenez R (2017) Human-avatar symbiosis for the treatment of auditory verbal hallucinations in schizophrenia through virtual/augmented reality and brain–computer interfaces. Front Neuroinform 11:64. https://doi.org/10.3389/fninf.2017.00064
Fernández-Sotos P, Navarro E, Torio I, Dompablo M, Fernández-Caballero A, Rodriguez-Jimenez R (2018) Pharmacological interventions in social cognition deficits: a systematic mapping review. Psychiatry Res 270:57–67. https://doi.org/10.1016/j.psychres.2018.09.012
Fernández-Sotos P, Torio I, Fernández-Caballero A, Navarro E, González P, Dompablo M, Rodriguez-Jimenez R (2019) Social cognition remediation interventions: a systematic mapping review. PLoS One 14:e0218720
García AS, Molina JP, Martínez D, González P (2008) Enhancing collaborative manipulation through the use of feedback and awareness in CVEs. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. ACM, New York, p 32. https://doi.org/10.1145/1477862.1477904
García AS, Navarro E, Fernández-Caballero A, González P (2018) Towards the design of avatar-based therapies for enhancing facial affect recognition. In: Novais P, Jung JJ, Villarrubia G, Fernández-Caballero A, Navarro E, González P, Carneiro D, Pinto A, Campbell AT, Durães D (eds) Ambient intelligence—software and applications, 9th international symposium on ambient intelligence. Springer, Cham, pp 306–313
Garrido MV, Lopes D, Prada M, Rodrigues D, Jerónimo R, Mourão RP (2016) The many faces of a face: comparing stills and videos of facial expressions in eight dimensions (SAVE database). Behav Res Methods 49(4):1343–1360. https://doi.org/10.3758/s13428-016-0790-5
Gottesman II (1991) Schizophrenia genesis: the origins of madness. WH Freeman/Times Books/Henry Holt & Co, New York
Green MF, Penn DL, Bentall R, Carpenter WT, Gaebel W, Gur RC, Kring AM, Park S, Silverstein SM, Heinssen R (2008) Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophr Bull 34(6):1211–1220. https://doi.org/10.1093/schbul/sbm145
Hair JF Jr, Hult GTM, Ringle C, Sarstedt M (2014) A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publications, Thousand Oaks
Henseler J, Ringle CM, Sarstedt M (2015) A new criterion for assessing discriminant validity in variance-based structural equation modeling. J Acad Mark Sci 43(1):115–135
Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB (2013) Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord 43(1):34–44. https://doi.org/10.1007/s10803-012-1544-6
Khalilzadeh J, Ozturk AB, Bilgihan A (2017) Security-related factors in extended UTAUT model for NFC based mobile payment in the restaurant industry. Comput Human Behav 70:460–474
Krishnappa Babu PR, Lahiri U (2019) Classification approach for understanding implications of emotions using eye-gaze. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-019-01329-8
Lahera G, Ruiz-Murugarren S, Iglesias P, Ruiz-Bennasar C, Herrería E, Montes JM, Fernández-Liria A (2012) Social cognition and global functioning in bipolar disorder. J Nerv Menta Dis 200(2):135–141. https://doi.org/10.1097/NMD.0b013e3182438eae
Laumer S, Gubler F, Maier C, Weitzel T (2018) Job seekers’ acceptance of job recommender systems: results of an empirical study. In: Proceedings of the 51st Hawaii international conference on system sciences. Curran Associates, Inc., pp 3172–3181
Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Hum Comput 8(4):567–578. https://doi.org/10.1007/s12652-017-0464-x
Manis KT, Choi D (2018) The virtual reality hardware acceptance model (vr-ham): Extending and individuating the technology acceptance model (TAM) for virtual reality hardware. J Bus Res. https://doi.org/10.1016/j.jbusres.2018.10.021
Marcoulides GA, Saunders C (2006) PLS: a silver bullet? Manag Inf Syst Q 30(2):1
McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex sdk: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, pp 3723–3726
Mendoza-Palechor F, Menezes ML, Sant’Anna A, Ortiz-Barrios M, Samara A, Galway L (2018) Affective recognition from EEG signals: an integrated data-mining approach. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-018-1065-z
Oechslein O, Fleischmann M, Hess T (2014) An application of utaut2 on social recommender systems: incorporating social information for performance expectancy. In: 2014 47th Hawaii international conference on system sciences. IEEE, pp 3297–3306
Oliver M, Teruel M, Molina J, Romero-Ayuso D, González P (2018) Ambient intelligence environment for home cognitive telerehabilitation. Sensors 18(11):3671
Pinkham AE, Penn DL, Green MF, Buck B, Healey K, Harvey PD (2014) The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophr Bull 40(4):813–823. https://doi.org/10.1093/schbul/sbt081
Pinkham AE, Penn DL, Green MF, Harvey PD (2016) Social cognition psychometric evaluation: results of the initial psychometric study. Schizophr Bull 42(2):494–504. https://doi.org/10.1093/schbul/sbv056
Roark DA, Barrett SE, Spence MJ, Abdi H, O’Toole AJ (2003) Psychological and neural perspectives on the role of motion in face recognition. Behav Cogn Neurosci Rev 2(1):15–46. https://doi.org/10.1177/1534582303002001002
Rosenberg H, McDonald S, Dethier M, Kessels RP, Westbrook RF (2014) Facial emotion recognition deficits following moderate-severe traumatic brain injury (TBI): re-examining the valence effect and the role of emotion intensity. J Int Neuropsychol Soc 20(10):994–1003. https://doi.org/10.1017/S1355617714000940
Rus-Calafell M, Gutiérrez-Maldonado J, Ribas-Sabaté J (2014) A virtual reality-integrated program for improving social skills in patients with schizophrenia: a pilot study. J Behav Ther Exp Psychiatry 45(1):81–89. https://doi.org/10.1016/j.jbtep.2013.09.002
Samara A, Galway L, Bond R, Wang H (2019) Affective state detection via facial expression analysis within a human–computer interaction context. J Ambient Intell Hum Comput 10(6):2175–2184. https://doi.org/10.1007/s12652-017-0636-8
Souto YM, Campo MV, Llenderrozas FD, Álvarez MR, Mateos R, Caballero AAG (2018) Randomized clinical trial with e-MotionalTraining® 1.0 for social cognition rehabilitation in schizophrenia. Front Psychiatry 26(9):40. https://doi.org/10.3389/fpsyt.2018.00040
Teruel MA, Navarro E, González P (2017) Exploiting awareness for the development of collaborative rehabilitation systems. Mobile Inf Syst. https://doi.org/10.1155/2017/4714328
Tracy JL, Randles D, Steckler CM (2015) The nonverbal communication of emotions. Curr Opin Behav Sci 3:25–30. https://doi.org/10.1016/j.cobeha.2015.01.001
Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q 27:425–478
Venkatesh V, Thong JY, Xu X (2012) Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 36:157–178
Woods S, Walters M, Koay KL, Dautenhahn K (2006) Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: 9th ieee international workshop on advanced motion control, 2006. IEEE, pp 750–755
Yang YJD, Allen T, Abdullahi SM, Pelphrey K, Volkmar F, Chapman S (2017) Brain responses to biological motion predict treatment outcome in young adults with autism receiving virtual reality social cognition training: preliminary findings. Behav Res Ther 93:55–66. https://doi.org/10.1016/j.brat.2017.03.014
This work was partially supported by Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (AEI)/European Regional Development Fund (FEDER, UE) under EmoBioFeedback (DPI2016-80894-R), Vi-SMARt (TIN2016-79100-R) and HA-SYMBIOSIS (TIN2015-72931-EXP) Grants, and by Biomedical Research Networking Centre in Mental Health (CIBERSAM) of the Instituto de Salud Carlos III.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
García, A.S., Fernández-Sotos, P., Fernández-Caballero, A. et al. Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits. J Ambient Intell Human Comput 11, 4513–4524 (2020). https://doi.org/10.1007/s12652-019-01418-8
- Social cognition
- Affect recognition
- Virtual reality