Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits

Abstract

This paper focuses on the validation of a tool designed to improve affect recognition, a fundamental aspect of social cognition as it greatly affects the functionality and quality of life of patients with mental disorders. The presented tool facilitates the generation of multi-modal avatar-based therapies by mental health professionals in this important clinical domain. Moreover, the tool for remediation of social cognitive deficits may be customised to each patient’s impairment. This paper describes how the tool was assessed by therapists after viewing a video explaining its most relevant aspects. The participants were asked to fill in a questionnaire based on UTAUT2 for the study of the acceptance and use of this technology. In light of the results obtained from 41 therapists about their intention of use, the most important statement is that their interest for this kind of tools is high. Nonetheless, there are some factors that negatively affect their behavioural intention.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Abirached B, Zhang Y, Aggarwal JK, Fernandes T, Carlos J, Orvalho V (2011) Improving communication skills of children with ASDs through interaction with virtual characters. In: IEEE 1st international conference on serious games and applications for health, IEEE, pp 3493–3502

  2. Adery LH, Ichinose M, Torregrossa LJ, Wade J, Nichols H, Bekele E, Bian D, Gizdic A, Granholm E, Sarkar N, Park S (2018) The acceptability and feasibility of a novel virtual reality based social skills training game for schizophrenia: preliminary findings. Psychiatry Res 270:496–502. https://doi.org/10.1016/j.psychres.2018.10.014

    Article  Google Scholar 

  3. Akalin N, Kiselev A, Kristoffersson A, Loutfi A (2017) An evaluation tool of the effect of robots in eldercare on the sense of safety and security. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan J-J, Eyssel F, He H (eds) International conference on social robotics. Springer, Cham, pp 628–637

    Google Scholar 

  4. Barclay D, Higgins C, Thompson R (1995) The partial least squares (PLS) approach to casual modeling: personal computer adoption ans use as an illustration. Technol Stud 2:285–309

    Google Scholar 

  5. Baron-Cohen S, Wheelwright S, Skinner R, Martin J, Clubley E (2001) The autism-spectrum quotient (AQ): evidence from asperger syndrome/high-functioning autism, malesand females, scientists and mathematicians. J Autism Dev Disord 31(1):5–17. https://doi.org/10.1023/A:1005653411471

    Article  Google Scholar 

  6. Beer JM, Fisk AD, Rogers WA (2009) Emotion recognition of virtual agents facial expressions: the effects of age and emotion intensity. Proc Hum Factors Ergon Soc 53(2):131–135. https://doi.org/10.1177/154193120905300205

    Article  Google Scholar 

  7. Brothers L (1990) The neural basis of primate social communication. Motiv Emot 14(2):81–91. https://doi.org/10.1007/BF00991637

    Article  Google Scholar 

  8. Castillo JC, Castro-González Á, Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A, Salichs MA (2016) Software architecture for smart emotion recognition and regulation of the ageing adult. Cogn Comput 8(2):357–367. https://doi.org/10.1007/s12559-016-9383-y

    Article  Google Scholar 

  9. Cerezo E, Hupont I, Baldassarri S, Ballano S (2012) Emotional facial sensing and multimodal fusion in a continuous 2d affective space. J Ambient Intell Hum Comput 3(1):31–46. https://doi.org/10.1007/s12652-011-0087-6

    Article  Google Scholar 

  10. Fernández-Caballero A, Latorre JM, Pastor JM, Fernández-Sotos A (2014) Improvement of the elderly quality of life and care through smart emotion regulation. In: Pecchia L, Chen LL, Nugent C, Bravo J (eds) Ambient assisted living and daily activities. Springer, Cham, pp 348–355. https://doi.org/10.1007/978-3-319-13105-4_50

    Google Scholar 

  11. Fernández-Caballero A, Martínez-Rodrigo A, Pastor JM, Castillo JC, Lozano-Monasor E, López MT, Zangr’oniz R, Latorre JM, Fernández-Sotos A (2016) Smart environment architecture for emotion detection and regulation. J Biomed Inform 64:55–73. https://doi.org/10.1016/j.jbi.2016.09.015

    Article  Google Scholar 

  12. Fernández-Caballero A, Navarro E, Fernández-Sotos P, González P, Ricarte JJ, Latorre JM, Rodriguez-Jimenez R (2017) Human-avatar symbiosis for the treatment of auditory verbal hallucinations in schizophrenia through virtual/augmented reality and brain–computer interfaces. Front Neuroinform 11:64. https://doi.org/10.3389/fninf.2017.00064

    Article  Google Scholar 

  13. Fernández-Sotos P, Navarro E, Torio I, Dompablo M, Fernández-Caballero A, Rodriguez-Jimenez R (2018) Pharmacological interventions in social cognition deficits: a systematic mapping review. Psychiatry Res 270:57–67. https://doi.org/10.1016/j.psychres.2018.09.012

    Article  Google Scholar 

  14. Fernández-Sotos P, Torio I, Fernández-Caballero A, Navarro E, González P, Dompablo M, Rodriguez-Jimenez R (2019) Social cognition remediation interventions: a systematic mapping review. PLoS One 14:e0218720

    Article  Google Scholar 

  15. García AS, Molina JP, Martínez D, González P (2008) Enhancing collaborative manipulation through the use of feedback and awareness in CVEs. In: Proceedings of the 7th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. ACM, New York, p 32. https://doi.org/10.1145/1477862.1477904

    Google Scholar 

  16. García AS, Navarro E, Fernández-Caballero A, González P (2018) Towards the design of avatar-based therapies for enhancing facial affect recognition. In: Novais P, Jung JJ, Villarrubia G, Fernández-Caballero A, Navarro E, González P, Carneiro D, Pinto A, Campbell AT, Durães D (eds) Ambient intelligence—software and applications, 9th international symposium on ambient intelligence. Springer, Cham, pp 306–313

    Google Scholar 

  17. Garrido MV, Lopes D, Prada M, Rodrigues D, Jerónimo R, Mourão RP (2016) The many faces of a face: comparing stills and videos of facial expressions in eight dimensions (SAVE database). Behav Res Methods 49(4):1343–1360. https://doi.org/10.3758/s13428-016-0790-5

    Article  Google Scholar 

  18. Gottesman II (1991) Schizophrenia genesis: the origins of madness. WH Freeman/Times Books/Henry Holt & Co, New York

    Google Scholar 

  19. Green MF, Penn DL, Bentall R, Carpenter WT, Gaebel W, Gur RC, Kring AM, Park S, Silverstein SM, Heinssen R (2008) Social cognition in schizophrenia: an NIMH workshop on definitions, assessment, and research opportunities. Schizophr Bull 34(6):1211–1220. https://doi.org/10.1093/schbul/sbm145

    Article  Google Scholar 

  20. Hair JF Jr, Hult GTM, Ringle C, Sarstedt M (2014) A primer on partial least squares structural equation modeling (PLS-SEM). Sage Publications, Thousand Oaks

    MATH  Google Scholar 

  21. Henseler J, Ringle CM, Sarstedt M (2015) A new criterion for assessing discriminant validity in variance-based structural equation modeling. J Acad Mark Sci 43(1):115–135

    Article  Google Scholar 

  22. Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB (2013) Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord 43(1):34–44. https://doi.org/10.1007/s10803-012-1544-6

    Article  Google Scholar 

  23. Khalilzadeh J, Ozturk AB, Bilgihan A (2017) Security-related factors in extended UTAUT model for NFC based mobile payment in the restaurant industry. Comput Human Behav 70:460–474

    Article  Google Scholar 

  24. Krishnappa Babu PR, Lahiri U (2019) Classification approach for understanding implications of emotions using eye-gaze. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-019-01329-8

    Article  Google Scholar 

  25. Lahera G, Ruiz-Murugarren S, Iglesias P, Ruiz-Bennasar C, Herrería E, Montes JM, Fernández-Liria A (2012) Social cognition and global functioning in bipolar disorder. J Nerv Menta Dis 200(2):135–141. https://doi.org/10.1097/NMD.0b013e3182438eae

    Article  Google Scholar 

  26. Laumer S, Gubler F, Maier C, Weitzel T (2018) Job seekers’ acceptance of job recommender systems: results of an empirical study. In: Proceedings of the 51st Hawaii international conference on system sciences. Curran Associates, Inc., pp 3172–3181

  27. Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Hum Comput 8(4):567–578. https://doi.org/10.1007/s12652-017-0464-x

    Article  Google Scholar 

  28. Manis KT, Choi D (2018) The virtual reality hardware acceptance model (vr-ham): Extending and individuating the technology acceptance model (TAM) for virtual reality hardware. J Bus Res. https://doi.org/10.1016/j.jbusres.2018.10.021

    Article  Google Scholar 

  29. Marcoulides GA, Saunders C (2006) PLS: a silver bullet? Manag Inf Syst Q 30(2):1

    Article  Google Scholar 

  30. McDuff D, Mahmoud A, Mavadati M, Amr M, Turcot J, Kaliouby Re (2016) Affdex sdk: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, pp 3723–3726

  31. Mendoza-Palechor F, Menezes ML, Sant’Anna A, Ortiz-Barrios M, Samara A, Galway L (2018) Affective recognition from EEG signals: an integrated data-mining approach. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-018-1065-z

    Article  Google Scholar 

  32. Oechslein O, Fleischmann M, Hess T (2014) An application of utaut2 on social recommender systems: incorporating social information for performance expectancy. In: 2014 47th Hawaii international conference on system sciences. IEEE, pp 3297–3306

  33. Oliver M, Teruel M, Molina J, Romero-Ayuso D, González P (2018) Ambient intelligence environment for home cognitive telerehabilitation. Sensors 18(11):3671

    Article  Google Scholar 

  34. Pinkham AE, Penn DL, Green MF, Buck B, Healey K, Harvey PD (2014) The social cognition psychometric evaluation study: results of the expert survey and RAND panel. Schizophr Bull 40(4):813–823. https://doi.org/10.1093/schbul/sbt081

    Article  Google Scholar 

  35. Pinkham AE, Penn DL, Green MF, Harvey PD (2016) Social cognition psychometric evaluation: results of the initial psychometric study. Schizophr Bull 42(2):494–504. https://doi.org/10.1093/schbul/sbv056

    Article  Google Scholar 

  36. Roark DA, Barrett SE, Spence MJ, Abdi H, O’Toole AJ (2003) Psychological and neural perspectives on the role of motion in face recognition. Behav Cogn Neurosci Rev 2(1):15–46. https://doi.org/10.1177/1534582303002001002

    Article  Google Scholar 

  37. Rosenberg H, McDonald S, Dethier M, Kessels RP, Westbrook RF (2014) Facial emotion recognition deficits following moderate-severe traumatic brain injury (TBI): re-examining the valence effect and the role of emotion intensity. J Int Neuropsychol Soc 20(10):994–1003. https://doi.org/10.1017/S1355617714000940

    Article  Google Scholar 

  38. Rus-Calafell M, Gutiérrez-Maldonado J, Ribas-Sabaté J (2014) A virtual reality-integrated program for improving social skills in patients with schizophrenia: a pilot study. J Behav Ther Exp Psychiatry 45(1):81–89. https://doi.org/10.1016/j.jbtep.2013.09.002

    Article  Google Scholar 

  39. Samara A, Galway L, Bond R, Wang H (2019) Affective state detection via facial expression analysis within a human–computer interaction context. J Ambient Intell Hum Comput 10(6):2175–2184. https://doi.org/10.1007/s12652-017-0636-8

    Article  Google Scholar 

  40. Souto YM, Campo MV, Llenderrozas FD, Álvarez MR, Mateos R, Caballero AAG (2018) Randomized clinical trial with e-MotionalTraining® 1.0 for social cognition rehabilitation in schizophrenia. Front Psychiatry 26(9):40. https://doi.org/10.3389/fpsyt.2018.00040

    Article  Google Scholar 

  41. Teruel MA, Navarro E, González P (2017) Exploiting awareness for the development of collaborative rehabilitation systems. Mobile Inf Syst. https://doi.org/10.1155/2017/4714328

    Article  Google Scholar 

  42. Tracy JL, Randles D, Steckler CM (2015) The nonverbal communication of emotions. Curr Opin Behav Sci 3:25–30. https://doi.org/10.1016/j.cobeha.2015.01.001

    Article  Google Scholar 

  43. Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Q 27:425–478

    Article  Google Scholar 

  44. Venkatesh V, Thong JY, Xu X (2012) Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 36:157–178

    Article  Google Scholar 

  45. Woods S, Walters M, Koay KL, Dautenhahn K (2006) Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: 9th ieee international workshop on advanced motion control, 2006. IEEE, pp 750–755

  46. Yang YJD, Allen T, Abdullahi SM, Pelphrey K, Volkmar F, Chapman S (2017) Brain responses to biological motion predict treatment outcome in young adults with autism receiving virtual reality social cognition training: preliminary findings. Behav Res Ther 93:55–66. https://doi.org/10.1016/j.brat.2017.03.014

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by Spanish Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (AEI)/European Regional Development Fund (FEDER, UE) under EmoBioFeedback (DPI2016-80894-R), Vi-SMARt (TIN2016-79100-R) and HA-SYMBIOSIS (TIN2015-72931-EXP) Grants, and by Biomedical Research Networking Centre in Mental Health (CIBERSAM) of the Instituto de Salud Carlos III.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Antonio Fernández-Caballero.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 119430 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

García, A.S., Fernández-Sotos, P., Fernández-Caballero, A. et al. Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits. J Ambient Intell Human Comput 11, 4513–4524 (2020). https://doi.org/10.1007/s12652-019-01418-8

Download citation

Keywords

  • Social cognition
  • Affect recognition
  • Virtual reality
  • Avatar