Inclusion of third-person perspective in CAVE-like immersive 3D virtual reality role-playing games for social reciprocity training of children with an autism spectrum disorder

Abstract

The present study aimed to improve the ability of children with autism to recognize emotions correctly. We used our third-person perspective role-playing game (TPP-RPG) method to teach social skills and help develop an improved understanding of the six basic emotions. The experiment was divided into two phases: The first involved working with traditional figure card emotional recognition and the second involved a subject entering a 3D cave automatic virtual environment (CAVE) to engage with interactive games. While the traditional graphic card is a static picture that represents one of the six basic human emotions, the virtual reality of CAVE-like immersive 3D role-playing games enables the use of actual picture scene syntheses plus the animation of 3D characters to express emotions. The participating children were instructed to role-play with (1) three-dimensional (3D) virtual role animations and observe (2) two different real-time switchable role-play animations of themselves and their counterpart socially interacting. This single-subject study was based on multiple-baseline, across-subject design and involved 5 weeks of TPP-RPG training intervention. From this research activity, we found that the role-play performance of all three participants rose substantially during the intervention phase and remained significantly higher in the maintenance phase compared to their baseline levels.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. 1.

    Adjorlu, A., Hussain, A., Mødekjær, C., Austad, N.W.: Head-mounted display-based virtual reality social story as a tool to teach social skills to children diagnosed with autism spectrum disorder. In: 2017 IEEE Virtual Reality Workshop on K-12 Embodied Learning Through Virtual & Augmented Reality (KELVAR). IEEE (2018)

  2. 2.

    Ahlers, K.P., Gabrielsen, T.P., Lewis, D., Brady, A.M., Litchford, A.: Supporting individuals with autism spectrum disorder in understanding and coping with complex social emotional issues. Sch. Psychol. Int. 38(6), 586–607 (2017)

    Article  Google Scholar 

  3. 3.

    American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). American Psychiatric Pub, Philadelphia (2013)

    Google Scholar 

  4. 4.

    Anderson, P.L., Rothbaum, B.O., Hodges, L.: Virtual reality: using the virtual world to improve quality of life in the real world. Bull. Menn. Clin. 65(1: Special issue), 78–91 (2001)

    Article  Google Scholar 

  5. 5.

    Axe, J.B., Evans, C.J.: Using video modeling to teach children with PDD-NOS to respond to facial expressions. Res. Autism Spect. Disord. 6(3), 1176–1185 (2012)

    Article  Google Scholar 

  6. 6.

    Ayres, K.M., Langone, J.: Intervention and instruction with video for students with autism: a review of the literature. Educ. Train. Dev. Disabil. 40, 183–196 (2005)

    Google Scholar 

  7. 7.

    Baron-Cohen, S.E., Tager-Flusberg, H.E., Cohen, D.J.: Understanding Other Minds: Perspectives from Developmental Cognitive Neuroscience. Oxford University Press, Oxford (2000)

    Google Scholar 

  8. 8.

    Baron-Cohen, S., Belmonte, M.K.: Autism: a window onto the development of the social and the analytic brain. Annu. Rev. Neurosci. 28, 109–126 (2005)

    Article  Google Scholar 

  9. 9.

    Baron-Cohen, S., Leslie, A.M., Frith, U.: Does the autistic child have a “theory of mind”. Cognition 21(1), 37–46 (1985)

    Article  Google Scholar 

  10. 10.

    Baker, J.: The Social Skills Picture Book: Teaching Communication, Play and Emotion. Future Horizons, Arlington (2017)

    Google Scholar 

  11. 11.

    Becker-Weidman, A.: Effects of early maltreatment on development: a descriptive study using the Vineland Adaptive Behavior Scales-II. Child Welf. 88(2), 137–161 (2009)

    Google Scholar 

  12. 12.

    Bellini, S., Akullian, J.: A meta-analysis of video modeling and video self-modeling interventions for children and adolescents with autism spectrum disorders. Except. Child. 73(3), 264–287 (2007)

    Article  Google Scholar 

  13. 13.

    Bierman, K.L., Greenberg, M.T., Coie, J.D., Dodge, K.A., Lochman, J.E., McMahon, R.J.: Social and Emotional Skills Training for Children: The Fast Track Friendship Group Manual. Guilford Publications, New York (2017)

    Google Scholar 

  14. 14.

    Bölte, S., Hallmayer, J. (eds.): Autism Spectrum Conditions: FAQs on Autism, Asperger Syndrome, and Atypical Autism Answered by International Experts. Hogrefe Publishing, Göttingen (2011)

    Google Scholar 

  15. 15.

    Cameirão, M.S., i Badia, S.B., Oller, E.D., Verschure, P.F.: Neurorehabilitation using the virtual reality based rehabilitation gaming system: methodology, design, psychometrics, usability and validation. J. Neuroeng. Rehabil. 7(1), 48 (2010)

    Article  Google Scholar 

  16. 16.

    Charlop-Christy, M.H., Daneshvar, S.: Using video modeling to teach perspective taking to children with autism. J. Posit. Behav. Interv. 5(1), 12–21 (2003)

    Article  Google Scholar 

  17. 17.

    Charlop, M.H., Lang, R., Rispoli, M.: Lights, camera, action! Teaching play and social skills to children with autism spectrum disorder through video modeling. In: Charlop, M.H., Lang, R., Rispoli, M. (eds.) Play and Social Skills for Children with Autism Spectrum disorder, pp. 71–94. Springer, Cham (2018)

    Google Scholar 

  18. 18.

    Chen, C.H., Lee, I.J., Lin, L.Y.: Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res. Dev. Disabil. 36, 396–403 (2015)

    Article  Google Scholar 

  19. 19.

    Chen, C.H., Lee, I.J., Lin, L.Y.: Augmented reality-based video-modeling storybook of nonverbal facial cues for children with autism spectrum disorder to improve their perceptions and judgments of facial expressions and emotions. Comput. Hum. Behav. 55, 477–485 (2016)

    Article  Google Scholar 

  20. 20.

    Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the facial action coding system. In: Coan, J.A., Allen, J.J.B. (eds.) The Handbook of Emotion Elicitation and Assessment, vol. 1, no. 3, pp. 203–221. Oxford University Press, New York (2007)

    Google Scholar 

  21. 21.

    Davies, R.R., Rogers, E.S.: Social skills training with persons who are mentally retarded. Ment. Retard. 23(4), 186 (1985)

    Google Scholar 

  22. 22.

    Ekman, P.: Emotional and conversational nonverbal signals. In: Larrazabal, J.M., Miranda, L.A.P. (eds.) Language, Knowledge, and Representation, pp. 39–50. Springer, Dordrecht (2004)

    Google Scholar 

  23. 23.

    Ghanouni, P., Jarus, T., Zwicker, J.G., Lucyshyn, J., Mow, K., Ledingham, A.: Social stories for children with autism spectrum disorder: validating the content of a virtual reality program. J. Autism Dev. Disord. 49(2), 660–668 (2019)

    Article  Google Scholar 

  24. 24.

    Goldstein, S., Ozonoff, S. (eds.): Assessment of Autism Spectrum Disorder. Guilford Publications, New York (2018)

    Google Scholar 

  25. 25.

    Gray, C.: Social Stories. Future Horizons, Arlington, TX (1994)

    Google Scholar 

  26. 26.

    Hillier, A., Fish, T., Cloppert, P., Beversdorf, D.Q.: Outcomes of a social and vocational skills support group for adolescents and young adults on the autism spectrum. Focus Autism Other Dev. Disabil. 22(2), 107–115 (2007)

    Article  Google Scholar 

  27. 27.

    Howlin, P., Yates, P.: The potential effectiveness of social skills groups for adults with autism. Autism 3(3), 299–307 (1999)

    Article  Google Scholar 

  28. 28.

    Ip, H.H., Wong, S.W., Chan, D.F., Byrne, J., Li, C., Yuan, V.S., Lau, K.S.Y., Wong, J.Y.: Enhance emotional and social adaptation skills for children with autism spectrum disorder: a virtual reality enabled approach. Comput. Educ. 117, 1–15 (2018)

    Article  Google Scholar 

  29. 29.

    Karal, M.A., Wolfe, P.S.: Social story effectiveness on social interaction for students with autism: a review of the literature. Educ. Train. Autism Dev. Disabil. 53(1), 44–58 (2018)

    Google Scholar 

  30. 30.

    Lee, S.Y., Lo, Y.Y., Lo, Y.: Teaching functional play skills to a young child with autism spectrum disorder through video self-modeling. J. Autism Dev. Disord. 47(8), 2295–2306 (2017)

    Article  Google Scholar 

  31. 31.

    Lee, I.J., Chen, C.H., Wang, C.P., Chung, C.H.: Augmented reality plus concept map technique to teach children with ASD to use social cues when meeting and greeting. Asia Pac. Educ. Res. 27(3), 227–243 (2018)

    Article  Google Scholar 

  32. 32.

    Lee, I.J., Lin, L.Y., Chen, C.H., Chung, C.H.: How to create suitable augmented reality application to teach social skills for children with ASD. In: Mohamudally, N. (eds.) State of the Art Virtual Reality and Augmented Reality Knowhow, vol. 8, pp. 119–138. IntechOpen, London (2018)

    Google Scholar 

  33. 33.

    Lillard, A.S.: Why do the children (pretend) play? Trends Cogn. Sci. 21(11), 826–834 (2017)

    MathSciNet  Article  Google Scholar 

  34. 34.

    Nikopoulos, C.K., Keenan, M.: Using video modeling to teach complex social sequences to children with autism. J. Autism Dev. Disord. 37(4), 678–693 (2007)

    Article  Google Scholar 

  35. 35.

    Parsons, S., Leonard, A., Mitchell, P.: Virtual environments for social skills training: comments from two adolescents with autistic spectrum disorder. Comput. Educ. 47(2), 186–206 (2006)

    Article  Google Scholar 

  36. 36.

    Pearson, A., Ropar, D., Hamilton, A.F.D.: A review of visual perspective taking in autism spectrum disorder. Front. Hum. Neurosci. 7, 652 (2013)

    Article  Google Scholar 

  37. 37.

    Reynhout, G., Carter, M.: Social Stories™ for children with disabilities. J. Autism Dev. Disord. 36(4), 445–469 (2006)

    Article  Google Scholar 

  38. 38.

    Richards, S.B.: Single Subject Research: Applications in Educational Settings. Cengage Learning, Boston (2018)

    Google Scholar 

  39. 39.

    Riva, G.: Virtual reality in psychotherapy. Cyberpsychol. Behav. 8(3), 220–230 (2005)

    Article  Google Scholar 

  40. 40.

    Sharples, S., Cobb, S., Moody, A., Wilson, J.R.: Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29(2), 58–69 (2008)

    Article  Google Scholar 

  41. 41.

    Shillingsburg, M.A., Cariveau, T.: Responding to conversational partners’ behavior: teaching children with autism strategies to recapture the interest of others. Evid. Based Commun. Assess. Interv. 10(3–4), 143–147 (2016)

    Article  Google Scholar 

  42. 42.

    Smyth, J.M.: Beyond self-selection in video game play: an experimental examination of the consequences of massively multiplayer online role-playing game play. CyberPsychol. Behav. 10(5), 717–721 (2007)

    Article  Google Scholar 

  43. 43.

    Tustin, F.: Autistic Barriers in Neurotic Patients. Routledge, London (2018)

    Google Scholar 

  44. 44.

    Van Wermeskerken, M., Grimmius, B., van Gog, T.: Attention to the model’s face when learning from video modeling examples in adolescents with and without autism spectrum disorder. J. Comput. Assist. Learn. 34(1), 32–41 (2018)

    Article  Google Scholar 

  45. 45.

    Wallace, S., Parsons, S., Westbury, A., White, K., White, K., Bailey, A.: Sense of presence and atypical social judgments in immersive virtual environments: responses of adolescents with autism spectrum disorders. Autism 14(3), 199–213 (2010)

    Article  Google Scholar 

  46. 46.

    Yuan, S.N.V., Ip, H.H.S.: Using virtual reality to train emotional and social skills in children with autism spectrum disorder. Lond. J. Prim. Care 10(4), 110–112 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to the participants, therapists, and family members who participated in the study as well as the participants who assisted in the various phases of the study. We would also like to thank the individuals who participated in this research and the Autism and Developmental Research Center in Taiwan.

Funding

This work is supported by the Ministry of Science and Technology of Taiwan (MOST 107-2218-E-027 -013 -MY2). The authors thank the referees very much for their valuable comments and suggestions on this paper.

Author information

Affiliations

Authors

Corresponding author

Correspondence to I-Jui Lee.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1

Appendix 1

Parts of the social scripts used in the experiments

  Scenes Related character Social scripts Related character feelings Respond to each other’s social actions
1 Park Friends Joe borrowed a toy car from Tom. Joe accidentally broke a wheel while playing with the car. Tom saw it and said: “Hum! You broke my toy car. I don’t want to be nice to you! “ Anger Apologize (bow)
2 Park Friends When Tina was playing in the park, she met her new friend Allen. Allen said to Tina: “Hi, we will play together in the future, we will be good friends.” Happiness Hold hands
3 Living room Mother Peter was at home because his mother had gone shopping. She was gone a long time and he had to spend a long time waiting for her. “Why didn’t my mother come back yet? Why is she so late?” Fear Ask hugs
4 Living room Mother Today, my mother cooked John’s favorite food for dinner. John said at dinner: “Awesome, I am getting my favorite—half-boiled eggs tonight!” Happiness Ask hugs
5 Park Friends On the way home, when Lion was going through an alley, he saw his good friend Ali suddenly appears. “Wow! Why are you hiding here? You scared me!” he exclaimed. Surprise Express surprise (action)
6 Living room Mother When Lina came home from practicing riding a bicycle, her mother saw her palm was injured. Her mother asked what happened to Lina and she said: “I accidentally fell and it hurts!” Sadness Ask hugs
7 Class room Friends Gino and Archie are good friends and both like animations. At school, Gino said to Archie: “Did you watch that animation yesterday? Super funny!” Happiness High five
8 Class room Friends Bob borrowed a pencil with Joanna, but Bob lost the pencil. Joanna complained to the teacher: “Teacher, he lost my pencil!” Disgust Raise hand
9 Park Friends Danny was lost in a hide-and-seek game with friends and said: “Where am I? I have accidentally run into a strange place” Fear Ask for help
10 Class room Classmate Bill made an appointment with his classmates to play after class, but when he was about to go out he saw that it was raining: “Oh! Why is it raining ~ Now I can’t go out to play!” Sadness Talk to classmate (action)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tsai, W., Lee, I. & Chen, C. Inclusion of third-person perspective in CAVE-like immersive 3D virtual reality role-playing games for social reciprocity training of children with an autism spectrum disorder. Univ Access Inf Soc (2020). https://doi.org/10.1007/s10209-020-00724-9

Download citation

Keywords

  • CAVE-like immersive environments
  • Autism
  • Third-person perspective
  • Virtual Reality CAVE
  • Role-playing game
  • Human–computer interaction
  • Empathy