Responses during Facial Emotional Expression Recognition Tasks Using Virtual Reality and Static IAPS Pictures for Adults with Schizophrenia

  • Esubalew Bekele
  • Dayi Bian
  • Zhi Zheng
  • Joel Peterman
  • Sohee Park
  • Nilanjan Sarkar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8526)


Technology-assisted intervention has the potential to adaptively individualize and improve outcomes of traditional schizophrenia (SZ) intervention. Virtual reality (VR) technology, in particular, has the potential to simulate real world social and communication interactions and hence could be useful as a therapeutic platform for SZ. Emotional face recognition is considered among the core building blocks of social communication. Studies have shown that emotional face processing and understanding is impaired in patients with SZ. The current study develops a novel VR-based system that presents avatars that can change their facial emotion dynamically for emotion recognition tasks. Additionally, this system allows real-time measurement of physiological signals and eye gaze during the emotion recognition tasks, which can be used to gain insight about the emotion recognition process in SZ population. This study further compares VR-based facial emotion recognition with that of the more traditional emotion recognition from static faces using a small usability study. Results from the usability study suggest that VR could be a viable platform for SZ intervention and implicit signals such as physiological signals and eye gaze can be utilized to better understand the underlying pattern that is not available from user reports and performance alone.


facial expression emotion recognition virtual reality IAPS adaptive interaction eye tracking physiological processing schizophrenia intervention 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hempel, R.J., Tulen, J.H., van Beveren, N.J., van Steenis, H.G., Mulder, P.G., Hengeveld, M.W.: Physiological responsivity to emotional pictures in schizophrenia. Journal of Psychiatric Research 39(5), 509–518 (2005)CrossRefGoogle Scholar
  2. 2.
    Bleuler, E.: Dementia praecox or the group of schizophrenias. International University Press, New York (1911)Google Scholar
  3. 3.
    Andreasen, N.C.: Scale for the assessment of negative symptoms. University of Iowa, Iowa City (1983)Google Scholar
  4. 4.
    Couture, S.M., Granholm, E.L., Fish, S.C.: A path model investigation of neurocognition, theory of mind, social competence, negative symptoms and real-world functioning in schizophrenia. Schizophrenia Research 125(2), 152–160 (2011)CrossRefGoogle Scholar
  5. 5.
    Herbener, E.S., Song, W., Khine, T.T., Sweeney, J.A.: What aspects of emotional functioning are impaired in schizophrenia? Schizophrenia Research 98(1), 239–246 (2008)CrossRefGoogle Scholar
  6. 6.
    Kring, A.M., Moran, E.K.: Emotional response deficits in schizophrenia: Insights from affective science. Schizophrenia Bulletin 34(5), 819–834 (2008)CrossRefGoogle Scholar
  7. 7.
    Freeman, D.: Studying and treating schizophrenia using virtual reality: A new paradigm. Schizophrenia Bulletin 34(4), 605–610 (2008)CrossRefMathSciNetGoogle Scholar
  8. 8.
    Kurtz, M.M., Baker, E., Pearlson, G.D., Astur, R.S.: A virtual reality apartment as a measure of medication management skills in patients with schizophrenia: A pilot study. Schizophrenia Bulletin 33(5), 1162–1170 (2007)CrossRefGoogle Scholar
  9. 9.
    Yellowlees, P., Cook, J.: Education about hallucinations using an internet virtual reality system: A qualitative survey. Academic Psychiatry 30(6), 534–539 (2006)CrossRefGoogle Scholar
  10. 10.
    Kim, K., Kim, J.-J., Kim, J., Park, D.-E., Jang, H.J., Ku, J., Kim, C.-H., Kim, I.Y., Kim, S.I.: Characteristics of social perception assessed in schizophrenia using virtual reality. Cyberpsychology & Behavior 10(2), 215–219 (2006)CrossRefGoogle Scholar
  11. 11.
    Park, K.-M., Ku, J., Choi, S.-H., Jang, H.-J., Park, J.-Y., Kim, S.I., Kim, J.-J.: A virtual reality application in role-plays of social skills training for schizophrenia: A randomized, controlled trial. Psychiatry Research 189(2), 166–172 (2011)CrossRefGoogle Scholar
  12. 12.
    Sorkin, A., Weinshall, D., Modai, I., Peled, A.: Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality. American Journal of Psychiatry 163(3), 512–520 (2006)CrossRefGoogle Scholar
  13. 13.
    Liu, C., Conn, K., Sarkar, N., Stone, W.: Physiology-based affect recognition for computer-assisted intervention of children with Autism Spectrum Disorder. International Journal of Human-Computer Studies 66(9), 662–677 (2008)CrossRefGoogle Scholar
  14. 14.
    Liu, C., Conn, K., Sarkar, N., Stone, W.: Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE Transactions on Robotics 24(4), 883–896 (2008)CrossRefGoogle Scholar
  15. 15.
    Andreasen, N.C.: Scale for the assessment of positive symptoms. University of Iowa, Iowa City (1984)Google Scholar
  16. 16.
    Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Technical manual and affective ratings. Gainesville, FL: The Center for Research in Psychophysiology, University of Florida (1999)Google Scholar
  17. 17.
    Bekele, E., Zheng, Z., Swanson, A., Crittendon, J., Warren, Z., Sarkar, N.: Understanding How Adolescents with Autism Respond to Facial Expressions in Virtual Reality Environments. IEEE Transactions on Visualization and Computer Graphics 19(4), 711–720 (2013)CrossRefGoogle Scholar
  18. 18.
    Lahiri, U., Warren, Z., Sarkar, N.: Design of a Gaze-Sensitive Virtual Social Interactive System for Children With Autism. IEEE Transactions on Neural Systems and Rehabilitation Engineering (99), 1 (2012)Google Scholar
  19. 19.
    Cacioppo, J.T., Tassinary, L.G., Berntson, G.G.: Handbook of psychophysiology. Cambridge Univ. Pr. (2007)Google Scholar
  20. 20.
    Welch, K., Lahiri, U., Liu, C., Weller, R., Sarkar, N., Warren, Z.: An Affect-Sensitive Social Interaction Paradigm Utilizing Virtual Reality Environments for Autism Intervention. Paper presented at the Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction (2009)Google Scholar
  21. 21.
    Bekele, E., Zheng, Z., Swanson, A., Crittendon, J., Warren, Z., Sarkar, N.: Understanding How Adolescents with Autism Respond to Facial Expressions in Virtual Reality Environments. IEEE Transactions on Visualizations and Computer Graphics (2013) (to appear) (special issue)Google Scholar
  22. 22.
    Blair, J.R., Spreen, O.: Predicting premorbid IQ: A revision of the National Adult Reading Test. The Clinical Neuropsychologist 3(2), 129–136 (1989)CrossRefGoogle Scholar
  23. 23.
    Andreasen, N.C., Pressler, M., Nopoulos, P., Miller, D., Ho, B.-C.: Antipsychotic dose equivalents and dose-years: A standardized method for comparing exposure to different drugs. Biological Psychiatry 67(3), 255–262 (2010)CrossRefGoogle Scholar
  24. 24.
    Overall, J.E., Gorham, D.R.: The brief psychiatric rating scale. Psychological Reports 10(3), 799–812 (1962)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Esubalew Bekele
    • 1
  • Dayi Bian
    • 1
  • Zhi Zheng
    • 1
  • Joel Peterman
    • 2
  • Sohee Park
    • 2
  • Nilanjan Sarkar
    • 1
    • 3
  1. 1.Electrical Engineering and Computer Science DepartmentVanderbilt UniversityNashvilleUSA
  2. 2.Psychology DepartmentVanderbilt UniversityNashvilleUSA
  3. 3.Mechanical Engineering DepartmentVanderbilt UniversityNashvilleUSA

Personalised recommendations