Assessing User Experience in Virtual Reality – A Comparison of Different Measurements

  • Carolin WienrichEmail author
  • Nina Döllinger
  • Simon Kock
  • Kristina Schindler
  • Ole Traupe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10918)


As other interactive technologies, the promising virtual reality (VR) applications and their success highly depend on the quality of the user’s experience. The present study gives first insights into relations between general and VR specific aspects of user experience by: (1) Analyzing the evaluation requirements for a large-scale multi-user use case; (2) relating evaluation concepts from the fields of (2D) user experience (UX) and (3D) VR experiences; (3) testing these relations by incorporating measurements from different research fields, and (4) discussing implications for a holistic evaluation framework. During and after experiencing a multi-user adventure on the Immersive Deck of Illusion Walk, participants rated their experience with respect to various components of general UX as well as other components specific to VR experiences. The results revealed positive correlations of presence and social presence with most of the employed post-experience UX measures. The relations between the post- and in-experience measurements showed some inconsistencies. Overall, the experience was positively appraised. The results encourage further investigations into integrating measurements from different lines of research in order to explore the evaluation space of VR experiences.


User experience Virtual reality Evaluation framework 


  1. 1.
    The Future of Virtual Reality isn’t Your Living Room - It’s The Mall. Accessed 22 Dec 2017
  2. 2.
    Zero Latency Madrid: Google Maps (2017). Accessed: 22 Dec 2017
  3. 3.
    The void (2007)Google Scholar
  4. 4.
    Understanding, Scoping and Defining User Experience: A survey approach. Accessed 22 Dec 2017
  5. 5.
    Home—Illusion Walk Berlin the Virtual Reality Immersive Experience Provider. Accessed 22 Dec 2017
  6. 6.
    Podkosova, I., Vasylevska, K., Schoenauer, C., Vonach, E., Fikar, P., Bronederk, E., Kaufmann, H.: ImmersiveDeck: a large-scale wireless VR system for multiple users (2016).
  7. 7.
    ISO 9241-11: Guidelines for specifying and measuring usability (1993)Google Scholar
  8. 8.
    McCarthy, J., Wright, P.: Technology as experience. Interactions 11(5), 42 (2004)CrossRefGoogle Scholar
  9. 9.
    Thüring, M., Mahlke, S.: Usability, aesthetics and emotions in human-technology interaction. Int. J. Psychol. 42(4), 253–264 (2007)CrossRefGoogle Scholar
  10. 10.
    Law, E., Vermeeren, A., Hassenzahl, M., Eds, M.B.: Towards a UX manifesto. In: COST294-MAUSE Affiliated Workshop, Structure, September 2007, vol. 2, pp. 205–206 (2007)Google Scholar
  11. 11.
    International Ergonomics Association. Accessed 22 Dec 2017
  12. 12.
    Stanney, K.M., Mourant, R.R., Kennedy, R.S.: Human factors issues in virtual environments: a review of the literature. Presence Teleoper. Virtual Environ. 7(4), 327–351 (1998)CrossRefGoogle Scholar
  13. 13.
    Tromp, J.G., Steed, A., Wilson, J.R.: Systematic usability evaluation and design issues for collaborative virtual environments. Presence Teleoper. Virtual Environ. 12(3), 241–267 (2003)CrossRefGoogle Scholar
  14. 14.
    Wienrich, C., Noller, F., Thüring, M.: Design principles for VR interaction models: an empirical pilot study. In: Virtuelle und erweiterte Realitäten 14. Workshop der GI-Fachgruppe VR/AR, pp. 162–171 (2017)Google Scholar
  15. 15.
    Slater, M.: Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos. Trans. Roy. Soc. B Biol. Sci. 364(1535), 3549–3557 (2009)CrossRefGoogle Scholar
  16. 16.
    Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic insights. Presence Teleoper. Virtual Environ. 10(3), 266–281 (2001)CrossRefGoogle Scholar
  17. 17.
    Hegemann, S.: Die Entstehung von Kinetosen. In: Plinkert, P.K., Klingmann, C. (eds.) Hören und Gleichgewicht, pp. 185–193. Springer, Vienna (2010). Scholar
  18. 18.
    Kennedy, R.S., Drexler, J., Kennedy, R.C.: Research in visually induced motion sickness. Appl. Ergon. 41(4), 494–503 (2010)CrossRefGoogle Scholar
  19. 19.
    Burdea, G.C., Coiffet, P.: Virtual Reality Technology. Wiley-IEEE Press, New York (2003)Google Scholar
  20. 20.
    Stanney, K.M., Kennedy, R.S., Drexler, J.M.: Cybersickness is not simulator sickness (1997). Scholar
  21. 21.
    Ryan, R.M., Deci, E.L.: Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol. 55(1), 68–78 (2000)CrossRefGoogle Scholar
  22. 22.
    Ijsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F.: Characterising and measuring user experiences in digital games. In: ACE Conference 2007, January 2007Google Scholar
  23. 23.
    Reinhardt, D., Kuge, J., Hurtienne, J.: CHAI: coding heuristics for assessing intuitive interaction. In: Marcus, A., Wang, W. (eds.) DUXU 2018. LNCS, vol. 10918, pp. 528–545 (2018)Google Scholar
  24. 24.
    Baños, R., Botella, C., Alcañiz, M., Liaño, V., Guerrero, B., Rey, B.: Immersion and emotion: their impact on the sense of presence. CyberPsychol. Behav. 7(6), 734–741 (2004)CrossRefGoogle Scholar
  25. 25.
    Riva, G., et al.: Affective interactions using virtual reality: the link between presence and emotions. CyberPsychol. Behav. 10(1), 45–56 (2007)CrossRefGoogle Scholar
  26. 26.
    Ryan, R.M., Rigby, C.S., Przybylski, A.: The motivational pull of video games: a self-determination theory approach. Motiv. Emot. 30(4), 347–363 (2006)CrossRefGoogle Scholar
  27. 27.
    Hertel, G., Konradt, U., Orlikowski, B.: Managing distance by interdependence: goal setting, task interdependence, and team-based rewards in virtual teams. Eur. J. Work Organ. Psychol. 13(1), 1–28 (2004)CrossRefGoogle Scholar
  28. 28.
    Ellis, J.B., Luther, K., Bessiere, K., Kellogg, W.A.: Games for virtual team building. In: Proceedings of the Designing Interactive Systems, DIS 2008, pp. 295–304 (2008)Google Scholar
  29. 29.
    Wienrich, C., Schindler, K., Döllinger, N., Kock, S., Traupe, O.: Social presence and cooperation in a large-scale, multi-user virtual reality – an empirical evaluation of a location-based adventure. In: IEEE VR (2018)Google Scholar
  30. 30.
    Minge, M., Riedel, L.: meCUE – Ein modularer Fragebogen zur Erfassung des Nutzungserlebens. In: Mensch und Computer, Munich, pp. 89–98 (2013)Google Scholar
  31. 31.
    IJsselsteijn, W.A., de Kort, Y.A.W., Poels, K.: Game Experience Questionnaire. FUGA the fun of gaming: Measuring the human experience of media enjoyment GAME (2015)Google Scholar
  32. 32.
    Krohne, W.H., Egloff, B., Kohlmann, C.-W., Tausch, A.: Untersuchungen mit einer deutschen Version der “Positive and Negative Affect Schedule’’ (PANAS). Diagnostica 42(2), 139–156 (1996)Google Scholar
  33. 33.
    Keshavarz, B., Hecht, H.: Validating an efficient method to quantify motion sickness. Hum. Factors 53(4), 415–426 (2011)CrossRefGoogle Scholar
  34. 34.
    Hudson, M., Cairns, P.: Measuring social presence in team-based digital games. In: Interacting with Presence HCI Sense Presence Computer Environments, p. 83 (2014)Google Scholar
  35. 35.
    Dörner, R., Broll, W., Grimm, P., Jung, B. (eds.): Virtual und Augmented Reality (VR/AR). Springer, Heidelberg (2013). Scholar
  36. 36.
    Skarratt, P.A., Cole, G.G., Kingstone, A.: Social inhibition of return. Acta Psychol. (Amst) 134(1), 48–54 (2010)CrossRefGoogle Scholar
  37. 37.
    Wienrich, C., Gross, R., Kretschmer, F., Müller-Plath, G.: Developing and proving a framework for reaction time experiments in VR to objectively measure social interaction with virtual agents. In: IEEE VR (2018)Google Scholar
  38. 38.
    Benford, S., Giannachi, G., Koleva, B., Rodden, T.: From interaction to trajectories: designing coherent journeys through user experiences. In: Chi 2009 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, vol. 1–4, pp. 709–718 (2009)Google Scholar
  39. 39.
    Freytag, S., Wienrich, C.: Evaluation of a virtual gaming environment designed to access emotional reactions while playing. In: Virtual Worlds and Games for Serious Applications (VS-Games), pp. 145–148. IEEE (2017)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Carolin Wienrich
    • 1
    Email author
  • Nina Döllinger
    • 2
  • Simon Kock
    • 2
  • Kristina Schindler
    • 2
  • Ole Traupe
    • 2
  1. 1.University of WürzburgWürzburgGermany
  2. 2.Technische Universität BerlinBerlinGermany

Personalised recommendations