Skip to main content
SpringerLink
Log in

Serious Games Evaluation: Processes, Models, and Concepts

  • Chapter
  • First Online:
Entertainment Computing and Serious Games

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9970))

Abstract

Serious games are developed with the goal of having a certain impact on players which goes beyond mere entertainment. This purpose-driven design is immanent to serious games and can be stated as the key characteristic that distinguishes serious games from other digital games. Hence, verifying that a serious game has the intended effect on the players needs to be an essential part in the development process. This and the following chapters are therefore dedicated to give a guidance how evaluation procedures can be planned and realized. The main focus is on aspects which are particularly distinctive to the evaluation of serious games, while methods and principles related to the evaluation of digital games in general will not be covered in detail. The structure of this chapter is as follows: After emphasizing the specific importance of evaluation for serious games, we describe a set of challenges which might occur in this context. In order to enable the reader to face these challenges, we present a framework of evaluation-driven design which offers guidance in the evaluation process. Other models which address different challenges are described before three examples of commendably evaluated serious games are discussed. These examples are intended to demonstrate how the presented abstract models can be applied in concrete evaluation procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.firsthand.com/services/pain.html.

  2. 2.

    http://freq1550.waag.org/.

References

  1. Baranowski, T.: Measurement method bias in games for health research. Games Health J. 3(4), 193–194 (2014)

    Article  Google Scholar 

  2. Chen, S., Michael, D.: Proof of learning: assessment in serious games (2005). http://www.gamasutra.com/features/20051019/chen_01.shtml

  3. Chin, J., Dukes, R., Gamson, W.: Assessment in simulation and gaming: a review of the last 40 years. Simul. Gaming 40(4), 553–568 (2009)

    Article  Google Scholar 

  4. Coleman, J.S., Livingston, S.A., Fennessey, G.M., Edwards, K.J., Kidder, S.J.: The hopkins games program: conclusions from seven years of research. Educ. Res. 2(8), 3–7 (1973)

    Article  Google Scholar 

  5. Connolly, T., Stansfield, M., Hainey, T.: Towards the development of a games-based learning evaluation framework. In: Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices. IGI Global, Hershey (2009)

    Google Scholar 

  6. Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M.: A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 59(2), 661–686 (2012)

    Article  Google Scholar 

  7. DeSmet, A., van Ryckeghem, D., Compernolle, S., Baranowski, T., Thompson, D., Crombez, G., Poels, K., van Lippevelde, W., Bastiaensens, S., van Cleemput, K., Vandebosch, H., de Bourdeaudhuij, I.: A meta-analysis of serious digital games for healthy lifestyle promotion. Prev. Med. 69, 95–107 (2014)

    Article  Google Scholar 

  8. Faber, A.W., Patterson, D.R., Bremer, M.: Repeated use of immersive virtual reality therapy to control pain during wound dressing changes in pediatric and adult burn patients. J. Burn Care Res. 34(5), 563–568 (2013)

    Article  Google Scholar 

  9. Girard, C., Ecalle, J., Magnan, A.: Serious games as new educational tools: how effective are they? a meta-analysis of recent studies. J. Comput. Assist. Learn. 29(3), 207–219 (2013)

    Article  Google Scholar 

  10. Göbel, S., Gutjahr, M., Hardy, S.: Evaluation of serious games. In: Bredl, K., Bösche, W. (eds.) Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare, pp. 105–115. IGI Global, Hershey (2013)

    Chapter  Google Scholar 

  11. Hainey, T., Connolly, T.: Evaluating games-based learning. Int. J. Virtual Per. Learn. Environ. (IJVPLE) 1(1), 57–71 (2010)

    Article  Google Scholar 

  12. Hoffman, H.G., Chambers, G.T., Meyer, W.J., Arceneaux, L.L., Russell, W.J., Seibel, E.J., Richards, T.L., Sharar, S.R., Patterson, D.R.: Virtual reality as an adjunctive non-pharmacologic analgesic for acute burn pain during medical procedures. Ann. Behav. Med. 41(2), 183–191 (2011)

    Article  Google Scholar 

  13. Hoffman, H.G., Patterson, D.R., Seibel, E., Soltani, M., Jewett-Leahy, L., Sharar, S.R.: Virtual reality pain control during burn wound debridement in the hydrotank. Clin. J. Pain 24(4), 299–304 (2008)

    Article  Google Scholar 

  14. Hoffman, H.G., Seibel, E.J., Richards, T.L., Furness, T.A., Patterson, D.R., Sharar, S.R.: Virtual reality helmet display quality influences the magnitude of virtual reality analgesia. J. Pain 7(11), 843–850 (2006)

    Article  Google Scholar 

  15. Huizenga, J., Admiraal, W., Akkerman, S., Dam, G.T.: Mobile game-based learning in secondary education: engagement, motivation and learning in a mobile city game. J. Comput. Assist. Learn. 25(4), 332–344 (2009)

    Article  Google Scholar 

  16. Kato, P.M., Cole, S.W., Bradlyn, A.S., Pollock, B.H.: A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics 122(2), e305–e317 (2008)

    Article  Google Scholar 

  17. Kirkley, S.E., Tomblin, S., Kirkley, J.: Instructional design authoring support for the development of serious games and mixed reality training. In: Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) (2005)

    Google Scholar 

  18. Kirkpatrick, D.L., Kirkpatrick, J.D.: Evaluating Training Programs: The Four Levels, 3rd edn. Berrett-Koehler Publishers, San Francisco (2006)

    MATH  Google Scholar 

  19. Loh, C.S., Sheng, Y., Ifenthaler, D.: Serious games analytics: theoretical framework. In: Loh, C.S., Sheng, Y., Ifenthaler, D. (eds.) Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement. Advances in Game-Based Learning, pp. 3–29. Springer International Publishing, Cham (2015)

    Chapter  Google Scholar 

  20. Maani, C.V., Hoffman, H.G., Morrow, M., Maiers, A., Gaylord, K., McGhee, L.L., DeSocio, P.A.: Virtual reality pain control during burn wound debridement of combat-related burn injuries using robot-like arm mounted VR goggles. J. Trauma: Injury, Infection Crit. Care 71(supplement), 125–130 (2011)

    Article  Google Scholar 

  21. Mayer, I.: Towards a comprehensive methodology for the research and evaluation of serious games. Procedia Comput. Sci. 15, 233–247 (2012)

    Article  Google Scholar 

  22. Michael, D.R., Chen, S.: Serious Games: Games That Educate, Train and Inform. Course Technology PTR, Boston (2005)

    Google Scholar 

  23. Mitgutsch, K., Alvarado, N.: Purposeful by design?: a serious game design assessment framework. In: Proceedings of the International Conference on the Foundations of Digital Games, FDG 2012, New York, NY, USA, pp. 121–128. ACM (2012)

    Google Scholar 

  24. Molenda, M.: In search of the elusive ADDIE model. Perform. Improv. 42(5), 34–36 (2003)

    Article  Google Scholar 

  25. O’Neil, H.F., Wainess, R., Baker, E.L.: Classification of learning outcomes: evidence from the computer games literature. Curriculum J. 16(4), 455–474 (2005)

    Article  Google Scholar 

  26. Papastergiou, M.: Exploring the potential of computer and video games for health and physical education: a literature review. Comput. Educ. 53(3), 603–622 (2009)

    Article  Google Scholar 

  27. Sanchez, A., Cannon-Bowers, J.A., Bowers, C.: Establishing a science of game based learning. In: Sanchez, A., Cannon-Bowers, J.A., Bowers, C. (eds.) Serious Game Design and Development: Technologies for Training and Learning, pp. 290–304. IGI Global, Hershey (2010)

    Chapter  Google Scholar 

  28. Schmitt, Y.S., Hoffman, H.G., Blough, D.K., Patterson, D.R., Jensen, M.P., Soltani, M., Carrougher, G.J., Nakamura, D., Sharar, S.R.: A randomized, controlled trial of immersive virtual reality analgesia, during physical therapy for pediatric burn injuries. Burns: J. Int. Soc. Burn Injuries 37(1), 61–68 (2011)

    Article  Google Scholar 

  29. Shapiro, M.A., Peña, J.: Generalizability and validity in digital game research. In: Ritterfeld, U., Cody, M.J., Vorderer, P. (eds.) Serious Games: Mechanisms and Effects, pp. 389–403. Routledge, New York (2009)

    Google Scholar 

  30. Tate, R., Haritatos, J., Cole, S.: HopeLab’s approach to re-mission. Int. J. Learn. Media 1(1), 29–35 (2009)

    Article  Google Scholar 

  31. Wender, R., Hoffman, H.G., Hunner, H.H., Seibel, E.J., Patterson, D.R., Sharar, S.R.: Interactivity influences the magnitude of virtual reality analgesia. J. Cyber Ther. Rehabil. 2(1), 27–33 (2009)

    Google Scholar 

Further Reading

  1. Abt, C.C.: Serious Games. University Press of America, Lanham (1987)

    Google Scholar 

  2. Ainsworth, S.: Evaluation methods for learning environments. In: A Tutorial for the 11th International Conference on Artificial Intelligence Education, Amsterdam. www.psychology.nottingham.ac.uk/staff/Shaaron.Ainsworth/Evaluationtutorial.ppt

  3. All, A., Castellar, E.P.N., Van Looy, J.: Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Comput. Educ. 88, 29–37 (2015)

    Article  Google Scholar 

  4. Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013 (2013). Article no. 1

    Google Scholar 

  5. Bernhaupt, R. (ed.): Evaluating User Experience in Games: Concepts and Methods. Human-Computer Interaction Series. Springer, London and New York (2010)

    Google Scholar 

  6. Bredl, K., Bösche, W. (eds.): Serious Games and Virtual Worlds in Education. Professional Development, and Healthcare. Premier reference source, Information Science Reference, Hershey (2013)

    Google Scholar 

  7. Dondi, C., Moretti, M.: A methodological proposal for learning games selection and quality assessment. Br. J. Educ. Technol. 38(3), 502–512 (2007). http://dx.doi.org/10.1111/j.1467-8535.2007.00713.x

    Article  Google Scholar 

  8. El-Nasr, M.S., Drachen, A., Canossa, A. (eds.): Game Analytics: Maximizing the Value of Player Data. Springer, London and New York (2013)

    Google Scholar 

  9. Göbel, S., Gutjahr, M., Steinmetz, R.: What makes a good serious game - conceptual approach towards a metadata format for the description and evaluation of serious games. In: Gouscous, D., Meimaris, M. (eds.) 5th European Conference on Games Based Learning, pp. 202–210. Academic Conferences Limited, Reading (2011)

    Google Scholar 

  10. Law, E.L.-C., Kickmeier-Rust, M.D., Albert, D., Holzinger, A.: Challenges in the development and evaluation of immersive digital educational games. In: Holzinger, A. (ed.) USAB 2008. LNCS, pp. 19–30. Springer, Heidelberg (2008). doi:10.1007/978-3-540-89350-9_2

    Chapter  Google Scholar 

  11. Loh, C.S., Sheng, Y., Ifenthaler, D.: Serious Games Analytics: Methodologiesfor Performance Measurement, Assessment, and Improvement. Advances in Game-Based Learning. Springer International Publishing, Cham (2015)

    Book  Google Scholar 

  12. Mayer, I., Bekebrede, G., Harteveld, C., Warmelink, H., Zhou, Q., Ruijven, T., Lo, J., Kortmann, R., Wenzler, I.: The research and evaluation of serious games: toward a comprehensive methodology. Br. J. Educ. Technol. 45(3), 502–527 (2014)

    Article  Google Scholar 

  13. Schultz, C.P., Bryant, R.: Game Testing All in One, 2nd edn. Mercury Learning & Information, Dulles (2011)

    Google Scholar 

  14. Serrano, Á., Marchiori, E.J., Blanco, Á.D., Torrente, J., Fernández-Manjón, B.: A framework to improve evaluation in educational games. In: Global Engineering Education Conference (EDUCON), pp. 1–8. IEEE (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Entertainment Computing Group, University of Duisburg-Essen, Forsthausweg 2, 47057, Duisburg, Germany

    Katharina Emmerich

  2. Graph Theory and Complex Network Analysis Group, University of Kaiserslautern, Gottlieb-Daimler-Straße 48, 67663, Kaiserslautern, Germany

    Mareike Bockholt

Authors
  1. Katharina Emmerich
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mareike Bockholt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Katharina Emmerich or Mareike Bockholt .

Editor information

Editors and Affiliations

  1. RheinMain University of Applied Sciences , Wiesbaden, Germany

    Ralf Dörner

  2. Technische Universität Darmstadt , Darmstadt, Germany

    Stefan Göbel

  3. TU Graz , Graz, Austria

    Michael Kickmeier-Rust

  4. University of Duisburg-Essen , Duisburg, Germany

    Maic Masuch

  5. TU Kaiserslautern , Kaiserslautern, Germany

    Katharina Zweig

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this chapter

Cite this chapter

Emmerich, K., Bockholt, M. (2016). Serious Games Evaluation: Processes, Models, and Concepts. In: Dörner, R., Göbel, S., Kickmeier-Rust, M., Masuch, M., Zweig, K. (eds) Entertainment Computing and Serious Games. Lecture Notes in Computer Science(), vol 9970. Springer, Cham. https://doi.org/10.1007/978-3-319-46152-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46152-6_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46151-9

  • Online ISBN: 978-3-319-46152-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation