Assessing Learning from, with, and in Games Revisited: A Heuristic for Emerging Methods and Commercial Off-the-Shelf Games

  • P. G. SchraderEmail author
  • Michael P. McCreery
  • Mark C. Carroll
  • Danielle L. Head
  • Jeffrey R. Laferriere
Part of the Advances in Game-Based Learning book series (AGBL)


Technology systems and tools have evolved; current technologies are connected, pervasive, and smart. Digital systems like games have become more dynamic, emergent, and complex. In response to these advancements, considerable effort has been applied to understand how people learn from, with, and within game-based environments. Scientists continue to expand the perspectives and views on humans and their interactions with systems like video games. Further, the methods and tools available to frame research, capture and extract data from these environments, and contextualize the inferences from the patterns discovered among the data have also evolved. This chapter coalesces experience from research with video games informed by three key perspectives: (1) games are complex systems; (2) human-computer interaction is a viable framework to describe learning with and in these systems; and (3) process-oriented data extracted from games can be informed from an analytics perspective. Specifically, six principles are presented that may help researchers engage in studies that involve the process of learning. These emerged from several years of research and include several key questions associated with special considerations, challenges, or pitfalls when deciding on a system, research questions, framework, etc. Subsequently, the chapter offers a discussion of existing/ongoing research involving Bully, The Deed, and League of Legends, including (a) detailed descriptions of the games, (b) discussions of each relevant class of affordances (i.e., player, researcher, and developer), and (c) practical research examples, including the purpose, method, and strategies to analyze data. Finally, the ways the heuristic and its principles guided each of the research examples are discussed as a means to contextualize broader implications for learning, assessment, research, and design.


Game research heuristic Game-based learning Emergent data Complex systems 


  1. Alexander, P. A. (1992). Domain knowledge: Evolving themes and emerging concerns. Educational Psychologist, 27(1), 33–51.CrossRefGoogle Scholar
  2. Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10–14.CrossRefGoogle Scholar
  3. Alexander, P. A., & Dochy, F. J. R. C. (1995). Conceptions of knowledge and beliefs: A comparison across varying cultural and educational communities. American Educational Research Journal, 32(2), 413–442.CrossRefGoogle Scholar
  4. Baek, Y. (2017). Game-based learning: Theory, strategies and performance outcomes. Hauppauge, NY: Nova Science Publishers.Google Scholar
  5. Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: Freeman.Google Scholar
  6. Barab, S. A., Gresalfi, M., & Ingram-Goble, A. (2010). Transformational play: Using games to position person, continent, and context. Educational Researcher, 39(7), 525–536. Scholar
  7. Bediou, B., Adams, D. M., Mayer, R. E., Tipton, E., Green, C. S., & Bavelier, D. (2018). Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin, 144(1), 77–110.CrossRefGoogle Scholar
  8. Bernacki, M. L., & Walkington, C. (2018). The role of situational interest in personalized learning. Journal of Educational Psychology, 110(6), 864–881. Scholar
  9. Carroll, J. M., & Campbell, R. L. (1999). Artifacts as psychological theories: The case of human-computer interaction. Behaviour & Information Technology, 8(4), 247–256.CrossRefGoogle Scholar
  10. Clark, D., Tanner-Smith, E., & Killingsworth, S. (2014). Digital games, design and learning: A systematic review and meta-analysis (executive summary). Menlo Park, CA: SRI International.Google Scholar
  11. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2012), 661–686.CrossRefGoogle Scholar
  12. Design Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.CrossRefGoogle Scholar
  13. DiCerbo, K. E., & Behrens, J. T. (2012). Implications of the digital ocean on current and future assessment. In R. Lissitz & H. Jiao (Eds.), Computers and their impact on state assessment: Recent history and predictions for the future (pp. 273–306). Charlotte, NC: Information Age Publishing.Google Scholar
  14. Ferdig, R. E. (2008). Handbook of research on effective electronic gaming in education (Vol. I–III). Hershey, PA: Information Science Reference.Google Scholar
  15. Gaver, W. H. (1991). Technology affordances. In Proceedings of CHI’91 (pp. 79–84). New York, NY: ACM.Google Scholar
  16. Ge, X., & Ifenthaler, D. (2017). Designing engaging educational games and assessing engagement in game-based learning. In R. Zheng & M. K. Gardner (Eds.), Handbook of research on serious games for educational applications (pp. 255–272). Hershey, PA: IGI Global.Google Scholar
  17. Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  18. Gibson, J. J. (1986). The ecological approach to visual perception. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  19. Girard, C., Ecalle, J., & Magnan, A. (2013). Serious games as new educational tools: How effective are they? a meta-analysis of recent studies. Journal of Computer Assisted Learning, 29(3), 207–219.CrossRefGoogle Scholar
  20. Grabthegames Studios. (2015). The deed. Grabthegames Studios.Google Scholar
  21. Greeno, J. G. (1994). Gibson’s affordances. The Psychological Review, 101(2), 336–342.CrossRefGoogle Scholar
  22. Hilpert, J. C., & Marchand, G. C. (2018). Complex systems research in educational psychology: Aligning theory and method. Educational Psychologist, 53(3), 185–202. Scholar
  23. Ifenthaler, D., Eseryel, D., & Ge, X. (2012). Assessment in game-based learning: Foundations, innovations, and perspectives. New York, NY: Springer.CrossRefGoogle Scholar
  24. Jonassen, D. H., Campbell, J. P., & Davidson, M. E. (1994). Learning with media: Restructuring the debate. Educational Technology Research and Development, 42(2), 31–39.CrossRefGoogle Scholar
  25. Ke, F. (2009). A qualitative meta-analysis of computer games as learning tools. In R. Ferdig (Ed.), Handbook of research on effective electronic gaming in education (Vol. I, pp. 1–32). Hershey, PA: Information Science Reference.Google Scholar
  26. King, F. B., Young, M. F., Drivere-Richmond, K., & Schrader, P. G. (2001). Defining distance learning and distance education. AACE Journal, 9(1), 1–14.Google Scholar
  27. Krach, S. K., & McCreery, M. P. (2015). Technology and positive behavioral interventions and support: Evaluation, selection, and implementation of computer-based socioemotional training. In S. Y. Tettegah & D. Espelage (Eds.), Emotions and technology, communication of feelings for, with and through digital media: Emotions, technology, and behavior. Waltham, MA: Elsevier.Google Scholar
  28. Laughlin, D., & Marchuk, N. D. (2005). A guide to educational computer games for NASA. Washington, DC: National Aeronautics and Space Administration’s Office of the Chief Education Officer, Technology and Products Office.Google Scholar
  29. Lejacq, Y. (2013, October 24). How fast is fast? Some pro gamers make 10 moves per second. ABC News. Retrieved from
  30. Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.). (2015). Serious games analytics. Methodologies for performance measurement, assessment, and improvement. New York, NY: Springer.Google Scholar
  31. Mace, W. M. (1977). James J. Gibson’s strategy for perceiving: Ask not what’s inside your head, but what your head’s inside of. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing (pp. 43–65). Hillsdale, NJ: LEA.Google Scholar
  32. Marchand, G. C., & Hilpert, J. C. (2017). Complex dynamic systems approaches to educational research. The Journal of Experimental Education, 85(2), 173–174. Scholar
  33. Marchand, G. C., & Hilpert, J. C. (2018). Design considerations for education scholars interested in complex systems research. Complicity, 15(1), 1–14.CrossRefGoogle Scholar
  34. Mayer, R. E. (2015). On the need for research evidence to guide the design of computer games for learning. Educational Psychologist, 50(4), 349–353. Scholar
  35. McCreery, M. P., Krach, S. K., & Nolen, A. L. (2014). The protagonist and their avatar: Learner characteristics in a culture of simulation. The International Journal of Gaming and Computer-Mediated Simulations, 6(2), 30–37.CrossRefGoogle Scholar
  36. McCreery, M. P., Krach, S. K., Schrader, P. G., & Boone, R. (2012). Defining the virtual self: Personality, behavior and the psychology of embodiment. Computers in Human Behavior, 28(3), 976–983.CrossRefGoogle Scholar
  37. McCreery, M. P., Schrader, P. G., Krach, S. K., & Boone, R. (2013). A sense of self: The role of presence in virtual environments. Computers in Human Behavior, 29(4), 1635–1640.CrossRefGoogle Scholar
  38. McCreery, M. P., Vallett, D., & Clark, C. (2015). Social interaction in a virtual environment: Examining sociospatial interactivity and social presence using behavioral analytics. Computers in Human Behavior, 51, 203–206.CrossRefGoogle Scholar
  39. McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York, NY: The Penguin Press.Google Scholar
  40. Mirza-Babaei, P., Long, S., Foley, E., & McAllister, G. (2011). Understanding the contribution of biometrics to games user research. In Proceedings of DiGRA 2011 Conference: Think Design Play (Vol. 6, pp. 1–13). Retrieved from
  41. Mirza-Babaei, P., Nacke, L. E., Gregory, J., Collins, N., & Fitzpatrick, G. (2013). How does it play better? Exploring user testing and biometric storyboards in games user research. In Conference on Human Factors in Computing Systems - Proceedings (pp. 1499–1508).
  42. O’Brien, D. A., Lawless, K. A., & Schrader, P. G. (2010). A taxonomy of educational games: Genres and applications. In Y. K. Baek (Ed.), Gaming for classroom-based learning: Digital role playing as a motivator of study (pp. 1–23). Hershey, PA: Information Science Reference.Google Scholar
  43. Qian, M., & Clark, K. R. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50–58.CrossRefGoogle Scholar
  44. Reiser, R. A. (2001). The history of instructional design and technology: Part 1: A history of instructional media. Educational Technology Research and Development, 49(1), 53–64. Scholar
  45. Riot Games. (2009). The league of legends. Los Angeles, CA: Riot Games.Google Scholar
  46. Rockstar Games. (2006). Bully. New York, NY: Rockstar Games.Google Scholar
  47. Salomon, G., Perkins, D. N., & Globerson, T. (1991). Partners in cognition: Extending human intelligence with intelligent technologies. Educational Researcher, 20, 2–9.CrossRefGoogle Scholar
  48. Schenk, S., Lech, R. K., & Suchan, B. (2017). Games people play: How video games improve probabilistic learning. Behavioural Brain Research, 335, 208–214. Scholar
  49. Schrader, P. G. (2008). Learning in technology: Reconceptualizing immersive environments. AACE Journal, 16(4), 457–475.Google Scholar
  50. Schrader, P. G., Deniz, H., & Keilty, J. (2016). Breaking SPORE: Building instructional value in science education using a commercial, off-the shelf game. Journal of Learning and Teaching in Digital Age, 1(1), 32–46.Google Scholar
  51. Schrader, P. G., & Lawless, K. A. (2007). Dribble files: Methodologies to evaluate learning and performance in complex environments. Performance Improvement, 46(1), 40–48.CrossRefGoogle Scholar
  52. Schrader, P. G., & McCreery, M. P. (2012). Are all games the same? Examining three frameworks for assessing learning from, with, and in games. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 11–28). New York, NY: Springer.CrossRefGoogle Scholar
  53. Schrader, P. G., McCreery, M. P., & Vallett, D. B. (2017). Performance in situ. In Y. Baek (Ed.), Game-based learning: Theory, strategies and performance outcomes (pp. 421–437). Hauppauge, NY: Nova Science Publishers.Google Scholar
  54. Shaffer, D. W., Squire, K. R., Halverson, R., & Gee, J. P. (2005). Video games and the future of learning. The Phi Delta Kappan, 87(2), 104–111. Scholar
  55. Shalizi, C. R. (2006). Methods and techniques of complex systems science: An overview. In T. S. Deisboeck & J. Y. Kresh (Eds.), Complex systems science in biomedicine (pp. 33–114). New York, NY: Springer.CrossRefGoogle Scholar
  56. Shute, V. J., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters & H. van Oostendorp (Eds.), Instructional techniques to facilitate learning and motivation of serious games. Cham, Switzerland: Springer International Publishing. Scholar
  57. Squire, K. D. (2006). From content to context: Videogames as designed experience. Educational Researcher, 35(8), 19–29. Scholar
  58. Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285.CrossRefGoogle Scholar
  59. Tettegah, S. Y., McCreery, M. P., & Blumberg, F. C. (2015). Toward a framework for learning and digital games research. Educational Psychologist, 50(4), 253–257.CrossRefGoogle Scholar
  60. Vallett, D. (2016). Soft failure in educational and learning games. In R. Lamb & D. McMahon (Eds.), Educational and learning games: New research. New York, NY: Nova Science Publishers.Google Scholar
  61. van Merrienboer, J. J. G., & Kirschner, P. A. (2018). Ten steps to complex learning: A systematic approach to four-component instructional design (3rd ed.). New York, NY: Routledge.Google Scholar
  62. van Merrienboer, J. J. G., & Sluijsmans, D. A. (2009). Toward a synthesis of cognitive load theory, four-component instructional design, and self-directed learning. Educational Psychology Review, 29, 5566.Google Scholar
  63. Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568–2572. Scholar
  64. Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Muse, K., & Wright, M. (2006). Computer gaming and interactive simulations for learning: A meta-analysis. Journal of Educational Computing Research, 34(3), 229–243.CrossRefGoogle Scholar
  65. Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249–265. Scholar
  66. Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., … Yukhymenko, M. (2012). Our princess is in another castle: A review of trends in serious gaming for education. Review of Educational Research, 82(1), 61–89.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • P. G. Schrader
    • 1
    Email author
  • Michael P. McCreery
    • 1
  • Mark C. Carroll
    • 1
  • Danielle L. Head
    • 1
  • Jeffrey R. Laferriere
    • 2
  1. 1.University of Nevada, Las VegasLas VegasUSA
  2. 2.Lebanon Valley CollegeAnnvilleUSA

Personalised recommendations