Advertisement

Evaluating content generators

  • Noor ShakerEmail author
  • Gillian Smith
  • Georgios N. Yannakakis
Chapter
Part of the Computational Synthesis and Creative Systems book series (CSACS)

Abstract

Evaluating your content generator is a very important task, but difficult to do well. Creating a game content generator in general is much easier than creating a good game content generator—but what is a “good” content generator? That depends very much on what you are trying to create and why. This chapter discusses the importance and the challenges of evaluating content generators, and more generally understanding a generator’s strengths and weaknesses and suitability for your goals. In particular, we discuss two different approaches to evaluating content generators: visualizing the expressive range of generators, and using questionnaires to understand the impact of your generator on the player. These methods could broadly be called top-down and bottom-up methods for evaluating generators.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Boden, M.A.: What is creativity? In: M.A. Boden (ed.) Dimensions of Creativity, pp. 75–117. MIT Press (1994)Google Scholar
  2. 2.
    Camilleri, E., Yannakakis, G.N., Dingli, A.: Platformer level design for player believability. In: Proceedings of the IEEE Conference on Computational Intelligence and Games (2016)Google Scholar
  3. 3.
    Chernova, S., Orkin, J., Breazeal, C.: Crowdsourcing HRI through online multiplayer games. In: AAAI Fall Symposium: Dialog with Robots, pp. 14–19 (2010)Google Scholar
  4. 4.
    Colton, S.: Creativity versus the perception of creativity in computational systems. In: Proceedings of the AAAI Spring Symposium on Creative Intelligent Systems, pp. 14–20 (2008)Google Scholar
  5. 5.
    Drachen, A., Thurau, C., Togelius, J., Yannakakis, G.N., Bauckhage, C.: Game data mining. In: M. Seif El-Nasr, A. Drachen, A. Canossa (eds.) Game Analytics, pp. 205–253. Springer (2013)Google Scholar
  6. 6.
    Holmg°ard, C., Yannakakis, G.N., Mart´ınez, H.P., Karstoft, K.I.: To rank or to classify? Annotating stress for reliable PTSD profiling. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 719–725 (2015)Google Scholar
  7. 7.
    IJsselsteijn,W., De Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F.: Characterising and measuring user experiences in digital games. In: Proceedings of the ACE 2007Workshop on Methods for Evaluating Games (2007)Google Scholar
  8. 8.
    8. Li, B., Lee-Urban, S., Appling, D.S., Riedl, M.O.: Crowdsourcing narrative intelligence. Advances in Cognitive Systems 2, 25–42 (2012)Google Scholar
  9. 9.
    Liapis, A., Yannakakis, G.N., Togelius, J.: Towards a generic method of evaluating game levels. In: Proceedings of the Artificial Intelligence for Interactive Digital Entertainment Conference, pp. 30–36 (2013)Google Scholar
  10. 10.
    Mart´ınez, H.P., Yannakakis, G.N., Hallam, J.: Don’t classify ratings of affect; rank them! IEEE Transactions on Affective Computing 5(3), 314–326 (2014)Google Scholar
  11. 11.
    Metallinou, A., Narayanan, S.: Annotation and processing of continuous emotional attributes: Challenges and opportunities. In: Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (2013)Google Scholar
  12. 12.
    Nelson, M.J.: Game metrics without players: Strategies for understanding game artifacts. In: Proceedings of the First AIIDE Workshop on AI in the Game-Design Process, pp. 14–18 (2011)Google Scholar
  13. 13.
    Shaker, N., Nicolau, M., Yannakakis, G.N., Togelius, J., ONeill, M.: Evolving levels for Super Mario Bros. using grammatical evolution. In: Proceedings of the IEEE Conference on Computational Intelligence and Games, pp. 304–311 (2012)Google Scholar
  14. 14.
    14. Shaker, N., Yannakakis, G., Togelius, J.: Crowd-sourcing the aesthetics of platform games. IEEE Transactions on Computational Intelligence and AI in Games 5(3), 276–290 (2013)Google Scholar
  15. 15.
    Smith, G., Whitehead, J.: Analyzing the expressive range of a level generator. In: Proceedings of the First Workshop on Procedural Content Generation in Games (2010)Google Scholar
  16. 16.
    16. Smith, G., Whitehead, J., Mateas, M., Treanor, M., March, J., Cha, M.: Launchpad: A rhythmbased level generator for 2-d platformers. IEEE Transactions on Computational Intelligence and AI in Games 3(1), 1–16 (2011)Google Scholar
  17. 17.
    Yannakakis, G.N.: Preference learning for affective modeling. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction (2009)Google Scholar
  18. 18.
    Yannakakis, G.N., Hallam, J.: Ranking vs. preference: A comparative study of self-reporting. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 437–446 (2011)Google Scholar
  19. 19.
    Yannakakis, G.N., Liapis, A., Alexopoulos, C.: Mixed-initiative co-creativity. In: Proceedings of the 9th Conference on the Foundations of Digital Games (2014)Google Scholar
  20. 20.
    Yannakakis, G.N., Mart´ınez, H.P.: Grounding truth via ordinal annotation. In: Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp. 574–580 (2015)Google Scholar
  21. 21.
    Yannakakis, G.N., Mart´ınez, H.P.: Ratings are overrated! Frontiers in ICT 2, 13 (2015)Google Scholar
  22. 22.
    Yannakakis, G.N., Mart´ınez, H.P., Garbarino, M.: Psychophysiology in games. In: K. Karpouzis, G.N. Yannakakis (eds.) Emotion in Games: Theory and Praxis. Springer (2016)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Noor Shaker
    • 1
    Email author
  • Gillian Smith
    • 2
  • Georgios N. Yannakakis
    • 3
  1. 1.Department of Architecture, Design and Media TechnologyAalborg University Copenhagen (AAU CPH)CopenhagenDenmark
  2. 2.College of Arts, Media and DesignNortheastern UniversityBostonUSA
  3. 3.Institute of Digital GamesUniversity of MaltaMsidaMalta

Personalised recommendations