Designing an Interdisciplinary User Evaluation for the Riu Computational Narrative System

  • Jichen Zhu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7648)


Evaluation is one of the major open problems in Interactive Digital Storytelling (IDS) research. As narrative systems grow in their capacities, the community needs a set of well-designed evaluation methods and criteria that can bring insights on the systems as well as the stories they provide. In this short paper, we examine existing evaluation methods in the area of generative narrative system, and identify several important properties of stories and reading that have so far been overlooked in empirical studies. We present our preliminary work of developing a more interdisciplinary evaluation approach that takes into account both the system and cultural aspects of the computational narrative system Riu.


User Study System Author Character Believability Major Open Problem Narrative Experience 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bringsjord, S., Ferrucci, D.A.: Artificial Intelligence and Literary Creativity: Inside the Mind of BRUTUS, a Storytelling Machine. Lawrence Erlbaum, Hillsdale (2000)Google Scholar
  2. 2.
    Gervás, P.: Computational approaches to storytelling and creativity. AI Magazine 30(3), 49–62 (2009)Google Scholar
  3. 3.
    Höök, K., Sengers, P., Andersson, G.: Sense and sensibility: evaluation and interactive art. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 241–248 (2003)Google Scholar
  4. 4.
    Jordanous, A.: Evaluating evaluation: Assessing progress in computational creativity research. In: Proceedings of the Second International Conference on Computational Creativity (ICCC 2011), pp. 102–107 (2011)Google Scholar
  5. 5.
    Lebowitz, M.: Story-telling as planning and learning. Poetics 14(6), 483–502 (1985)CrossRefGoogle Scholar
  6. 6.
    Mateas, M.: Expressive ai: A hybrid art and science practice. Leonardo 34(2), 147–153 (2001)CrossRefGoogle Scholar
  7. 7.
    Meehan, J.: Tale-spin. In: Inside Computer Understanding: Five Programs Plus Miniatures. Lawrence Erlbaum Associates, New Haven (1981)Google Scholar
  8. 8.
    Mehta, M., Dow, S., Mateas, M., MacIntyre, B.: Evaluating a conversation-centered interactive drama. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1–8 (2007)Google Scholar
  9. 9.
    Miall, D.S., Kuiken, D.: Foregrounding, defamiliarization, and affect response to literary stories. Poetics 22, 389–407 (1994)CrossRefGoogle Scholar
  10. 10.
    Ontañón, S., Zhu, J.: On the role of domain knowledge in analogy-based story generation. In: Proceedings of the Twenty-Second International Joint Conferences on Artificial Intelligence (IJCAI 2011), pp. 1717–1722 (2011)Google Scholar
  11. 11.
    Ontañón, S., Zhu, J.: Story and text generation through computational analogy in the riu system. In: Proceedings of AI and Interactive Digital Entertainment Conference (AIIDE 2010), pp. 51–56. AAAI Press (2010)Google Scholar
  12. 12.
    Penny, S.: Experience and abstraction: the arts and the logic of machines. In: Proceedings of PerthDAC 2007: 7th Digital Arts and Culture Conference (2007)Google Scholar
  13. 13.
    Pérez y Pérez, R., Sharples, M.: Mexica: A computer model of a cognitive account of creative writing. Journal of Experimental & Theoretical Artificial Intelligence 13(2), 119–139 (2001)zbMATHGoogle Scholar
  14. 14.
    Riedl, M.: Narrative Generation: Balancing Plot and Character. Ph.D. thesis, North Carolina State University (2004)Google Scholar
  15. 15.
    Schoenau-Fog, H.: Hooked! – Evaluating Engagement as Continuation Desire in Interactive Narratives. In: Si, M., Thue, D., André, E., Lester, J.C., Tanenbaum, J., Zammitto, V. (eds.) ICIDS 2011. LNCS, vol. 7069, pp. 219–230. Springer, Heidelberg (2011)Google Scholar
  16. 16.
    Sengers, P.: Anti-Boxology: Agent Design in Cultural Context. Ph.D. thesis, Carnegie Mellon University (1998)Google Scholar
  17. 17.
    Thue, D., Bulitko, V., Spetch, M., Romanuik, T.: A computational model of perceived agency in video games. In: Proceedings of the Seventh Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 91–96 (2011)Google Scholar
  18. 18.
    Zhu, J.: Towards a new evaluation approach in computational narrative systems. In: Proceedings of the Third International Conference on Computational Creativity (ICCC 2012), pp. 150–154 (2012)Google Scholar
  19. 19.
    Zhu, J., Harrell, D.F.: Navigating the Two Cultures: a Critical Approach to AI-based Literary Practice, pp. 222–246. World Scientific, Singapore (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Jichen Zhu
    • 1
  1. 1.Drexel UniversityPhiladelphiaUSA

Personalised recommendations