Augmented Narrative: Assisting the Reader with Sound

  • Susana Sanchez PerezEmail author
  • Naohito OkudeEmail author
  • Kai KunzeEmail author
Part of the Cognitive Science and Technology book series (CSAT)


Augmented Narrative is literature that assists readers by augmenting the narrative text with sound effects. The assistance comes when the reader is not being engaged into the storyline, using smart glasses as a biofeedback that measure levels of engagement through the temperature of the nose. The concept emerges from literary theory, translated into cognitive and computational terms, and put into a framework of cognitive ecology to design literature that provides with an embodied experience of the storyworld.


  1. 1.
    Angelotti M, Behnke RR, Carlile LW (1975) Heart rate: a measure of reading involvement. Res Teach Engl 192–199Google Scholar
  2. 2.
    Ashby J, Rayner K (2012) Reading in alphabetic writing systems: evidence from cognitive neuroscience. Neurosci Educ Good Bad Ugly 61Google Scholar
  3. 3.
    Benton M (2005) Reader-response criticism. Understanding children’s literature: key essays from the second edition of the international companion encyclopedia of children’s literature, 86Google Scholar
  4. 4.
    Berka C, Levendowski DJ, Lumicao MN, Yau A, Davis G, Zivkovic VT, Olmstead RE, Tremoulet PD, Craven PL (2007) Eeg correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviat Space Environ Med 78(Supplement 1):B231–B244Google Scholar
  5. 5.
    Bertelson P, de Gelder B (2004) The psychology of multimodal perception. Crossmodal space and crossmodal attention, pp 141–177Google Scholar
  6. 6.
    Biedert R, Buscher G, Schwarz S, Möller M, Dengel A, Lottermann T (2010) The text 2.0 framework. In: Workshop on eye gaze in intelligent human machine interaction. Citeseer, pp 114–117Google Scholar
  7. 7.
    Cannon WB (1927) The james-lange theory of emotions: A critical examination and an alternative theory. Am J Psychol 106–124Google Scholar
  8. 8.
    Cohen B, Izard C, Facial Simons R (1986) physiological indexes of emotions in mother-infant interactions. In Psychophysiology, vol. 23, Soc Psychophysiol Res, (1010) Vermont Ave Nw Suite 1100. Washington, DC 20005:429–429Google Scholar
  9. 9.
    Corradini A, Antonietti A (2013) Mirror neurons and their function in cognitively understood empathy. Conscious Cogn 22(3):1152–1161CrossRefGoogle Scholar
  10. 10.
    Coulson S, King JW, Kutas M (1998) Expect the unexpected: event-related brain response to morphosyntactic violations. Lang Cogn Process 13(1):21–58CrossRefGoogle Scholar
  11. 11.
    Csikszentmihalyi M, Csikzentmihaly M (1991) Flow: the psychology of optimal experience, vol 41. Harper Perennial, New YorkGoogle Scholar
  12. 12.
    Eason SH, Goldberg LF, Young KM, Geist MC, Cutting LE (2012) Reader-text interactions: How differential text and question types influence cognitive skills needed for reading comprehension. J Educ Psychol 104(3):515CrossRefGoogle Scholar
  13. 13.
    Ekman P, Levenson RW, Friesen WV (1983) Autonomic nervous system activity distinguishes among emotions. Sci 221(4616):1208–1210CrossRefGoogle Scholar
  14. 14.
    Engle RW, Tuholski SW, Laughlin JE, Conway AR (1999) Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. J Exp Psychol Gen 128(3):309CrossRefGoogle Scholar
  15. 15.
    Ericsson KA, Kintsch W (1995) Long-term working memory. Psychol Rev 102(2):211CrossRefGoogle Scholar
  16. 16.
    Fludernik M (2002) Towards a ‘natural’ narratology. RoutledgeGoogle Scholar
  17. 17.
    Folkerth W (2014) The sound of Shakespeare. RoutledgeGoogle Scholar
  18. 18.
    Gibson JJ. The senses considered as perceptual systemsGoogle Scholar
  19. 19.
    Gibson JJ (2014) The ecological approach to visual perception: classic edition. Psychology PressGoogle Scholar
  20. 20.
    Gladwell M (2007) Blink: the power of thinking without thinking. Back Bay BooksGoogle Scholar
  21. 21.
    Goldman AI (2005) 2 imitation, mind reading, and simulation. Perspectives on imitation: imitation, human development, and culture, 79Google Scholar
  22. 22.
    Gouvea AC. How to examine the p600 using language theory: what are the syntactic processes reflected in this component?Google Scholar
  23. 23.
    Graesser AC, McNamara DS (2011) Computational analyses of multilevel discourse comprehension. Top Cogn Sci 3(2):371–398CrossRefGoogle Scholar
  24. 24.
    Havelock EA (1980) The oral composition of greek drama. Quaderni Urbinati di Cultura Classica, 61–113Google Scholar
  25. 25.
    Healy SD (1984) Boredom, self, and culture. Fairleigh Dickinson University PressGoogle Scholar
  26. 26.
    Herman D (2000) Narratology as a cognitive science. Image Narrative 1:1MathSciNetGoogle Scholar
  27. 27.
    Hollan J, Hutchins E, Kirsh D (2000) Distributed cognition: toward a new foundation for human-computer interaction research. ACM Trans Comput Hum Interact (TOCHI) 7(2):174–196Google Scholar
  28. 28.
    Houix O, Lemaitre G, Misdariis N, Susini P, Urdapilleta I (2012) A lexical analysis of environmental sound categories. J Exp Psychol Appl 18(1):52CrossRefGoogle Scholar
  29. 29.
    Hutchins E (2010) Cognitive ecology. Topics. Cogn Sci 2(4):705–715CrossRefGoogle Scholar
  30. 30.
    Ioannou S, Ebisch S, Aureli T, Bafunno D, Ioannides HA, Cardone D, Manini B, Romani GL, Gallese V, Merla A. The autonomic signature of guilt in children: a thermal infrared imaging studyGoogle Scholar
  31. 31.
    Ioannou S, Gallese V, Merla A (2014) Thermal infrared imaging in psychophysiology: potentialities and limits. Psychophysiology 51(10):951–963CrossRefGoogle Scholar
  32. 32.
    Jalilehvand M (2012) The effects of text length and picture on reading comprehension of iranian efl students. Asian Soc Sci 8(3):329CrossRefGoogle Scholar
  33. 33.
    Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A (2008) Measuring and defining the experience of immersion in games. Int J Hum Comput Stud 66(9):641–661CrossRefGoogle Scholar
  34. 34.
    Kadir D (2006) Comparative literature in a world become tlon. Comp Crit Stud 3(1):125–138CrossRefGoogle Scholar
  35. 35.
    Keller J, Bless H, Blomann F, Kleinböhl D (2011) Physiological aspects of flow experiences: skills-demand-compatibility effects on heart rate variability and salivary cortisol. J Exp Soc Psychol 47(4):849–852CrossRefGoogle Scholar
  36. 36.
    Kiesling LL (2012) Mirror neuron research and adam smith’s concept of sympathy: three points of correspondence. Rev Austrian Econ 25(4):299–313CrossRefGoogle Scholar
  37. 37.
    Kintsch W (1988) The role of knowledge in discourse comprehension: a construction-integration model. Psychol Rev 95(2):163CrossRefGoogle Scholar
  38. 38.
    Kintsch W, Patel VL, Ericsson KA (1999) The role of long-term working memory in text comprehension. Psychologia 42(4):186–198Google Scholar
  39. 39.
    Kunze K, Sanchez S, Dingler T, Augereau O, Kise K, Inami M, Tsutomu T (2015) The augmented narrative: toward estimating reader engagement. In: Proceedings of the 6th augmented human international conference. ACM, pp 163–164Google Scholar
  40. 40.
    Lebrecht J, Kaye D (1999) The art and technique of design, sound and music for the theatreGoogle Scholar
  41. 41.
    Lemaitre G, Heller LM (2013) Evidence for a basic level in a taxonomy of everyday action sounds. Exp Brain Res 226(2):253–264CrossRefGoogle Scholar
  42. 42.
    Lewis D (1978) Truth in fiction. Am Philos Q 15(1):37–46Google Scholar
  43. 43.
    MacDonald RA, Hargreaves DJ, Miell D, Davidson JW, North AC (2002) Musical identities, vol 13. Oxford University Press, OxfordGoogle Scholar
  44. 44.
    MacLuhan M (2005) The effect of the printed book on language in the 16th century. Gingko PressGoogle Scholar
  45. 45.
    Mar RA, Oatley K (2008) The function of fiction is the abstraction and simulation of social experience. Perspect Psychol Sci 3(3):173–192CrossRefGoogle Scholar
  46. 46.
    McAdams SE, Bigand EE (1993) Thinking in sound: the cognitive psychology of human audition. In: Based on the fourth workshop in the tutorial workshop series organized by the hearing group of the French acoustical society. Clarendon Press/Oxford University PressGoogle Scholar
  47. 47.
    McCraty R, Atkinson M, Tomasino D, Bradley RT (2009) The coherent heart: heart-brain interactions, psychophysiological coherence, and the emergence of system-wide order. Integr Rev 5(2):10–115Google Scholar
  48. 48.
    Mcluhan HM (2010) Understanding me: lectures and interviews. McClelland & StewartGoogle Scholar
  49. 49.
    Merleau-Ponty M, Smith C (1996) Phenomenology of perception. Motilal Banarsidass PublisheGoogle Scholar
  50. 50.
    Michaels CF, Carello C (1981) Direct perception. Prentice-Hall Englewood Cliffs, NJGoogle Scholar
  51. 51.
    Moreno R, Mayer RE (2002) Verbal redundancy in multimedia learning: when reading helps listening. J Educ Psychol 94(1):156CrossRefGoogle Scholar
  52. 52.
    Nakanishi R, Imai-Matsumura K (2008) Facial skin temperature decreases in infants with joyful expression. Infant Behav Dev 31(1):137–144CrossRefGoogle Scholar
  53. 53.
    Or CK, Duffy VG (2007) Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occup Ergon 7(2):83Google Scholar
  54. 54.
    Paivio A (1978) A dual coding approach to perception and cognition. Modes of perceiving and processing information, pp 39–51Google Scholar
  55. 55.
    Pauletto S (2014) Film and theatre-based approaches for sonic interaction design. Digit Creativity 25(1):15–26CrossRefGoogle Scholar
  56. 56.
    Penney CG (1989) Modality effects and the structure of short-term verbal memory. Mem Cogn 17(4):398–422CrossRefGoogle Scholar
  57. 57.
    Pollard AW (2010) Shakespeare’s fight with the pirates and the problems of the transmission of his text. Cambridge University PressGoogle Scholar
  58. 58.
    Purves AC, Rippere V. Elements of writing about a literary work—a study of response to literatureGoogle Scholar
  59. 59.
    Rabinowitz PJ (1997) Before reading: narrative conventions and the politics of interpretation. The theory and interpretation of narrative series, ERICGoogle Scholar
  60. 60.
    Rose PA (2012) The classical trivium: the place of thomas nashe in the learning of his time (by marshall mcluhan). Can J Commun 37:4CrossRefGoogle Scholar
  61. 61.
    Ryan M-L (1991) Possible worlds, artificial intelligence, and narrative theory. Indiana University PressGoogle Scholar
  62. 62.
    Sadoski M (1983) An exploratory study of the relationships between reported imagery and the comprehension and recall of a story. Read Res Q 110–123Google Scholar
  63. 63.
    Schütz A (1951) Making music together: a study in social relationship. Soc Res 76–97Google Scholar
  64. 64.
    Scolari CA (2012) Media ecology: exploring the metaphor to expand the theory. Commun Theory 22(2):204–225CrossRefGoogle Scholar
  65. 65.
    Shokoff J (2001) What is an audiobook? J Popular Cult 34(4):171CrossRefGoogle Scholar
  66. 66.
    Sowa JF. Conceptual structures: information processing in mind and machineGoogle Scholar
  67. 67.
    Spacks PM (1995) Boredom: the literary history of a state of mind. University of Chicago PressGoogle Scholar
  68. 68.
    Thierry G, Martin CD, Gonzalez-Diaz V, Rezaie R, Roberts N, Davis PM (2008) Event-related potential characterisation of the shakespearean functional shift in narrative sentence structure. Neuroimage 40(2):923–931CrossRefGoogle Scholar
  69. 69.
    Tribble E, Sutton J et al (2011) Cognitive ecology as a framework for shakespearean studies. Shakespeare Stud 39:94–103Google Scholar
  70. 70.
    Verstraete P. The frequency of imagination: auditory distress and aurality in contemporary music theatreGoogle Scholar
  71. 71.
    Walter O (1982) Orality and literacy. TJPress, London, The technologizing of the wordGoogle Scholar
  72. 72.
    Weaver P (2012) Radio drama: a “visual sound” analysis of John, George and drew baby. PhD thesis, University of Central Florida Orlando, FloridaGoogle Scholar
  73. 73.
    Zajonc RB (1985) Emotion and facial efference: a theory reclaimed. Science 228(4695):15–21CrossRefGoogle Scholar
  74. 74.
    Zhao H (2012) Emotion-driven interactive storytelling. PhD thesis, Bournemouth UniversityGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.KMD, Keio UniversityTokyoJapan

Personalised recommendations