Modeling and Evaluating Emotion, Contextual Head Movement and Voices for a Social Robot Storyteller

  • Hendrik Striepe
  • Melissa Donnermann
  • Martina Lein
  • Birgit LugrinEmail author
S.I.: Embodied Interactive Robots


Storytelling bears great potential as prospective application for social robots. Through their embodiment and potential for multi-modal behavior, they can enhance current storytelling media, for example books or audio books. We combined theoretical knowledge about designing storytelling robots with an analysis of human storytellers to implement the behavior of a robot storyteller. We tested the following modalities in three perception studies: emotional behavior, contextual head movements, and voice acting through different synthetic voices. Our results suggest, that an emotional robot storyteller is able to transport participants equally well as the traditional audio book, contextual head movement has to be carefully modeled to be recognized, and voice acting can improve narrative presence and can make the robot more interesting. With the present contribution, we provide guidance for modeling and evaluating the behavior of a social robot storyteller, and suggest that the careful integration of emotional display and different voices can enhance the perception of a robot storytelling experience.


Social robots Storytelling Emotion Non-verbal behavior Transportation Robot voice 


Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Standards

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national). All subjects were voluntarily part of the presented studies.


  1. 1.
    Adam C, Cavedon L (2013) A companion robot that can tell stories. In: Proceedings of intelligent virtual agents. SpringerGoogle Scholar
  2. 2.
    Appel M, Gnambs T, Richter T, Green MC (2015) The transportation scale–short form (TS–SF). Media Psychol 18(2):243–266 testCrossRefGoogle Scholar
  3. 3.
    Bartneck C, Kulić D, Croft E, Zoghbi S (2009) Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int J Soc Robot 1(1):71–81CrossRefGoogle Scholar
  4. 4.
    Bates J (1994) The role of emotion in believable agents. Commun ACM 37(7):122–125CrossRefGoogle Scholar
  5. 5.
    Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59CrossRefGoogle Scholar
  6. 6.
    Brandtzæg PB, Følstad A, Heim J (2018) Enjoyment: lessons from Karasek. Springer, Cham, pp 331–341.
  7. 7.
    Breazeal C, Dautenhahn K, Kanda T (2016) Social robotics. In: Springer handbook of robotics. Springer, pp 1935–1972Google Scholar
  8. 8.
    Breazeal CL (2004) Designing sociable robots. MIT Press, CambridgeCrossRefzbMATHGoogle Scholar
  9. 9.
    Busselle R, Bilandzic H (2008) Fictionality and perceived realism in experiencing stories: a model of narrative comprehension and engagement. Commun Theory 18(2):255–280CrossRefGoogle Scholar
  10. 10.
    Busselle R, Bilandzic H (2009) Measuring narrative engagement. Media Psychol 12(4):321–347CrossRefGoogle Scholar
  11. 11.
    Cassell J, McNeill D (1991) Gesture and the poetics of prose. Poet Today 12(3):375–404CrossRefGoogle Scholar
  12. 12.
    Chella A, Barone RE, Pilato G, Sorbello R (2008) An emotional storyteller robot. In: AAAI spring symposium: emotion, personality, and social behavior, pp 17–22Google Scholar
  13. 13.
    Clair AS, Mead R, Mataric MJ (2011) Investigating the effects of visual saliency on deictic gesture production by a humanoid robot. In: 2011 RO-MAN, pp 210–216Google Scholar
  14. 14.
    Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Taylor & Francis IncGoogle Scholar
  15. 15.
    Coles S, Dautenhahn K (2000) A robotic story-teller. In: Proceedings of SIRS2000, 8th symposium on intelligent robotic systems, grundlage Soziale Robotik, Erste Experimente. The University of Reading, England, pp 18–20Google Scholar
  16. 16.
    Costa S, Brunete A, Bae BC, Mavridis N (2016) Emotional storytelling using virtual and robotic agents. arXiv preprint arXiv:1607.05327
  17. 17.
    Cuijpers RH, Knops MAMH (2015) Motions of robots matter! The social effects of idle and meaningful motions. In: Tapus A, André E, Martin JC, Ferland F, Ammi M (eds) Social robotics. Springer, Cham, pp 174–183CrossRefGoogle Scholar
  18. 18.
    Damiano R, Pizzo A (2008) Emotions in drama characters and virtual agents. In: AAAI spring symposium: emotion, personality, and social behavior, pp 30–37Google Scholar
  19. 19.
    Dautenhahn K, Nehaniv C (1998) Artificial life and natural stories. In: International symposium on artificial life and robotics, pp 435–439Google Scholar
  20. 20.
    DiSalvo CF, Gemperle F, Forlizzi J, Kiesler S (2002) All robots are not created equal: the design and perception of humanoid robot heads. In: Proceedings of the 4th conference on designing interactive systems: processes, practices, methods, and techniques. ACM, pp 321–326Google Scholar
  21. 21.
    Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. IshkGoogle Scholar
  22. 22.
    Ekman P, Friesen WV, Hager JC (2002) Facial action coding system: the manual. Research Nexus, Salt Lake City, UTGoogle Scholar
  23. 23.
    Francisco V, Gervás P, González M, León C (2007) Expressive synthesis of read aloud tales. In: Olivier P, Kray C (eds) AISB’07, Artificial and ambient intelligence. pp 179–186Google Scholar
  24. 24.
    Francisco V, Hervás R, Peinado F, Gervás P (2012) Emotales: creating a corpus of folk tales with emotional annotations. Lang Resources Eval 46(3):341–381CrossRefGoogle Scholar
  25. 25.
    Gelin R, d’Alessandro C, Le QA, Deroo O, Doukhan D, Martin JC, Pelachaud C, Rilliard A, Rosset S (2010) Towards a storytelling humanoid robot. In: AAAI fall symposium: dialog with robotsGoogle Scholar
  26. 26.
    Goetz J, Kiesler S, Powers A (2003) Matching robot appearance and behavior to tasks to improve human–robot cooperation. In: Proceedings of the 12th IEEE international workshop on robot and human interactive communication. IEEE Press Piscataway, NJ, pp 55–60Google Scholar
  27. 27.
    Green MC, Brock TC (2000) The role of transportation in the persuasiveness of public narratives. J Personal Soc Psychol 79(5):701CrossRefGoogle Scholar
  28. 28.
    Ham J, Bokhorst R, Cuijpers R, van der Pol D, Cabibihan JJ (2011) Making robots persuasive: the influence of combining persuasive strategies (gazing and gestures) by a storytelling robot on its persuasive power. In: Mutlu B, Bartneck C, Ham J, Evers V, Kanda T (eds) Social robotics. Springer, Berlin, pp 71–83CrossRefGoogle Scholar
  29. 29.
    Ham J, Cuijpers RH, Cabibihan JJ (2015) Combining robotic persuasive strategies: the persuasive power of a storytelling robot that uses gazing and gestures. Int J Soc Robot 7(4):479–487. CrossRefGoogle Scholar
  30. 30.
    Hegel F, Muhl C, Wrede B, Hielscher-Fastabend M, Sagerer G (2009) Understanding social robots. In: 2nd International conferences on advances in computer–human interactions, 2009. ACHI’09. IEEE, pp 169–174Google Scholar
  31. 31.
    Huang CM, Mutlu B (2013) Modeling and evaluating narrative gestures for humanlike robots. In: Robotics: science and systems, pp 57–64Google Scholar
  32. 32.
    Jara-Rivera I, Castro-González Á, Castillo JC, Alonso-Martín F, Malfaz M, Salichs MA (2017) Creating lively behaviors in social robots. In: International conference on social robotics. Springer, pp 283–292Google Scholar
  33. 33.
    Kory Westlund J, Breazeal C (2014) Storytelling with robots: learning companions for preschool children’s language development. In: the 23rd IEEE international symposium on robot and human interactive communication, 2014 RO-MAN. IEEE, pp 643–648Google Scholar
  34. 34.
    Kory Westlund J, Gordon G, Spaulding S, Lee JJ, Plummer L, Martinez M, Das M, Breazeal C (2016) Lessons from teachers on performing HRI studies with young children in schools. In: 11th ACM/IEEE international conference on human robot interaction. IEEE Press, Piscataway, NJ, USA, HRI ’16, pp 383–390Google Scholar
  35. 35.
    Kory Westlund J, Jeong S, Park HW, Ronfard S, Adhikari A, Harris PL, DeSteno D, Breazeal CL (2017) Flat vs. expressive storytelling: young children’s learning and retention of a social robot’s narrative. Front Hum Neurosci 11:295CrossRefGoogle Scholar
  36. 36.
    KPMG (2016) Social robots—2016’s new breed of social robots is ready to enter your world. Accessed 06 Feb 2019
  37. 37.
    Lang A (2000) The limited capacity model of mediated message processing. J Commun 50(1):46–70CrossRefGoogle Scholar
  38. 38.
    Lombardo V, Damiano R (2012) Semantic annotation of narrative media objects. Multimed Tools Appl 59(2):407–439CrossRefGoogle Scholar
  39. 39.
    Lugmayr A, Sutinen E, Suhonen J, Sedano CI, Hlavacs H, Montero CS (2017) Serious storytelling—a first definition and review. Multimed Tools Appl 76(14):15707–15733. CrossRefGoogle Scholar
  40. 40.
    Montaño R, Alías F, Ferrer J (2013) Prosodic analysis of storytelling discourse modes and narrative situations oriented to text-to-speech synthesis. In: 8th ISCA workshop on speech synthesisGoogle Scholar
  41. 41.
    Mutlu B, Forlizzi J, Hodgins J (2006) A storytelling robot: modeling and evaluation of human-like gaze behavior. In: 6th IEEE-RAS international conference on humanoid robots, 2006. IEEE, pp 518–523Google Scholar
  42. 42.
    Niculescu A, Van Dijk B, Nijholt A, See SL (2011) The influence of voice pitch on the evaluation of a social robot receptionist. In: 2011 International conference on user science and engineering (i-USEr). IEEE, pp 18–23Google Scholar
  43. 43.
    Paradeda RB, Martinho C, Paiva A (2017) Persuasion based on personality traits: using a social robot as storyteller. In: Companion of the 2017 ACM/IEEE international conference on human–robot interaction, HRI 2017, Vienna, Austria, 6–9 March 2017, pp 367–368.
  44. 44.
    Park HW, Gelsomini M, Lee JJ, Breazeal C (2017) Telling stories to robots: the effect of backchanneling on a child’s storytelling. In: Proceedings of the 2017 ACM/IEEE international conference on human–robot interaction. ACM, pp 100–108Google Scholar
  45. 45.
    Pelachaud C, Gelin R, Martin JC, Le QA (2010) Expressive gestures displayed by a humanoid robot during a storytelling application. In: New frontiers in human-robot interaction (AISB), Leicester, GBGoogle Scholar
  46. 46.
    Pr̆ibil J, Pr̆ibilová A (2008) Application of expressive speech in TTS system with cepstral description. In: Verbal and nonverbal features of human–human and human–machine interaction. Springer, pp 200–212Google Scholar
  47. 47.
    Picard RW (1997) Affective computing. MIT Press, ISBN 978-0262661157Google Scholar
  48. 48.
    Plutchik R (2001) The nature of emotions: human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci 89(4):344–350CrossRefGoogle Scholar
  49. 49.
    Pratchett T, Stewart I, Cohen JS (2003) The science of discworld: the globe, vol 2. Random House, New YorkGoogle Scholar
  50. 50.
    Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161CrossRefGoogle Scholar
  51. 51.
    Striepe H, Lugrin B (2017) There once was a robot storyteller: measuring the effects of emotion and non-verbal behaviour. In: Kheddar A, Yoshida E, Ge SS, Suzuki K, Cabibihan JJ, Eyssel F, He H (eds) Social robotics. Springer, Cham, pp 126–136CrossRefGoogle Scholar
  52. 52.
    Theune M, Meijs K, Heylen D, Ordelman R (2006) Generating expressive speech for storytelling applications. IEEE Trans Audio Speech Lang Process 14(4):1137–1144CrossRefGoogle Scholar
  53. 53.
    Tolkien JRR (1947) On fairy-stories. Oxford University Press, OxfordGoogle Scholar
  54. 54.
    Van Laer T, De Ruyter K, Visconti LM, Wetzels M (2013) The extended transportation–imagery model: a meta-analysis of the antecedents and consequences of consumers’ narrative transportation. J Consum Res 40(5):797–817CrossRefGoogle Scholar
  55. 55.
    Walters ML, Syrdal DS, Koay KL, Dautenhahn K, Te Boekhorst R (2008) Human approach distances to a mechanical-looking robot with different robot voice styles. In: 17th IEEE international symposium on robot and human interactive communication, 2008. RO-MAN 2008. IEEE, pp 707–712Google Scholar
  56. 56.
    Xu J, Broekens J, Hindriks K, Neerincx MA (2015) Effects of a robotic storyteller’s moody gestures on storytelling perception. In: 2015 International conference on affective computing and intelligent interaction (ACII). IEEE, pp 449–455Google Scholar
  57. 57.
    Zipes J (2012) Fairy tales and the art of subversion. Routledge, LondonCrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Hendrik Striepe
    • 1
  • Melissa Donnermann
    • 1
  • Martina Lein
    • 1
  • Birgit Lugrin
    • 1
    Email author
  1. 1.Media Informatics, Human-Computer InteractionUniversity of WuerzburgWürzburgGermany

Personalised recommendations