The Effect of Metaphoric Gestures on Schematic Understanding of Instruction Performed by a Pedagogical Conversational Agent

  • Dai HasegawaEmail author
  • Shinichi Shirakawa
  • Naoya Shioiri
  • Toshiki Hanawa
  • Hiroshi Sakuta
  • Kouzou Ohara
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9192)


In this paper, we examine the impact of metaphoric gestures performed by Pedagogical Conversational Agent (PCA) on learners’ memorization of technical terms, understanding of relationships between abstract concepts, learning experience, and perception of the PCA. The study employed a one-factor three-level between-participants design where we manipulated gesture factor (speech-gesture match vs. speech-gesture mismatch vs. no-gesture). The data of 97 students were acquired in on-line learning environment. As the results, while there was no effect found on memorization of technical terms, we found that students showed accurate schematic understanding of the relationship between abstract concepts when the PCA used metaphoric gestures matched to speech content than when used gestures mismatched, and no gesture. Contrary to the result, we also found that students judged the PCA useful, helpful, and felt the PCA looked like a teacher when performed mismatched gestures to speech content than when performed matched gesture.


Pedagogical agent Metaphoric gesture Understanding Reliability 



This work was partially supported by JSPS Grant-in-Aid for Young Scientists (B) Grant Number 25870698, JSPS Grant-in-Aid for Challenging Exploratory Research Grant Number 26540185, and Council for Science, Technology and Innovation(CSTI), Cross-ministerial Strategic Innovation Promotion Program (SIP), Structural Materials for Innovation (SM4I) (Funding agency:JST).


  1. 1.
    Argyle, M.: Bodily communication. Methuen, London (1975)Google Scholar
  2. 2.
    Arzarello, F., Paola, D., Robutti, O., Sabena, C.: Gestures as semiotic resources in the mathematics classroom. Educ. Stud. Math. 70(2), 97–109 (2009)CrossRefGoogle Scholar
  3. 3.
    Atkinson, R.K.: Optimizing learning from examples using animated pedagogical agents. J. Educ. Psychol. 94(2), 416–427 (2002)CrossRefGoogle Scholar
  4. 4.
    Baylor, A.L., Kim, Y.: Pedagogical agent design: the impact of agent realism, gender, ethnicity, and instructional role. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 592–603. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  5. 5.
    Brebner, J.: Personality theory and movement. In: Kirkcaldy, B.D. (ed.) Individual Differences in Movement, pp. 27–41. Springer, The Netherlands (1985)CrossRefGoogle Scholar
  6. 6.
    Buisine, S., Abrilian, S., Martin, J.C.: Evaluation of multimodal behaviour of embodied agents. In: Ruttkay, Z., Pelachaud, C. (eds.) From Brows to Trust, pp. 217–238. Springer, The Netherlands (2005)Google Scholar
  7. 7.
    Buisine, S., Martin, J.C.: The effects of speech-gesture cooperation in animated agents’ behavior in multimedia presentations. Interact. Comput. 19(4), 484–493 (2007)CrossRefGoogle Scholar
  8. 8.
    Cassell, J., McNeill, D.: Gesture and the poetics of prose. Poetics Today 12, 375–404 (1991)CrossRefGoogle Scholar
  9. 9.
    Cassell, J., McNeill, D., McCullough, K.E.: Speech-gesture mismatches: evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics Cogn. 7(1), 1–34 (1999)CrossRefGoogle Scholar
  10. 10.
    Cook, S.W., Duffy, R.G., Fenn, K.M.: Consolidation and transfer of learning after observing hand gesture. Child Dev. 84(6), 1863–1871 (2013)CrossRefGoogle Scholar
  11. 11.
    Hasegawa, D., Ugurlu, Y., Sakuta, H.: A human-like embodied agent learning tour guide for e-learning systems. In: 2014 IEEE Global Engineering Education Conference (EDUCON), pp. 50–53, April 2014Google Scholar
  12. 12.
    Hayashi, Y.: Togetherness: multiple pedagogical conversational agents as companions in collaborative learning. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 114–123. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  13. 13.
    Heidig, S., Clarebout, G.: Do pedagogical agents make a difference to student motivation and learning? Educ. Res. Rev. 6(1), 27–54 (2011)CrossRefGoogle Scholar
  14. 14.
    Isbister, K., Nass, C.: Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics. Int. J. Hum. Comput. Stud. 53(2), 251–267 (2000)CrossRefGoogle Scholar
  15. 15.
    Johnson, W.L., Rickel, J.: Steve: an animated pedagogical agent for procedural training in virtual environments. SIGART Bull. 8(1–4), 16–21 (1997)CrossRefGoogle Scholar
  16. 16.
    Kellerman, S.: ‘I see what you mean’: the role of kinesic behaviour in listening and implications for foreign and second language learning. Appl. Linguist. 13(3), 239–258 (1992)CrossRefGoogle Scholar
  17. 17.
    Kelly, S.D., Barr, D.J., Church, R.B., Lynch, K.: Offering a hand to pragmatic understanding: the role of speech and gesture in comprehension and memory. J. Mem. Lang. 40(4), 577–592 (1999)CrossRefGoogle Scholar
  18. 18.
    Lester, J.C., Voerman, J.L., Towns, S.G., Callaway, C.B.: Cosmo: A life-like animated pedagogical agent with deictic believability. In: Working Notes of the IJCAI 1997 Workshop on Animated Interface Agents: Making ThemIntelligent, pp. 61–69. Citeseer, August 1997Google Scholar
  19. 19.
    McNeil, D.: Hand and Mind. University of Chicago Press, Chicago (1992)Google Scholar
  20. 20.
    McNeill, D.: Gesture: a psycholinguistic approach. In: Brown, E., Anderson, A. (eds.) The Encyclopedia of Language and Linguistics, pp. 58–66. Elsevier, Amsterdam (2006)CrossRefGoogle Scholar
  21. 21.
    Moreno, R., Mayer, R.E., Spires, H.A., Lester, J.C.: The case for social agency in computer-based teaching: do students learn more deeply when they interact with animated pedagogical agents? Cogn. Instr. 19(2), 177–213 (2001)CrossRefGoogle Scholar
  22. 22.
    Neff, M., Wang, Y., Abbott, R., Walker, M.: Evaluating the effect of gesture and language on personality perception in conversational agents. In: Allbeck, J., Badler, N., Bickermore, T., Pelachcaud, C., Safonova, A. (eds.) IVA 2010. LNCS, vol. 6356, pp. 222–235. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  23. 23.
    Ogan, A., Finkelstein, S., Mayfield, E., D’Adamo, C., Matsuda, N., Cassell, J.: "Oh dear stacy!": social interaction, elaboration, and learning with teachable agents. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2012, pp. 39–48. ACM, New York, NY, USA (2012)Google Scholar
  24. 24.
    Popescu, V., Adamo-Villani, N., Wu, M.L., Rajasekaran, S.D., Alibali, M.W., Nathan, M., Cook, S.W.: Animation killed the video star. In: Proceedings of Gesture-based Interaction Design: Communication and Cognition, 2014 CHI Workshop, pp. 55–59 (2014)Google Scholar
  25. 25.
    Reynolds, F.J., Reeve, R.A.: Gesture in collaborative mathematics problem-solving. J. Math. Behav. 20(4), 447–460 (2001)CrossRefGoogle Scholar
  26. 26.
    Singer, M.A., Goldin-Meadow, S.: Children learn when their teacher’s gestures and speech differ. Psychol. Sci. 16(2), 85–89 (2005)CrossRefGoogle Scholar
  27. 27.
    Valenzeno, L., Alibali, M.W., Klatzky, R.: Teachers’ gestures facilitate students’ learning: a lesson in symmetry. Contemp. Educ. Psychol. 28(2), 187–204 (2003)CrossRefGoogle Scholar
  28. 28.
    Voerman, J.L., FitzGerald, P.J.: Deictic and emotive communication in animated pedagogical agents. In: Cassell, J., Sullican, J., Prevost, S., Churchill, E. (eds.) Embodied Conversational Agents, pp. 123–154. The MIT Press, Cambridge (2000)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Dai Hasegawa
    • 1
    Email author
  • Shinichi Shirakawa
    • 1
  • Naoya Shioiri
    • 1
  • Toshiki Hanawa
    • 1
  • Hiroshi Sakuta
    • 1
  • Kouzou Ohara
    • 1
  1. 1.Aoyama Gakuin UniversitySagamihara-shiJapan

Personalised recommendations