Systematicity and Idiosyncrasy in Iconic Gesture Use: Empirical Analysis and Computational Modeling

  • Kirsten Bergmann
  • Stefan Kopp
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5934)


Why an iconic gesture takes its particular form is a largely open question, given the variations one finds across both situations and speakers. We present results of an empirical study that analyzes correlations between contextual factors (referent features, discourse) and gesture features, and tests whether they are systematic (shared among speakers) or idiosyncratic (inter-individually different). Based on this, a computational model of gesture formation is presented that combines data-based, probabilistic and model-based decision making.


Iconic gesture meaning-form mapping systematicity idiosyncrasy 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alibali, M., Kita, S., Young, A.: Gesture and the process of speech production: We think, therefore we gesture. Language and cognitive processes 15, 593–613 (2000)CrossRefGoogle Scholar
  2. 2.
    Bavelas, J., Gerwing, J., Sutton, C., Prevost, D.: Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language 58, 495–520 (2008)CrossRefGoogle Scholar
  3. 3.
    Bavelas, J., Kenwood, C., Johnson, T., Philips, B.: An experimental study of when and how speakers use gestures to communicate. Gesture 1, 1–17 (2002)CrossRefGoogle Scholar
  4. 4.
    Bergmann, K., Kopp, S.: Increasing expressiveness for virtual agents–Autonomous generation of speech and gesture for spatial description tasks. In: Decker, K., Sichman, J., Sierra, C., Castelfranchi, C. (eds.) Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems, pp. 361–368 (2009)Google Scholar
  5. 5.
    Bergmann, K., Kopp, S.: GNetIc – using bayesian decision networks for iconic gesture generation. In: Ruttkay, Z., Kipp, M., Nijholt, A., Vilhjálmsson, H.H. (eds.) IVA 2009. LNCS, vol. 5773, pp. 76–89. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Cassell, J., Stone, M., Yan, H.: Coordination and context-dependence in the generation of embodied conversation. In: Proceedings of the First International Conference on Natural Language Generation (2000)Google Scholar
  7. 7.
    de Ruiter, J.: The production of gesture and speech. In: McNeill, D. (ed.) Language and gesture. Cambridge University Press, Cambridge (2000)Google Scholar
  8. 8.
    Denis, M.: The description of routes: A cognitive approach to the production of spatial discourse. Current Psychology of Cognition 16, 409–458 (1997)Google Scholar
  9. 9.
    Gullberg, M., Hendriks, H., Hickmann, M.: Learning to talk and gesture about motion in french. First Language 28(2), 200–236 (2008)CrossRefGoogle Scholar
  10. 10.
    Gwet, K.: Handbook of Inter-Rater Reliability. STATAXIS Publishing Company (2001)Google Scholar
  11. 11.
    Halliday, M.: Notes on transitivity and theme in english (part 2). Journal of Linguistics 3, 199–247 (1967)CrossRefGoogle Scholar
  12. 12.
    Hostetter, A., Alibali, M.: Raise your hand if you’re spatial–Relations between verbal and spatial skills and gesture production. Gesture 7(1), 73–95 (2007)CrossRefGoogle Scholar
  13. 13.
    Hostetter, A., Alibali, M., Kita, S.: I see it in my hands’ eye: Representational gestures reflect conceptual demands. Language and Cogn. Processes 22, 313–336 (2007)CrossRefGoogle Scholar
  14. 14.
    Howard, R., Matheson, J.: Influence diagrams. Decision Analysis 2(3), 127–143 (2005)CrossRefGoogle Scholar
  15. 15.
    Kendon, A.: Gesture–Visible Action as Utterance. Cambridge University Press, Cambridge (2004)Google Scholar
  16. 16.
    Kita, S., Özyürek, A.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)CrossRefGoogle Scholar
  17. 17.
    Kopp, S., Bergmann, K., Wachsmuth, I.: Multimodal communication from multimodal thinking–Towards an integrated model of speech and gesture production. International Journal of Semantic Computing 2(1), 115–136 (2008)CrossRefGoogle Scholar
  18. 18.
    Kopp, S., Tepper, P., Cassell, J.: Towards integrated microplanning of language and iconic gesture for multimodal output. In: Proc. Intern. Conf. on Multimodal Interfaces, pp. 97–104. ACM Press, New York (2004)CrossRefGoogle Scholar
  19. 19.
    Kopp, S., Tepper, P., Ferriman, K., Striegnitz, K., Cassell, J.: Trading spaces: How humans and humanoids use speech and gesture to give directions. In: Nishida, T. (ed.) Conversational Informatics, pp. 133–160. John Wiley, New York (2007)CrossRefGoogle Scholar
  20. 20.
    Kopp, S., Wachsmuth, I.: Synthesizing multimodal utterances for conversational agents. Computer Animation and Virtual Worlds 15(1), 39–52 (2004)CrossRefGoogle Scholar
  21. 21.
    Kruijff-Korbayova, I., Steedman, M.: Discourse and information structure. Journal of Logic, Language and Information 12, 249–259 (2003)zbMATHCrossRefGoogle Scholar
  22. 22.
    Lauritzen, S.L.: The EM algorithm for graphical association models with missing data. Computational Statistics and Data Analysis 19, 191–201 (1995)zbMATHCrossRefGoogle Scholar
  23. 23.
    Madsen, A., Jensen, F., Kjærulff, U., Lang, M.: HUGIN–The tool for bayesian networks and influence diagrams. International Journal of Artificial Intelligence Tools 14(3), 507–543 (2005)CrossRefGoogle Scholar
  24. 24.
    McNeill, D.: Hand and Mind - What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)Google Scholar
  25. 25.
    McNeill, D., Duncan, S.: Growth points in thinking-for-speaking. In: Language and gesture. Cambridge Univ. Press, Cambridge (2000)CrossRefGoogle Scholar
  26. 26.
    Müller, C.: Redebegleitende Gesten: Kulturgeschichte–Theorie–Sprachvergleich. Berlin Verlag, Berlin (1998)Google Scholar
  27. 27.
    Neff, M., Kipp, M., Albrecht, I., Seidel, H.-P.: Gesture modeling and animation based on a probabilistic re-creation of speaker style. ACM Transactions on Graphics 27(1), 1–24 (2008)CrossRefGoogle Scholar
  28. 28.
    Ritz, J., Dipper, S., Götze, M.: Annotation of information structure: An evaluation across different types of texts. In: Proceedings of the 6th LREC conference, pp. 2137–2142 (2008)Google Scholar
  29. 29.
    Sowa, T., Wachsmuth, I.: A model for the representation and processing of shape in coverbal iconic gestures. In: Proc. KogWis 2005, pp. 183–188 (2005)Google Scholar
  30. 30.
    Steck, H., Tresp, V.: Bayesian belief networks for data mining. In: Proceedings of the 2nd Workshop on Data Mining and Data Warehousing (1999)Google Scholar
  31. 31.
    Stone, M., Doran, C., Webber, B., Bleam, T., Palmer, M.: Microplanning with Communicative Intentions: The SPUD System. Comput. Intelligence 19(4), 311–381 (2003)CrossRefMathSciNetGoogle Scholar
  32. 32.
    Streeck, J.: Depicting by gesture. Gesture 8(3), 285–301 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Kirsten Bergmann
    • 1
  • Stefan Kopp
    • 2
  1. 1.SFB 673 Alignment in CommunicationBielefeld UniversityBielefeldGermany
  2. 2.Sociable Agents Group, CITECBielefeld UniversityBielefeldGermany

Personalised recommendations