Advertisement

The Role of Iconic Gestures in Production and Comprehension of Language: Evidence from Brain and Behavior

  • Asli Özyürek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5934)

Abstract

Speakers in all cultures and ages use gestures as they speak (i.e., cospeech gestures). There have been different views in the literature with regard to whether and how a specific type of gestures speakers use, i.e., iconic gestures, interacts with language processing. Here I review evidence showing that iconic gestures are not produced merely from the spatial and/or motoric imagery but from an in interface representation of imagistic and linguistic representation during online speaking Similarly, for comprehension, neuroimaging and behavioral studies indicate that speech and gesture influences semantic processing of each other during online comprehension. These findings show overall that processing of information in both modalities interacts during both comprehension and production of language arguing against models that propose independent processing of each modality. They also have implications for AI models that aim to simulate cospeech gesture use in conversational agents.

Keywords

iconic cospeech gesture interface production comprehension brain behavior 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Calvert, A.: Crossmodal processing in the human brain: Insights from functional neuroimaging studies. Cerebral Cortex 11, 1110–1123 (2001)CrossRefGoogle Scholar
  2. 2.
    McNeill, D.: Hand and mind. University of Chicago Press, Chicago (1992)Google Scholar
  3. 3.
    Bavelas, J.B., Gerwing, J., Sutton, C., Prevost, D.: Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language 58, 495–520 (2008)CrossRefGoogle Scholar
  4. 4.
    Özyürek, A.: Gesture in sign and spoken language. In: Pfau, R., Steinbach, M., Woll, B. (eds.) Sign language: An international handbook. Mouton, Berlin (in press)Google Scholar
  5. 5.
    Kendon, A.: Gesture. Cambridge University Press, Cambridge (2004)Google Scholar
  6. 6.
    McNeill, D.: So you think gestures are nonverbal? Psychological Review 92, 350–371 (1985)CrossRefGoogle Scholar
  7. 7.
    Clark, H.: Using language. Cambridge University Press, Cambridge (1996)CrossRefGoogle Scholar
  8. 8.
    Kita, S., Özyürek, A.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)CrossRefGoogle Scholar
  9. 9.
    Hostetter, A.B., Alibali, M.W.: Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review (2008)Google Scholar
  10. 10.
    Krauss, R.M., Chen, Y., Gottesman, R.: Lexical gestures and lexical access: A process model. In: McNeill, D. (ed.) Language and Gesture, pp. 261–284. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  11. 11.
    Feyreisen, P., Lanoy, J.D.: Gestures and speech: Psychological investigations. Cambridge University Press, Cambridge (1991)Google Scholar
  12. 12.
    De Ruiter, J.P.: The production of gesture and speech. In: McNeill, D. (ed.) Language and Gesture, pp. 284–312. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  13. 13.
    Mayberry, R., Jaques, J.: Gesture productino during stuttered speech: insights into the nature of speech-gesture integration. In: McNeill, D. (ed.) Language and Gesture, pp. 199–215. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  14. 14.
    Özyürek, A.: Do speakers design their co-speech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language 46, 688–704 (2002)CrossRefGoogle Scholar
  15. 15.
    Krauss, R.M., Morrel-Samuels, P., Colasante, C.: Do conversational hand gestures communicate? Journal of Personality and Social Psychology 61, 743–754 (1991)CrossRefGoogle Scholar
  16. 16.
    Kelly, S., Özyürek, A., Maris, E.: Two sides of the same coin: Speech and gesture interact to enhance comprehension. Psychological Science (in press)Google Scholar
  17. 17.
    Özyürek, A., Kita, S.: Expressing manner and path in English and Turkish: Differences in speech, gesture, and conceptualization. In: Hahn, M., Stoness, S.C. (eds.) Proceedings of the twenty first annual conference of the Cognitive Science Society, pp. 507–512. Lawrence Erlbaum, Mahwah (1999)Google Scholar
  18. 18.
    Kita, S.: How representational gestures help speaking. In: McNeill, D. (ed.) Language and Gesture, pp. 261–284. Cambridge University Press, Cambridge (2000)Google Scholar
  19. 19.
    Talmy, L.: Semantics and syntax of motion. In: Shopen, T. (ed.) Language typology and syntactic description. Grammatical categories and the lexicon, vol. 3, pp. 57–149. Cambridge University Press, Cambridge (1985)Google Scholar
  20. 20.
    Bock, K.: Towards a cognitive psychology of syntax: Information processing contributions to sentence formulation. Psychological Review 89, 1–47 (1982)CrossRefGoogle Scholar
  21. 21.
    Levelt, P.: Speaking. MIT Press, Cambridge (1989)Google Scholar
  22. 22.
    Özyürek, A., Kita, S., Allen, S., Furman, R., Brown, A.: How does linguistic framing influence co-speech gestures? Insights from cross-linguistic differences and similarities. Gesture 5, 216–241 (2005)CrossRefGoogle Scholar
  23. 23.
    Pederson, E., Danziger, E., Wilkins, D., Levinson, S.C., Kita, S., Senft, G.: Semantic typology and spatial conceptualization. Language 74, 557–589 (1998)CrossRefGoogle Scholar
  24. 24.
    Kita, S., Özyürek, A., Allen, S., Brown, A., Furman, R., Ishizuka, T.: Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production. Language and Cognitive Processes 22(8), 1212–1236 (2007)CrossRefGoogle Scholar
  25. 25.
    Beattie, G., Shovelton, H.: Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica 123, 1 (1999)CrossRefGoogle Scholar
  26. 26.
    Özyürek, A., Willems, R., Kita, S., Hagoort, P.: On-line integration of information from speech and gesture: Insight from event-related potentials. Journal of Cognitive Neuroscience 19(4), 605–616 (2007)CrossRefGoogle Scholar
  27. 27.
    Willems, R., Özyürek, A., Hagoort, P.: When language meets action. The neural integration of speech and gesture. Cerebral Cortex 17, 2322–2333 (2007)CrossRefGoogle Scholar
  28. 28.
    Hagoort, P.: How the brain solves the binding problem for language: a neurocomputational model of syntactic processing. Neuroimage 20, S18–S29 (2003)CrossRefGoogle Scholar
  29. 29.
    Hagoort, P., Hald, L., Bastiaansen, M., Petersson, K.M.: Integration of word meaning and world knowledge in language comprehension. Science 304, 438–441 (2004)CrossRefGoogle Scholar
  30. 30.
    Kopp, S., Bergmann, K., Wachsmuth, I.: Multimodal communication from multimodal thinking – Towards an integrated model of speech and gesture production. Semantic Computing 2(1), 115–136 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Asli Özyürek
    • 1
  1. 1.Center for Language Studies & Max Planck Institute for PsycholinguisticsRadboud University Nijmegen 

Personalised recommendations