Skip to main content
SpringerLink
Log in

On Speech and Gestures Synchrony

  • Conference paper
Book cover Analysis of Verbal and Nonverbal Communication and Enactment. The Processing Issues

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6800))

Abstract

Previous research works proved the existence of synchronization between speech and holds in adults and in 9 year old children with a rich linguistic vocabulary and advanced language skills. When and how does this synchrony develop during child language acquisition? Could it be observed also in children younger than 9? The present work aims to answer the above questions reporting on the analysis of narrations produced by three different age groups of Italian children (9, 5 and 3 year olds). Measurements are provided on the amount of synchronization between speech pauses and holds in the three different groups, as a function of the duration of the narrations. The results show that, as far as the reported data concerns, in children, as in adults, holds and speech pauses are to a certain extent synchronized and play similar functions, suggesting that they may be considered as a multi-determined phenomenon exploited by the speaker under the guidance of a unified planning process to satisfy a communicative intention. In addition, considering the role that speech pauses play in communication, we speculate on the possibility that holds may serve to similar purposes supporting the hypothesis that gestures as speech are an expressive resource that can take on different functions depending on the communicative demand. While speech pauses are likely to play the role of signalling mental activation processes aimed at replacing the “old spoken content” of the communicative plan with a new one, holds may signal mental activation processes aimed at replacing the “old visible bodily action” with new ones reflecting the representational and/or propositional contribution of gestures to the new communicative plan.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abrams, K., Bever, T.G.: Syntactic Structure Modifies Attention During Speech Perception and Recognition. Quarterly Journal of Experimental Psychology 21, 280–290 (1969)

    Article  Google Scholar 

  2. Beaugrande, R.: Text Production. Text Publishing Corporation, Norwood (1984)

    Google Scholar 

  3. Bryll, R., Quek, F., Esposito, A.: Automatic Hand Hold Detection in Natural Conversation. In: Proc. of IEEE Workshop on Cues in Communication, Hawai, December 9 (2001)

    Google Scholar 

  4. Butterworth, B.L., Hadar, U.: Gesture, Speech, and Computational Stages: A Reply to McNeill. Psychological Review 96, 168–174 (1989)

    Article  Google Scholar 

  5. Butterworth, B.L., Beattie, G.W.: Gestures and silence as indicator of planning in speech. In: Campbell, R.N., Smith, P.T. (eds.) Recent Advances in the Psychology of Language, pp. 347–360. Olenum Press, New York (1978)

    Chapter  Google Scholar 

  6. Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., Rich, C.: Non-verbal Cues for Discourse Structure. In: Association for Computational Linguistics Joint EACL-ACL Conference (2001a)

    Google Scholar 

  7. Cassell, J., Vilhjalmsson, H., Bickmore, T.: BEAT: The Behavior Expression Animation Toolkit. In: Proc. of SIGGRAPH (2001b)

    Google Scholar 

  8. Chafe, W.L.: Language and Consciousness. Language 50, 111–133 (1974)

    Article  Google Scholar 

  9. Chafe, W.L.: The Deployment of Consciousness in the Production of a Narrative. In: Chafe, W.L. (ed.) The Pear Stories, pp. 9–50. Ablex, Norwood (1980)

    Google Scholar 

  10. Chafe, W.L.: Cognitive Constraint on Information Flow. In: Tomlin, R. (ed.) Coherence and Grounding in Discourse, pp. 20–51. John Benjamins, Amsterdam (1987)

    Google Scholar 

  11. Chen, L., Liu, Y., Harper, M.P., Shriberg, E.: Multimodal Model Integration for Sentence Unit Detection. In: Proceedings of ICMI, State College Pennsylvania, USA, October 13-15 (2004)

    Google Scholar 

  12. Condon, W.S., Sander, L.W.: Synchrony Demonstrated between Movements of the Neonate and Adult Speech. Child Development 45(2), 456–462 (1974)

    Article  Google Scholar 

  13. De Ruiter, J.P.: The Production of Gesture and Speech. In: McNeill, D. (ed.) Language and Gesture, pp. 284–311. Cambridge University Press, UK (2000)

    Chapter  Google Scholar 

  14. Dunn, J.: Children as Psychologist: The Later Correlates of Individual Differences in Understanding Emotion and Other Minds. Cognition and Emotion 9, 187–201 (1995)

    Article  MathSciNet  Google Scholar 

  15. Esposito, A.: Affect in Multimodal Information. In: Tao, J., Tan, T. (eds.) Affective Information Processing, pp. 211–234. Springer, Heidelberg (2008)

    Google Scholar 

  16. Esposito, A., Marinaro, M.: What Pauses Can Tell Us About Speech and Gesture Partnership. In: Esposito, A., et al. (eds.) Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. NATO Publishing Sub-Series E: Human and Societal Dynamics, vol. 18, pp. 45–57. IOS Press, The Netherlands (2007)

    Google Scholar 

  17. Esposito, A., Esposito, D., Refice, M., Savino, M., Shattuck-Hufnagel, S.: A Preliminary Investigation of the Relationships between Gestures and Prosody in Italian. In: Esposito, A., et al. (eds.) Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. NATO Publishing Sub-Series E: Human and Societal Dynamics, vol. 18, pp. 65–74. IOS Press, The Netherlands (2007)

    Google Scholar 

  18. Esposito, A.: Children’s Organization of Discourse Structure Through Pausing Means. In: Faundez-Zanuy, M., Janer, L., Esposito, A., Satue-Villar, A., Roure, J., Espinosa-Duro, V., et al. (eds.) NOLISP 2005. LNCS (LNAI), vol. 3817, pp. 108–115. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  19. Esposito, A.: Pausing Strategies in Children. In: Proceedings of the International Conference in Nonlinear Speech Processing, Cargraphics, Barcelona, Spain, April 19-22, pp. 42–48 (2005)

    Google Scholar 

  20. Esposito, A., Marinaro, M., Palombo, G.: Children Speech Pauses as Markers of Different Discourse Structures and Utterance Information Content. In: Proceedings of the International Conference: From Sound to Sense: +50 Years of Discoveries in Speech Communication, June 10-13, pp. C139–C144. MIT, Cambridge (2004)

    Google Scholar 

  21. Esposito, A., Natale, A., Duncan, S., McNeill, D., Quek, F.: Speech and Gestures Pauses Relationships: A Hypothesis of Synchronization. In: Proceedings of the V National Conference on Italian Psychology, AIP, Grafica80-Modugno, Bari, Italy, pp. 95–98 (2003) (in Italian)

    Google Scholar 

  22. Esposito, A., Duncan, S., Quek, F.: Holds as gestural correlates to empty and filled pauses. In: Proc. of ICSLP, Colorado, vol. 1, pp. 541–544 (2002a)

    Google Scholar 

  23. Esposito, A., Gutierrez-Osuna, R., Kakumanu, P., Garcia, O.N.: Optimal Data Encoding for Speech Driven Facial Animation. Wright State University Technical Report N. CS-WSU-04-02, Dayton, Ohio, USA 1-11 (2002b)

    Google Scholar 

  24. Esposito, A.: On Vowel Height and Consonantal Voicing Effects: Data from Italian. Phonetica 9(4), 197–231 (2002c)

    Article  Google Scholar 

  25. Esposito, A., McCullough, K.E., Quek, F.: Disfluencies in Gesture: Gestural Correlates to Filled and Unfilled Speech Pauses. In: Proc. of IEEE Workshop on Cues in Communication, Hawai (2001)

    Google Scholar 

  26. Esposito, A., Stevens, K.N.: Notes on Italian Vowels: An Acoustical Study (Part I). Research Laboratory of Electronic, Speech Communication Working Papers 10, 1–42 (1995)

    Google Scholar 

  27. Erbaugh, M.S.: A Uniform Pause and Error Strategy for Native and Non-native Speakers. In: Tomlin, R. (ed.) Coherence and Grounding in Discourse, pp. 109–130. John Benjamins, Amsterdam (1987)

    Chapter  Google Scholar 

  28. Ezzat, T., Geiger, G., Poggio, T.: Trainable Video Realistic Speech Animation. In: Proc. of SIGGRAPH, San Antonio, Texas, pp. 388–397 (2002)

    Google Scholar 

  29. Fasel, B., Luettin, J.: Automatic Facial Expression Analysis: A Survey. Pattern Recognition 36(1), 259–275 (2003)

    Article  MATH  Google Scholar 

  30. Flavell, J.H.: Cognitive development: Children’s Knowledge About the Mind. Annual Review of Psychology 50, 21–45 (1999)

    Article  Google Scholar 

  31. Freedman, N.: The Analysis of Movement Behaviour During the Clinical Interview. In: Siegmann, A.W., Pope, B. (eds.) Studies in Dyadic Communication, pp. 177–208. Pergamon Press, Oxford (1972)

    Google Scholar 

  32. Freedman, N., Van Meel, J., Barroso, F., Bucci, W.: On the Development of Communicative Competence. Semiotica 62, 77–105 (1986)

    Article  Google Scholar 

  33. Fu, S., Gutierrez-Osuna, R., Esposito, A., Kakumanu, P., Garcia, O.N.: Audio/Visual Mapping with Cross-Modal Hidden Markov Models. IEEE Transactions on Multimedia 7(2), 243–252 (2005)

    Article  Google Scholar 

  34. Goldmar Eisler, F.: Psycholinguistic: Experiments in Spontaneous Speech. Academic Press, London (1968)

    Google Scholar 

  35. Goldin-Meadow, S.: Gesture: How Our Hands Help Us Think. Harvard University Press, Cambridge (2003)

    Google Scholar 

  36. Green, D.W.: The Immediate Processing of Sentence. Quarterly Journal of Experimental Psychology 29, 135–146 (1977)

    Article  Google Scholar 

  37. Gutierrez-Osuna, R., Kakumanu, P., Esposito, A., Garcia, O.N., Bojorquez, A., Castello, J., Rudomin, I.: Speech-Driven Facial Animation with Realistic Dynamics. IEEE Transactions on Multimedia 7(1), 33–42 (2005)

    Article  Google Scholar 

  38. Hadar, U., Butterworth, B.L.: Iconic Gestures, Imagery and Word Retrieval in Speech. Semiotica 115, 147–172 (1997)

    Article  Google Scholar 

  39. Kähler, K., Haber, J., Seidel, H.: Geometry-based Muscle Modeling for Facial Animation. In: Proc. of Inter. Conf. on Graphics Interface, pp. 27–36 (2001)

    Google Scholar 

  40. Kakumanu, P., Esposito, A., Gutierrez-Osuna, R., Garcia, O.N.: Comparing Different Acoustic Data-Encoding for Speech Driven Facial Animation. Speech Communication 48(6), 598–615 (2006)

    Article  Google Scholar 

  41. Kakumanu, P., Gutierrez-Osuna, R., Esposito, A., Bryll, R., Goshtasby, A., Garcia, O.N.: Speech Dirven Facial Animation. In: Proc. of ACM Workshop on Perceptive User Interfaces, Orlando, November 15-16 (2001)

    Google Scholar 

  42. Kendon, A.: Spacing and Orientation in Co-present Interaction. In: Esposito, A., Campbell, N., Vogel, C., Hussain, A., Nijholt, A. (eds.) Second COST 2102. LNCS, vol. 5967, pp. 1–15. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  43. Kendon, A.: Some Topic in Gesture Study. In: Esposito, A., et al. (eds.) Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. NATO Publishing Sub-Series E: Human and Societal Dynamics, vol. 18, pp. 1–17. IOS Press, The Netherlands (2007)

    Google Scholar 

  44. Kendon, A.: Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge (2004)

    Book  Google Scholar 

  45. Kendon, A.: Sign Languages of Aboriginal Australia: Cultural, Semiotic and Communicative. Cambridge University Press, Cambridge (1988)

    Google Scholar 

  46. Kendon, A.: Current Issues in the Study of Gesture. In: Nespoulous, J.L., et al. (eds.) The Biological Foundations of Gestures: Motor and Semiotic Aspects, pp. 23–27. LEA Publishers, Hillsdale (1986)

    Google Scholar 

  47. Kendon, A.: Gesticulation and Speech: Two Aspects of the Process of Utterance. In: Ritchie Key, M. (ed.) The Relationship of Verbal and Nonverbal Communication, pp. 207–227. Mouton and Co., The Hague (1980)

    Google Scholar 

  48. Kipp, M.: From Human Gesture to Synthetic Action. In: Proc. of Workshop on Multimodal Communication and Context in Embodied Agents, Montreal, pp. 9–14 (2001)

    Google Scholar 

  49. Kita, S., Özyürek, A.: What Does Cross-Linguistic Variation in Semantic Coordination of Speech and Gesture Reveal? Evidence for an Interface Representation of Spatial Thinking and Speaking. Journal of Memory and Language 48, 16–32 (2003)

    Article  Google Scholar 

  50. Kita, S.: How Representational Gestures Help Speaking. In: McNeill, D. (ed.) Language and Gesture, pp. 162–185. Cambridge University Press, UK (2000)

    Chapter  Google Scholar 

  51. Kowal, S., O’Connell, D.C., Sabin, E.J.: Development of Temporal Patterning and Vocal Hesitations in Spontaneous Narratives. Journal of Psycholinguistic Research 4, 195–207 (1975)

    Article  Google Scholar 

  52. Krauss, R., Chen, Y., Gottesman, R.F.: Lexical Gestures and Lexical Access: A Process Model. In: McNeill, D. (ed.) Language and Gesture, pp. 261–283. Cambridge University Press, UK (2000)

    Chapter  Google Scholar 

  53. Krauss, R., Morrel-Samuels, P., Colasante, C.: Do Conversational Hand Gestures Communicate? Journal of Personality and Social Psychology 61(5), 743–754 (1991)

    Article  Google Scholar 

  54. Lee, Y., Terzopoulos, D., Waters, K.: Realistic Modeling for Facial Animation. In: Proc. of SIGGRAPH, pp. 55–62 (1995)

    Google Scholar 

  55. McNeill, D.: Gesture and Thought. In: Esposito, A., et al. (eds.) Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. NATO Publishing Sub-Series E: Human and Societal Dynamics, vol. 18, pp. 18–31. IOS Press, The Netherlands (2007)

    Google Scholar 

  56. McNeill, D.: Gesture and Thought. University of Chicago Press, Chicago (2005)

    Book  Google Scholar 

  57. McNeill, D., Duncan, S.: Growth Points in Thinking for Speaking. In: McNeill, D. (ed.) Language and Gesture, pp. 141–161. Cambridge University Press, UK (2000)

    Chapter  Google Scholar 

  58. McNeill, D.: Hand and Mind: What Gesture Reveal about Thought. University of Chicago Press, Chicago (1992)

    Google Scholar 

  59. Morsella, E., Krauss, R.M.: Muscular Activity in the Arm During Lexical Retrieval: Implications for Gesture-Speech Theories. Journal of Psycholinguistic Research 34, 415–437 (2005)

    Article  Google Scholar 

  60. Morsella, E., Krauss, R.M.: Can Motor States Influence Semantic Processing? Evidence from an Interference Paradigm. In: Columbus, A. (ed.) Advances in Psychology Research, vol. 36, pp. 163–182. Nova, New York (2005a)

    Google Scholar 

  61. Munhall, K.G., Jones, J.A., Callan, D.E., Kuratate, T., Vatikiotis-Bateson, E.: Visual Prosody and Speech Intelligibility. Psychological Science 15(2), 133–137 (2004)

    Article  Google Scholar 

  62. O’Shaughnessy, D.: Timing Patterns in Fluent and Disfluent Spontaneous Speech. In: Proceedings of ICASSP Conference, Detroit, Detroit, pp. 600–603 (1995)

    Google Scholar 

  63. Oliveira, M.: Pausing Strategies as Means of Information Processing Narratives. In: Proceedings of the International Conference on Speech Prosody, Aix-en-Provence, pp. 539–542 (2002)

    Google Scholar 

  64. Prinosil, J., Smekal, Z., Esposito, A.: Combining Features for Recognizing Emotional Facial Expressions in Static Images. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I., et al. (eds.) HH and HM Interaction. LNCS (LNAI), vol. 5042, pp. 56–69. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  65. Rimé, B., Schiaratura, L.: Gesture and Speech. In: Feldman, R.S., Rimé, B. (eds.) Fundamentals of Nonverbal Behavior, Cambridge University Press, pp. 239–284. Cambridge University Press, Cambridge (1992)

    Google Scholar 

  66. Rimé, B.: The Elimination of Visible Behaviour from Social Interactions: Effects of Verbal, Nonverbal, and Interpersonal Variables. European Journal of Social Psychology 12, 113–129 (1982)

    Article  Google Scholar 

  67. Rogers, W.T.: The Contribution of Kinesic Illustrators Towards the Comprehension of Verbal Behaviour Within Utterances. Human Communication Research 5, 54–62 (1978)

    Article  Google Scholar 

  68. Rosenfield, B.: Pauses in Oral and Written Narratives. Boston University Press (1987)

    Google Scholar 

  69. Short, J., Williams, E., Christie, B.: The Social Psychology of Telecommunications. Wiley, New York (1976)

    Google Scholar 

  70. Stocky, T., Cassell, J.: Shared Reality: Spatial Intelligence in Intuitive User Interfaces. In: Proc. of Intelligent User Interfaces, San Francisco, CA, pp. 224–225 (2002)

    Google Scholar 

  71. Shattuck-Hufnagel, S., Yasinnik, Y., Veilleux, N., Renwick, M.: A Method for Studying the Time Alignment of Gestures and Prosody in American English: ‘Hits’ and Pitch Accents in Academic-Lecture-Style Speech. In: Esposito, A., et al. (eds.) Fundamentals of Verbal and Nonverbal Communication and the Biometric Issue. NATO Publishing Sub-Series E: Human and Societal Dynamics, vol. 18, pp. 32–42. IOS Press, The Netherlands (2007)

    Google Scholar 

  72. Thompson, L.A., Massaro, D.W.: Evaluation and Integration of Speech and Pointing Gestures During Referential Understanding. Journal of Experimental Child Psychology 42, 144–168 (1986)

    Article  Google Scholar 

  73. Wellman, H.M.: Early Understanding of Mind: The Normal Case. In: Baron-Cohen, S., et al. (eds.) Understanding Other Mind: Perspective from Children with Autism, pp. 10–39. Oxford Univ. Press, Oxford (1993)

    Google Scholar 

  74. Williams, E.: Experimental Comparisons of Face-to-Face and Mediated Communication: A Review. Psychological Bulletin 84, 963–976 (1977)

    Article  Google Scholar 

  75. Yasinnik, Y., Renwick, M., Shattuck-Hufnagel, S.: The Timing of Speech-Accompanying Gestures with Respect to Prosody. In: Proceedings of the International Conference: From Sound to Sense: +50 Years of Discoveries in Speech Communication, June 10-13, pp. C97–C102. MIT, Cambridge (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dep. of Psychology, Second University of Naples, Via Vivaldi 43, 81100, Caserta, Italy

    Anna Esposito

  2. IIASS, Via Pellegrino 19, 84019, Vietri sul Mare, SA, Italy

    Anna Esposito

  3. Istituto Nazionale di Geofisica e Vulcanologia, sezione di Napoli Osservatorio Vesuviano, Napoli, Italy

    Antonietta M. Esposito

Authors
  1. Anna Esposito
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonietta M. Esposito
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dept. of Psychology and IIASS, International Institute for Advanced Scientific Studies, Second University of Naples, Vietri sul Mare, SA, Italy

    Anna Esposito

  2. School of Computing Science, University of Glasgow, Glasgow, UK

    Alessandro Vinciarelli

  3. Department of Telecommunication and Media Informatics, Laboratory of Speech Acoustics, Budapest University of Technology and Economics, 1117, Budapest, Hungary

    Klára Vicsi

  4. TELECOM ParisTech, CNRS-LTCI UMR 5141, 75014, Paris, France

    Catherine Pelachaud

  5. Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, 7500 AE, Enschede, The Netherlands

    Anton Nijholt

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Esposito, A., Esposito, A.M. (2011). On Speech and Gestures Synchrony. In: Esposito, A., Vinciarelli, A., Vicsi, K., Pelachaud, C., Nijholt, A. (eds) Analysis of Verbal and Nonverbal Communication and Enactment. The Processing Issues. Lecture Notes in Computer Science, vol 6800. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-25775-9_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-25775-9_25

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-25774-2

  • Online ISBN: 978-3-642-25775-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation