Towards a Gesture-Sound Cross-Modal Analysis

  • Baptiste Caramiaux
  • Frédéric Bevilacqua
  • Norbert Schnell
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5934)


This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from subjects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual subjects.


Gesture analysis Gesture-Sound Relationship Sound Perception Canonical Correlation Analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bergmann, K., Kopp, S.: Co-expressivity of speech and gesture: Lessons for models of aligned speech and gesture production. In: Symposium at the AISB Annual Convention: Language, Speech and Gesture for Expressive Characters, December 2007, pp. 153–158 (2007)Google Scholar
  2. 2.
    Berthoz, A.: Le Sens du mouvement. Odile Jacob, Paris (1997)Google Scholar
  3. 3.
    Cadoz, C., Wanderley, M.M.: Gesture-music. In: Trends in Gestural Control of Music, pp. 1–55. Ircam, Paris (2000)Google Scholar
  4. 4.
    Camurri, A., Lagerlöf, I., Volpe, G.: Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques. International Journal of Human-Computer Studies 59(1-2), 213–225 (2003)CrossRefGoogle Scholar
  5. 5.
    Dahl, S., Friberg, A.: Expressiveness of musician’s body movements in performances on marimba. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 479–486. Springer, Heidelberg (2004)Google Scholar
  6. 6.
    Godøy, R.I., Haga, E., Jensenius, A.R.: Playing “Air instruments”: Mimicry of sound-producing gestures by novices and experts. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 256–267. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Haga, E.: Correspondences between music and body movement. PhD thesis, University of Oslo, Department of Musicology (2008)Google Scholar
  8. 8.
    Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E.: Multivariate Data Analysis, 7th edn. Prentice Hall, New Jersey (2009)Google Scholar
  9. 9.
    Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)zbMATHCrossRefGoogle Scholar
  10. 10.
    Jensenius, A.R.: Action-Sound, Developing Methods and Tools to Study Music-Related Body Movement. PhD thesis, University of Oslo, Department of Musicology (2007)Google Scholar
  11. 11.
    Kidron, E., Schechner, Y.Y., Elad, M.: Pixels that sound. IEEE Computer Vision & Pattern Recognition (CVPR 2005) 1, 88–95 (2005)CrossRefGoogle Scholar
  12. 12.
    Kita, S., Asli, Ö.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)CrossRefGoogle Scholar
  13. 13.
    Kohler, E., Keysers, C., Umiltà, A., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: Actions representation in mirror neurons. Science 297, 846–848 (2002)CrossRefGoogle Scholar
  14. 14.
    Kopp, S., Wachsmuth, I.: Synthesizing multimodal utterances for conversational agents. Computer Animation and Virtual Worlds 15(1), 39–52 (2004)CrossRefGoogle Scholar
  15. 15.
    Large, E.W.: On synchronizing movements to music. Human Movement Science 19(4), 527–566 (2000)CrossRefGoogle Scholar
  16. 16.
    Leman, M.: Embodied Music Cognition and Mediation Technology. Massachusetts Institute of Technology Press, Cambridge (2008)Google Scholar
  17. 17.
    Luck, G., Toiviainen, P.: Ensemble musicians’ synchronization with conductors’ gestures: An automated feature-extraction analysis. Music Perception 24(2), 189–200 (2006)CrossRefGoogle Scholar
  18. 18.
    Noë, A.: Action in Perception. Massachusetts Institute of Technology Press, Cambridge (2005)Google Scholar
  19. 19.
    Nusseck, M., Wanderley, M.M.: Music and motion - how music-related ancillary body movements contribute to the experience of music. Music Perception 26, 335–353 (2009)CrossRefGoogle Scholar
  20. 20.
    Peeters, G.: A large set of audio features for sound description. CUIDADO Project (2004)Google Scholar
  21. 21.
    Rasamimanana, N., Fléty, E., Bévilacqua, F.: Gesture analysis of violin bow strokes. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 144–155. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  22. 22.
    Repp, B.H.: Musical Synchronization. In: Altenmüller, E., Wiesendanger, M., Kesselring, J. (eds.) Music, motor control and the brain, pp. 55–76. Oxford University Press, Oxford (2006)Google Scholar
  23. 23.
    Scherer, K.R., Ellgring, H.: Multimodal expression of emotion: Affect programs or componential appraisal patterns? Emotion 7(1), 158–171 (2007)CrossRefGoogle Scholar
  24. 24.
    Styns, F., van Noorden, L., Moelants, D., Leman, M.: Walking on music. Human Movement Science 26(5), 769–785 (2007)CrossRefGoogle Scholar
  25. 25.
    Thomas, M., Vittorio, G.: The emergence of a shared action ontology: Building blocks for a theory. Consciousness and Cognition 12(4), 549–571 (2003)CrossRefGoogle Scholar
  26. 26.
    Varela, F., Thompson, E., Rosch, E.: The Embodied Mind: Cognitive Science and Human Experience. Massachusetts Institute of Technology Press, Cambridge (1991)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Baptiste Caramiaux
    • 1
  • Frédéric Bevilacqua
    • 1
  • Norbert Schnell
    • 1
  1. 1.Real Time Musical Interactions TeamIRCAM, CNRS - STMSParisFrance

Personalised recommendations