Towards a Gesture-Sound Cross-Modal Analysis
This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from subjects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual subjects.
KeywordsGesture analysis Gesture-Sound Relationship Sound Perception Canonical Correlation Analysis
Unable to display preview. Download preview PDF.
- 1.Bergmann, K., Kopp, S.: Co-expressivity of speech and gesture: Lessons for models of aligned speech and gesture production. In: Symposium at the AISB Annual Convention: Language, Speech and Gesture for Expressive Characters, December 2007, pp. 153–158 (2007)Google Scholar
- 2.Berthoz, A.: Le Sens du mouvement. Odile Jacob, Paris (1997)Google Scholar
- 3.Cadoz, C., Wanderley, M.M.: Gesture-music. In: Trends in Gestural Control of Music, pp. 1–55. Ircam, Paris (2000)Google Scholar
- 5.Dahl, S., Friberg, A.: Expressiveness of musician’s body movements in performances on marimba. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 479–486. Springer, Heidelberg (2004)Google Scholar
- 7.Haga, E.: Correspondences between music and body movement. PhD thesis, University of Oslo, Department of Musicology (2008)Google Scholar
- 8.Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E.: Multivariate Data Analysis, 7th edn. Prentice Hall, New Jersey (2009)Google Scholar
- 10.Jensenius, A.R.: Action-Sound, Developing Methods and Tools to Study Music-Related Body Movement. PhD thesis, University of Oslo, Department of Musicology (2007)Google Scholar
- 16.Leman, M.: Embodied Music Cognition and Mediation Technology. Massachusetts Institute of Technology Press, Cambridge (2008)Google Scholar
- 18.Noë, A.: Action in Perception. Massachusetts Institute of Technology Press, Cambridge (2005)Google Scholar
- 20.Peeters, G.: A large set of audio features for sound description. CUIDADO Project (2004)Google Scholar
- 22.Repp, B.H.: Musical Synchronization. In: Altenmüller, E., Wiesendanger, M., Kesselring, J. (eds.) Music, motor control and the brain, pp. 55–76. Oxford University Press, Oxford (2006)Google Scholar
- 26.Varela, F., Thompson, E., Rosch, E.: The Embodied Mind: Cognitive Science and Human Experience. Massachusetts Institute of Technology Press, Cambridge (1991)Google Scholar