Skip to main content
SpringerLink
Log in

Multimodal Analysis of Expressive Gesture in Music Performance

  • Chapter
Musical Robots and Interactive Multimodal Systems

Part of the book series: Springer Tracts in Advanced Robotics ((STAR,volume 74))

Abstract

This chapter focuses on systems and interfaces for multimodal analysis of expressive gesture as a key element of music performance. Research on expressive gesture became particularly relevant in recent years. Psychological studies have been a fundamental source for automatic analysis of expressive gesture since their contribution in identifying the most significant features to be analysed. A further relevant source has been research in the humanistic tradition, in particular choreography. As a major example, in his Theory of Effort, choreographer Rudolf Laban describes the most significant qualities of movement. Starting from these sources, several models, systems, and techniques for analysis of expressive gesture were developed. This chapter presents an overview of methods for the analysis, modelling, and understanding of expressive gesture in musical performance. It introduces techniques resulted from the research developed over the years by the authors: from early experiments of human-robot interaction in the context of music performance up to recent set-ups of innovative interfaces and systems for active experience of sound and music content. The chapter ends with an overview of possible future research challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bernhardt, D., Robinson, P.: Detecting affect from non-stylised body motions. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 59–70. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  2. Boone, R.T., Cunningham, J.G.: Children’s decoding of emotion in expressive body movement: The development of cue attunement. Developmental Psychology 34, 1007–1016 (1998)

    Article  Google Scholar 

  3. Cadoz, C., Wanderley, M.: Gesture – Music. In: Wanderley, M., Battier, M. (eds.) Trends in Gestural Control of Music, Ircam, Paris (2000)

    Google Scholar 

  4. Castellano, G., Mortillaro, M., Camurri, A., Volpe, G., Scherer, K.: Automated Analysis of Body Movement in Emotionally Expressive Piano Performances. Music Perception 26(2), 103–120 (2008)

    Article  Google Scholar 

  5. Camurri, A.: Interactive Dance/Music Systems. In: Proc. Intl. Computer Music Conference ICMC 1995, The Banff Centre for the Arts, pp. 45–252. ICMA-Intl. Comp. Mus. Association, Canada (1995)

    Google Scholar 

  6. Camurri, A., Volpe, G. (eds.): Gesture-based Communication in Human-Computer Interaction. LNCS(LNAI), vol. 2915. Springer, Heidelberg (2004)

    Google Scholar 

  7. Camurri, A., Catorcini, A., Innocenti, C., Massari, A.: Music and Multimedia Knowledge Representation and Reasoning: the HARP System. Computer Music Journal 18(2), 34–58 (1995)

    Article  Google Scholar 

  8. Camurri, A., Coletta, P., Ricchetti, M., Volpe, G.: Expressiveness and Physicality in Interaction. Journal of New Music Research 29(3), 187–198 (2000)

    Article  Google Scholar 

  9. Camurri, A., Hashimoto, S., Ricchetti, M., Trocca, R., Suzuki, K., Volpe, G.: EyesWeb – Toward Gesture and Affect Recognition in Interactive Dance and Music Systems. Computer Music Journal 24(1), 57–69 (2000)

    Article  Google Scholar 

  10. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 20–39. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  11. Camurri, A., Mazzarino, B., Volpe, G.: Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 460–467. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  12. Camurri, A., De Poli, G., Leman, M., Volpe, G.: Toward Communicating Expressiveness and Affect in Multimodal Interactive Systems for Performing Art and Cultural Applications. IEEE Multimedia Magazine 12(1), 43–53 (2005)

    Article  Google Scholar 

  13. Camurri, A., Canepa, C., Drioli, C., Massari, A., Mazzarino, B., Volpe, G.: Multimodal and cross-modal processing in interactive systems based on Tangible Acoustic Interfaces. In: Proc. International Conference Sound and Music Computing 2005, Salerno, Italy (2005)

    Google Scholar 

  14. Camurri, A., Coletta, P., Demurtas, M., Peri, M., Ricci, A., Sagoleo, R., Simonetti, M., Varni, G., Volpe, G.: A Platform for Real-Time Multi-modal Processing. In: Proc. International Conference Sound and Music Computing, Lefkada, Greece, pp. 354–358 (2007)

    Google Scholar 

  15. Camurri, A., Canepa, C., Coletta, P., Ferrari, N., Mazzarino, B., Volpe, G.: The Interactive Piece The Bow is bent and drawn. In: Proc. 3rd ACM International Conference on Digital Interactive Media in Entertainment and Arts (DIMEA 2008), Athens, Greece, pp. 376–383 (2008)

    Google Scholar 

  16. Canazza, S., De Poli, G., Drioli, C., Rodà, A., Vidolin, A.: Audio Morphing Different Expressive Intentions for Multimedia Systems. IEEE Multimedia Magazine 7(3), 79–83 (2000)

    Google Scholar 

  17. Clarke, E.F., Davidson, J.W.: The body in music as mediator between knowledge and action. In: Thomas, W. (ed.) Composition, Performance, Reception: Studies in the Creative Process in Music, pp. 74–92. Oxford University Press, Oxford (1998)

    Google Scholar 

  18. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.: Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine 1 (2001)

    Google Scholar 

  19. De Meijer, M.: The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior 13, 247–268 (1989)

    Article  Google Scholar 

  20. de Rosis, F., Cavalluzzi, A., Mazzotta, I., Novielli, N.: Can embodied conversational agents induce empathy in users? In: Proc. of AISB 2005 Virtual Social Character Symposium (2005)

    Google Scholar 

  21. Dong, W., Lepri, B., Cappelletti, A., Pentland, A., Pianesi, F., Zancanaro, M.: Using the influence model to recognize functional roles in meetings. In: Proc. 9th Intl. ACM Conf. on Multimodal Interfaces, pp. 271–278 (2007)

    Google Scholar 

  22. Gibet, S., Courty, N., Kamp, J.F.: Gesture in Human-Computer Interaction and Simulation. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  23. Hashimoto, S.: KANSEI as the Third Target of Information Processing and Related Topics in Japan. In: Camurri, A. (ed.) Proc. of the International Workshop on KANSEI: The technology of emotion, pp. 101–104. AIMI (Italian Computer Music Association) and DIST-University of Genova (1997)

    Google Scholar 

  24. Juslin, P.N.: Cue utilization in communication of emotion in music performance: relating performance to perception. Journal of Experimental Psychology: Human Perception and Performance 26(6), 1797–1813 (2000)

    Article  Google Scholar 

  25. Kapur, A., Kapur, A., Virji-Babul, N., Tzanetakis, G., Driessen, P.F.: Gesture-based affective computing on motion capture data. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 1–7. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  26. Kopp, S., Wachsmuth, I.: Gesture in Embodied Communication and Human-Computer Interaction. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  27. Krumhansl, C.L.: Can dance reflect the structural and expressive qualities of music? A perceptual experiment on Balanchine’s choreography of Mozart’s Divertimento No. 15. Musicae Scientiae 1, 63–85 (1997)

    Google Scholar 

  28. Kurtenbach, G., Hulteen, E.: Gestures in Human Computer Communication. In: Laurel, B. (ed.) The Art and Science of Interface Design, pp. 309–317. Addison-Wesley, Reading (1990)

    Google Scholar 

  29. Laban, R.: Modern Educational Dance. Macdonald & Evans Ltd., London (1963)

    Google Scholar 

  30. Laban, R., Lawrence, F.C.: Effort. Macdonald & Evans Ltd., London (1947)

    Google Scholar 

  31. Leman, M., et al.: Correlation of gestural musical audio cues and perceived expressive qualities. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 40–54. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  32. Lesaffre, M., Leman, M., Tanghe, K., De Baets, B., De Meyer, H., Martens, J.P.: User-Dependent Taxonomy of Musical Features as a Conceptual Framework for Musical Audio-Mining Technology. In: Proc. Stockholm Music Acoustics Conference SMAC 2003, KTH, Stockholm, Sweden (2003)

    Google Scholar 

  33. Marwan, N., Romano, M.C., Thiel, M., Kurths, J.: Recurrence plots for the analysis of complex systems. Physics Reports 438, 237–329 (2007)

    Article  MathSciNet  Google Scholar 

  34. McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University Of Chicago Press, Chicago (1992)

    Google Scholar 

  35. McQuiggan, S.W., Lester, J.C.: Modeling and evaluating empathy in embodied companion agents. Intl. J. Human-Computer Studies 65(4), 348–360 (2007)

    Article  Google Scholar 

  36. Palmer, C.: Music Performance. Annual Review of Psychology 48, 115–138 (1997)

    Article  Google Scholar 

  37. Pentland, A.: Social signal processing. IEEE Signal Processing Magazine 24(4), 108–111 (2007)

    Article  Google Scholar 

  38. Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39, 1161–1178 (1980)

    Article  Google Scholar 

  39. Sales Dias, M., et al.: Using hand gesture and speech in a multimodal augmented reality environment. In: Sales Dias, M., et al. (eds.) GW 2007. LNCS (LNAI), vol. 5085, pp. 175–180. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  40. Scherer, K.R.: Appraisal considered as a process of multi-level sequential checking. In: Scherer, K.R., Schorr, A., Johnstone, T. (eds.) Appraisal processes in emotion: Theory, Methods, Research, pp. 92–120. Oxford University Press, New Yark (2001)

    Google Scholar 

  41. Scherer, K.R.: Why music does not produce basic emotions: pleading for a new approach to measuring the emotional effects of music. In: Proc. Stockholm Music Acoustics Conference SMAC 2003, KTH, Stockholm, Sweden, pp. 25–28 (2003)

    Google Scholar 

  42. Sloboda, J.A., Lehmann, A.C.: Tracking performance correlates of changes in perceived intensity of emotion during different interpretations of a Chopin piano prelude. Music Perception 19(1), 87–120 (2001)

    Article  Google Scholar 

  43. Tellegen, A., Watson, D., Clark, L.A.: On the dimensional and hierarchical structure of affect. Psychological Science 10(4), 297–303 (1999)

    Article  Google Scholar 

  44. Timmers, R., Marolt, M., Camurri, A., Volpe, G.: Listeners’ emotional engagement with performances of a Skriabin etude: An explorative case study. Psychology of Music 34(4), 481–510 (2006)

    Article  Google Scholar 

  45. Varni, G., Camurri, A., Coletta, P., Volpe, G.: Emotional Entrainment in Music Performance. In: Proc. 8th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2008), Amsterdam, The Netherlands (2008)

    Google Scholar 

  46. Varni, G., Camurri, A., Coletta, P., Volpe, G.: Toward Real-time Automated Measure of Empathy and Dominance. In: Proc. 2009 IEEE International Conference on Social Computing (SocialCom 2009), Vancouver, Canada (2009)

    Google Scholar 

  47. Varni, G., Mancini, M., Volpe, G., Camurri, A.: Sync’n’Move: social interaction based on music and gesture. In: Proc: 1st Intl. ICST Conference on User Centric Media, Venice, Italy (2009)

    Google Scholar 

  48. Vines, B.W., Krumhansl, C.L., Wanderley, M.M., Ioana, M.D., Levitin, D.J.: Dimensions of Emotion in Expressive Musical Performance. Ann. N.Y. Acad. Sci. 1060, 462–466 (2005)

    Article  Google Scholar 

  49. Wallbott, H.G.: Bodily expression of emotion. Eu. J. of Social Psychology 28, 879–896 (1998)

    Article  Google Scholar 

  50. Zbilut, J., Webber Jr., C.L.: Embeddings and delays as derived from quantification of recurrence plots. Phys. Lett. A 5, 199–203 (1992)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Casa Paganini – InfoMus, DIST, University of Genova, Viale Causa 13, 16145, Genova, Italy

    Antonio Camurri & Gualtiero Volpe

Authors
  1. Antonio Camurri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gualtiero Volpe
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Humanoid Robotics Institute Faculty of Science and Engineering, Waseda University, 3-4-1 Ookubo, 168-8555, Shinjuku-ku, Tokyo, Japan

    Jorge Solis

  2. Interdisciplinary Centre for Scientific Research in Music (ICSRiM), University of Leeds School of Computing & School of Music, LS2 9JT, Leeds, UK

    Kia Ng

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Camurri, A., Volpe, G. (2011). Multimodal Analysis of Expressive Gesture in Music Performance. In: Solis, J., Ng, K. (eds) Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics, vol 74. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22291-7_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-22291-7_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-22290-0

  • Online ISBN: 978-3-642-22291-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation