Skip to main content
SpringerLink
Log in

Online Gesture Analysis and Control of Audio Processing

  • Chapter
Musical Robots and Interactive Multimodal Systems

Part of the book series: Springer Tracts in Advanced Robotics ((STAR,volume 74))

Abstract

This chapter presents a general framework for gesture-controlled audio processing. The gesture parameters are assumed to be multi-dimensional temporal profiles obtained from movement or sound capture systems. The analysis is based on machine learning techniques, comparing the incoming dataflow with stored templates. The mapping procedures between the gesture and the audio processing include a specific method we called temporal mapping. In this case, the temporal evolution of the gesture input is taken into account in the mapping process. We describe an example of a possible use of the framework that we experimented with in various contexts, including music and dance performances, music pedagogy and installations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Arfib, D., Couturier, J., Kessous, L., Verfaille, V.: Strategies of mapping between gesture data and synthesis model parameters using perceptual spaces. Organized Sound 7(2), 127–144 (2002)

    Google Scholar 

  2. Artieres, T., Marukatat, S., Gallinari, P.: Online handwritten shape recognition using segmental hidden markov models. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(2), 205–217 (2007)

    Article  Google Scholar 

  3. Aucouturier, J., Daudet, L.: Editorial: Pattern recognition of non-speech audio. Pattern Recognition Letters 31(12), 1487–1488 (2010)

    Article  Google Scholar 

  4. Bell, B., Kleban, J., Overholt, D., Putnam, L., Thompson, J., Kuchera-Morin, J.: The multimodal music stand. In: NIME 2007: Proceedings of the 7th International Conference on New Interfaces for Musical Expression, pp. 62–65 (2007)

    Google Scholar 

  5. Bettens, F., Todoroff, T.: Real-time DTW-based Gesture Recognition External Object for Max/MSP and PureData. In: SMC: Proceedings of the 6th Sound and Music Computing Conference, pp. 30–35 (2009)

    Google Scholar 

  6. Bevilacqua, F.: Momentary notes on capturing gestures. In (capturing intentions). Emio Greco/PC and the Amsterdam School for the Arts (2007)

    Google Scholar 

  7. Bevilacqua, F., Guédy, F., Schnell, N., Fléty, E., Leroy, N.: Wireless sensor interface and gesture-follower for music pedagogy. In: NIME 2007: Proceedings of the 7th International Conference on New Interfaces for Musical Expression, pp. 124–129 (2007)

    Google Scholar 

  8. Bevilacqua, F., Muller, R.: A gesture follower for performing arts. In: Proceedings of the International Gesture Workshop (2005)

    Google Scholar 

  9. Bevilacqua, F., Muller, R., Schnell, N.: MnM: a max/msp mapping toolbox. In: NIME 2005: Proceedings of the 5th International Conference on New Interfaces for Musical Expression, pp. 85–88 (2005)

    Google Scholar 

  10. Bevilacqua, F., Zamborlin, B., Sypniewski, A., Schnell, N., Guédy, F., Rasamimanana, N.: Continuous Realtime Gesture Following and Recognition. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 73–84. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  11. Bianco, T., Freour, V., Rasamimanana, N., Bevilaqua, F., Caussé, R.: On Gestural Variation and Coarticulation Effects in Sound Control. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 134–145. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  12. Bloit, J., Rasamimanana, N., Bevilacqua, F.: Modeling and segmentation of audio descriptor profiles with segmental models. Pattern Recognition Letters 31, 1507–1513 (2010)

    Article  Google Scholar 

  13. Bloit, J., Rasamimanana, N., Bevilacqua, F.: Towards morphological sound description using segmental models. In: Proceedings of DAFx (2009)

    Google Scholar 

  14. Bobick, A.F., Wilson, A.D.: A state-based approach to the representation and recognition of gesture. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(12), 1325–1337 (1997)

    Article  Google Scholar 

  15. Camurri, A., De Poli, G., Friberg, A., Leman, M., Volpe, G.: The mega project: analysis and synthesis of multisensory expressive gesture in performing art applications. The Journal of New Music Research 34(1), 5–21 (2005)

    Article  Google Scholar 

  16. Camurri, A., Volpe, G., Poli, G.D., Leman, M.: Communicating expressiveness and affect in multimodal interactive systems. IEEE MultiMedia 12(1), 43–53 (2005)

    Article  Google Scholar 

  17. Caramiaux, B., Bevilacqua, F., Schnell, N.: Towards a Gesture-Sound Cross-Modal Analysis. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 158–170. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  18. Caramiaux, B., Bevilacqua, F., Schnell, N.: Analysing Gesture and Sound Similarities with a HMM-based Divergence Measure. In: Proceedings of the Sound and Music Computing Conference, SMC (2010)

    Google Scholar 

  19. Castellano, G., Bresin, R., Camurri, A., Volpe, G.: Expressive control of music and visual media by full-body movement. In: NIME 2007: Proceedings of the 7th International Conference on New Interfaces for Musical Expression, pp. 390–391 (2007)

    Google Scholar 

  20. Chafe, C.: Simulating performance on a bowed instrument. In: Current Directions in Computer Music Research, pp. 185–198. MIT Press, Cambridge (1989)

    Google Scholar 

  21. Cont, A.: Antescofo: Anticipatory synchronization and control of interactive parameters in computer music. In: Proceedings of the International Computer Music Conference, ICMC (2008)

    Google Scholar 

  22. Cont, A., Coduys, T., Henry, C.: Real-time gesture mapping in pd environment using neural networks. In: NIME 2004: Proceedings of the 2004 Conference on New Interfaces for Musical Expression, pp. 39–42 (2004)

    Google Scholar 

  23. Demoucron, M., Rasamimanana, N.: Score-based real-time performance with a virtual violin. In: Proceedings of DAFx (2009)

    Google Scholar 

  24. Dourish, P.: Where the action is: the foundations of embodied interaction. MIT Press, Cambridge (2001)

    Google Scholar 

  25. Fels, S., Hinton, G.: Glove-talk: a neural network interface between a data-glove and a speech synthesizer. IEEE Transactions on Neural Networks 3(6) (1992)

    Google Scholar 

  26. Fine, S., Singer, Y.: The hierarchical hidden markov model: Analysis and applications. Machine Learning 32(1), 41–62 (1998)

    Article  MATH  Google Scholar 

  27. Godøy, R., Haga, E., Jensenius, A.R.: Exploring music-related gestures by sound-tracing - a preliminary study. In: 2nd ConGAS International Symposium on Gesture Interfaces for Multimedia Systems, Leeds, UK (2006)

    Google Scholar 

  28. Godøy, R., Leman, M. (eds.): Musical Gestures: Sound, Movement and Meaning. Routledge, New Yark (2009)

    Google Scholar 

  29. Guédy, F., Bevilacqua, F., Schnell, N.: Prospective et expérimentation pédagogique dans le cadre du projet I-Maestro. In: JIM 2007-Lyon

    Google Scholar 

  30. Guédy, F.: Le traitement du son en pédagogie musicale, vol. 2. Ircam – Editions Léo, L’Inouï (2006)

    Google Scholar 

  31. Henry, C.: Physical modeling for pure data (pmpd) and real time interaction with an audio synthesis. In: Proceedings of the Sound and Music Computing Conference, SMC (2004)

    Google Scholar 

  32. Hoffman, M., Cook, P.R.: Feature-based synthesis: Mapping from acoustic and perceptual features to synthesis parameters. In: Proceedings of International Computer Music Conference, ICMC (2006)

    Google Scholar 

  33. Hunt, A., Wanderley, M., Paradis, M.: The importance of parameter mapping in electronic instrument design. The Journal of New Music Research 32(4) (2003)

    Google Scholar 

  34. Hunt, A., Wanderley, M.M.: Mapping performer parameters to synthesis engines. Organised Sound 7(2), 97–108 (2002)

    Article  Google Scholar 

  35. Jensenius, A.R.: Action-sound: Developing methods and tools to study music-related body movement. PhD thesis, University of Oslo, Department of Musicology, Oslo, Norway (2007)

    Google Scholar 

  36. Lee, E., Grüll, I., Kiel, H., Borchers, J.: Conga: a framework for adaptive conducting gesture analysis. In: NIME 2006: Proceedings of the 2006 Conference on New Interfaces for Musical Expression, pp. 260–265 (2006)

    Google Scholar 

  37. Lee, E., Wolf, M., Borchers, J.: Improving orchestral conducting systems in public spaces: examining the temporal characteristics and conceptual models of conducting gestures. In: CHI 2005: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 731–740 (2005)

    Google Scholar 

  38. Leman, M.: Embodied Music Cognition and Mediation Technology. Massachusetts Institute of Technology Press, Cambridge (2008)

    Google Scholar 

  39. Levitin, D., McAdams, S., Adams, R.: Control parameters for musical instruments: a foundation for new mappings of gesture to sound. Organised Sound 7(2), 171–189 (2002)

    Article  Google Scholar 

  40. Maestre, E.: Modeling Instrumental Gestures: An Analysis/Synthesis Framework for Violin Bowing. PhD thesis, Universitat Pompeu Fabra (2009)

    Google Scholar 

  41. Minka, T.P.: From hidden markov models to linear dynamical systems. Technical report, Tech. Rep. 531, Vision and Modeling Group of Media Lab, MIT (1999)

    Google Scholar 

  42. Miranda, E., Wanderley, M.: New Digital Musical Instruments: Control And Interaction Beyond the Keyboard. A-R Editions, Inc (2006)

    Google Scholar 

  43. Mitra, S., Acharya, T., Member, S., Member, S.: Gesture recognition: A survey. IEEE Transactions on Systems, Man and Cybernetics - Part C 37, 311–324 (2007)

    Article  Google Scholar 

  44. Momeni, A., Henry, C.: Dynamic independent mapping layers for concurrent control of audio and video synthesis. Computer Music Journal 30(1), 49–66 (2006)

    Article  Google Scholar 

  45. Mori, A., Uchida, S., Kurazume, R., Ichiro Taniguchi, R., Hasegawa, T., Sakoe, H.: Early recognition and prediction of gestures. In: Proceedings of the International Conference on Pattern Recognition, vol. 3, pp. 560–563 (2006)

    Google Scholar 

  46. Myers, C.S., Rabiner, L.R.: A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical Journal 60(7), 1389–1409 (1981)

    Google Scholar 

  47. Rabiner, L.R.: A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 257–286 (1989)

    Google Scholar 

  48. Rajko, S., Qian, G.: A hybrid hmm/dpa adaptive gesture recognition method. In: International Symposium on Visual Computing (ISVC), pp. 227–234 (2005)

    Google Scholar 

  49. Rajko, S., Qian, G.: Hmm parameter reduction for practical gesture recognition. In: 8th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2008), pp. 1–6 (2008)

    Google Scholar 

  50. Rajko, S., Qian, G., Ingalls, T., James, J.: Real-time gesture recognition with minimal training requirements and on-line learning. In: CVPR 2007: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  51. Rank, E.: A player model for midi control of synthetic bowed strings. In: Diderot Forum on Mathematics and Music (1999)

    Google Scholar 

  52. Rasamimanana, N., Guedy, F., Schnell, N., Lambert, J.-P., Bevilacqua, F.: Three pedagogical scenarios using the sound and gesture lab. In: Proceedings of the 4th i-Maestro Workshop on Technology Enhanced Music Education (2008)

    Google Scholar 

  53. Rasamimanana, N.H., Bevilacqua, F.: Effort-based analysis of bowing movements: evidence of anticipation effects. The Journal of New Music Research 37(4), 339–351 (2008)

    Article  Google Scholar 

  54. Rasamimanana, N.H., Kaiser, F., Bevilacqua, F.: Perspectives on gesture-sound relationships informed from acoustic instrument studies. Organised Sound 14(2), 208–216 (2009)

    Article  Google Scholar 

  55. Roebel, A.: A new approach to transient processing in the phase vocoder. In: Proceedings of DAFx (September 2003)

    Google Scholar 

  56. Schnell, N., Borghesi, R., Schwarz, D., Bevilacqua, F., Müller, R.: Ftm - complex data structures for max. In: Proceedings of the International Computer Music Conference, ICMC (2005)

    Google Scholar 

  57. Schnell, N., Röbel, A., Schwarz, D., Peeters, G., Borghesi, R.: Mubu and friends: Assembling tools for content based real-time interactive audio processing in max/msp. In: Proceedings of the International Computer Music Conference, ICMC (2009)

    Google Scholar 

  58. Schnell, N., et al.: Gabor, Multi-Representation Real-Time Analysis/Synthesis. In: Proceedings of DAFx (September 2005)

    Google Scholar 

  59. Schwarz, D., Orio, N., Schnell, N.: Robust polyphonic midi score following with hidden markov models. In: Proceedings of the International Computer Music Conference, ICMC (2004)

    Google Scholar 

  60. Serafin, S., Burtner, M., Nichols, C., O’Modhrain, S.: Expressive controllers for bowed string physical models. In: DAFX Conference, pp. 6–9 (2001)

    Google Scholar 

  61. Turaga, P., Chellappa, R., Subrahmanian, V., Udrea, O.: Machine recognition of human activities: A survey. IEEE Transactions on Circuits and Systems for Video Technology 18(11), 1473–1488 (2008)

    Article  Google Scholar 

  62. Van Nort, D., Wanderley, M., Depalle, P.: On the choice of mappings based on geometric properties. In: Proceedings of the International Conference on New Interfaces for Musical Expression, NIME (2004)

    Google Scholar 

  63. Van Nort, D.: Modular and Adaptive Control of Sonic Processes. PhD thesis, McGill University (2010)

    Google Scholar 

  64. Verfaille, V., Wanderley, M., Depalle, P.: Mapping strategies for gestural and adaptive control of digital audio effects. The Journal of New Music Research 35(1), 71–93 (2006)

    Article  Google Scholar 

  65. Volpe, G.: Expressive gesture in performing arts and new media. Journal of New Music Research 34(1) (2005)

    Google Scholar 

  66. Wanderley, M., Depalle, P.: Gestural control of sound synthesis. Proceedings of the IEEE 92, 632–644 (2004)

    Article  Google Scholar 

  67. Wanderley, M.: (guest ed.) Mapping strategies in real-time computer music. Organised Sound, vol. 7(02) (2002)

    Google Scholar 

  68. Wilson, A.D., Bobick, A.F.: Realtime online adaptive gesture recognition. In: Proceedings of the International Conference on Pattern Recognition (1999)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IRCAM - Real Time Musical Interactions Team, STMS IRCAM-CNRS-UPMC, Place Igor Stravinsky, Paris, 75004, France

    Frédéric Bevilacqua, Norbert Schnell, Nicolas Rasamimanana, Bruno Zamborlin & Fabrice Guédy

Authors
  1. Frédéric Bevilacqua
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Norbert Schnell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Rasamimanana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bruno Zamborlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fabrice Guédy
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Humanoid Robotics Institute Faculty of Science and Engineering, Waseda University, 3-4-1 Ookubo, 168-8555, Shinjuku-ku, Tokyo, Japan

    Jorge Solis

  2. Interdisciplinary Centre for Scientific Research in Music (ICSRiM), University of Leeds School of Computing & School of Music, LS2 9JT, Leeds, UK

    Kia Ng

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Bevilacqua, F., Schnell, N., Rasamimanana, N., Zamborlin, B., Guédy, F. (2011). Online Gesture Analysis and Control of Audio Processing. In: Solis, J., Ng, K. (eds) Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics, vol 74. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22291-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-22291-7_8

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-22290-0

  • Online ISBN: 978-3-642-22291-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation