Skip to main content
SpringerLink
Log in

Estimation of Guitar Fingering and Plucking Controls Based on Multimodal Analysis of Motion, Audio and Musical Score

  • Conference paper
  • First Online:
Music, Mind, and Embodiment (CMMR 2015)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9617))

Included in the following conference series:

Abstract

This work presents a method for the extraction of instrumental controls during guitar performances. The method is based on the analysis of multimodal data consisting of a combination of motion capture, audio analysis and musical score. High speed video cameras based on marker identification are used to track the position of finger bones and articulations and audio is recorded with a transducer measuring vibration on the guitar body. The extracted parameters are divided into left hand controls, i.e. fingering (which string and fret is pressed with a left hand finger) and right hand controls, i.e. the plucked string, the plucking finger and the characteristics of the pluck (position, velocity and angles with respect to the string). Controls are estimated based on probability functions of low level features, namely, the plucking instants (i.e. note onsets), the pitch and the distances of the fingers (both hands) to strings and frets. Note onsets are detected via audio analysis, the pitch is extracted from the score and distances are computed from 3D Euclidean Geometry. Results show that by combination of multimodal information, it is possible to estimate such a comprehensive set of control features, with special high performance for the fingering and plucked string estimation. Regarding the plucking finger and the pluck characteristics, their accuracy gets lower but improvements are foreseen including a hand model and the use of high-speed cameras for calibration and evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    http://sourceforge.net/projects/wavesurfer/.

  2. 2.

    http://www.qualisys.com/.

  3. 3.

    http://mathworld.wolfram.com/EulerAngles.html.

References

  1. Abesser, G.S.J., Lukashevich, H.: Feature-based extraction of plucking and expression styles of the electric bass. In: Proceedings of the ICCASP Conference, Kyoto, Japan (2012)

    Google Scholar 

  2. Bevilacqua, F., Schnell, N., Rasamimanana, N., Zamborlin, B., Guédy, F.: Online gesture analysis and control of audio processing. In: Solis, J., Ng, K. (eds.) Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics, vol. 74, pp. 127–142. Springer, Heidelberg (2011). doi:10.1007/978-3-642-22291-78

    Chapter  Google Scholar 

  3. Burns, A.M., Wanderley, M.M.: Visual methods for the retrieval of guitarist fingering. In: Proceedings of NIME, pp. 196–199. IRCAM, Paris (2006). http://dl.acm.org/citation.cfm?id=1142215.1142263

  4. Chadefaux, D., Carrou, J.L.L., Fabre, B., Daudet, L.: Experimentally based description of harp plucking. J. Acoust. Soc. Am. 131(1), 844–855 (2012). http://link.aip.org/link/?JAS/131/844/1

    Article  Google Scholar 

  5. de Cheveigné, A., Kawahara, H.: YIN, a fundamental frequency estimator for speech and music. J. Acoust. Soc. Am. 111(4), 1917–1930 (2002). http://link.aip.org/link/?JAS/111/1917/1

    Article  Google Scholar 

  6. Collins, N., Kiefer, C., Patoli, Z., White, M.: Musical exoskeletons: experiments with a motion capture suit. In: New Interfaces for Musical Expression, Sydney, Australia (2010)

    Google Scholar 

  7. Duxbury, C., Bello, J.P., Davies, M., Sandler, M.: Complex domain onset detection for musical signals. In: Proceedings of the DAFx. Queen Mary University, London (2003)

    Google Scholar 

  8. Eberly, D.H.: 3D Game Engine Design: A Practical Approach to Real-time Computer Graphics. Morgan Kaufmann Publishers Inc., San Francisco (2000)

    Google Scholar 

  9. Erkut, C., Välimäki, V., Karjalainen, M., Laurson, M.: Extraction of physical and expressive parameters for model-based sound synthesis of the classical guitar. In: Audio Engineering Society Convention 108, February 2000. http://www.aes.org/e-lib/browse.cfm?elib=9224

  10. Guaus, E., Arcos, J.L.: Analyzing left hand fingering in guitar playing. In: Proceedings of the SMC. Universitat Pompeu Fabra, Barcelona, July 2010

    Google Scholar 

  11. Heijink, H., Meulenbroek, R.: On the complexity of classical guitar playing: functional adaptations to task constraints. J. Motor Behav. 34(4), 339–351 (2002)

    Article  Google Scholar 

  12. Linden, J.V.D., Schoonderwaldt, E., Bird, J.: Towards a real-time system for teaching novices correct violin bowing technique. In: IEEE International Workshop on Haptic Audio visual Environments and Games, pp. 81–86, November 2009

    Google Scholar 

  13. Maestre, E., Blaauw, M., Bonada, J., Guaus, E., Pérez, A.: Statistical modeling of bowing control applied to sound synthesis. IEEE Trans. Audio Speech Lang. Process. Special Issue on Virtual Analog Audio Effects and Musical Instruments (2010)

    Google Scholar 

  14. Maestre, E., Bonada, J., Blaauw, M., Pérez, A., Guaus, E.: Acquisition of violin instrumental gestures using a commercial EMF device. In: International Computer Music Conference, Copenhagen, Denmark (2007)

    Google Scholar 

  15. Norton, J.: Motion capture to build a foundation for a computer-controlled instrument by study of classical guitar performance. Ph.D. thesis, Department of Music, Stanford University (2008)

    Google Scholar 

  16. Penttinen, H., Välimäki, V.: A time-domain approach to estimating the plucking point of guitar tones obtained with an under-saddle pickup. Appl. Acoust. 65(12), 1207–1220 (2004). http://www.sciencedirect.com/science/article/pii/S0003682X04001057

    Article  Google Scholar 

  17. Pérez-Carrillo, A., Wanderley, M.: Indirect acquisition of violin instrumental controls from audio signal with hidden markov models. IEEE/ACM Trans. Audio Speech Lang. Process. 23(5), 932–940 (2015)

    Article  Google Scholar 

  18. Pérez-Carrillo, A.: Enhancing spectral synthesis techniques with performance gestures using the violin as a case study. Ph.D. thesis, Universitat Pompeu Fabra, Barcelona, Spain (2009). www.mtg.upf.edu/static/media/Perez-Alfonso-PhD-2009.pdf

  19. Pérez-Carrillo, A., Bonada, J., Maestre, E., Guaus, E., Blaauw, M.: Performance control driven violin timbre model based on neural networks. IEEE Trans. Audio Speech Lang. Process. 20(3), 1007–1021 (2012)

    Article  Google Scholar 

  20. Reboursière, L., Lähdeoja, O., Drugman, T., Dupont, S., Picard-Limpens, C., Riche, N.: Left and right-hand guitar playing techniques detection. In: NIME. University of Michigan, Ann Arbor, 21–23 May 2012

    Google Scholar 

  21. Scherrer, B.: Physically-informed indirect acquisition of instrumental gestures on the classical guitar: extracting the angle of release. Ph.D. thesis, McGill University, Montréal, QC (2013)

    Google Scholar 

  22. Scherrer, B., Depalle, P.: Extracting the angle of release from guitar tones: preliminary results. In: Proceedings of Acoustics, Nantes, France (2012)

    Google Scholar 

  23. Schneider, J.: The Contemporary Guitar, New Instrumentation, vol. 5. University of California Press, Berkeley (1985)

    Google Scholar 

  24. Schoonderwaldt, E., Guettler, K., Askenfelt, A.: An empirical investigation of bow-force limits in the Schelleng diagram. AAA 94(4), 604–622 (2008)

    Google Scholar 

  25. Traube, C., Depalle, P.: Deriving the plucking point location along a guitar string from a least-square estimation of a comb filter delay. In: IEEE Canadian Conference on Electrical and Computer Engineering, vol. 3, pp. 2001–2004, May 2003

    Google Scholar 

  26. Traube, C., Smith, J.O.: Estimating the plucking point on a guitar string. In: Proceedings of COST-G6 Conference on Digital Audio Effects, Verona, Italy (2000). http://profs.sci.univr.it/dafx/DAFx-final-papers.html

  27. Visentin, P., Shan, G., Wasiak, E.B.: Informing music teaching and learning using movement analysis technology. Int. J. Music Educ. 26(1), 73–87 (2008). http://ijm.sagepub.com/content/26/1/73.abstract

    Article  Google Scholar 

  28. Wanderley, M.M., Depalle, P.: Gestural control of sound synthesis. In: Proceedings of the IEEE, pp. 632–644 (2004)

    Google Scholar 

  29. Zhang, B., Wang, Y.: Automatic music transcription using audio-visual fusion for violin practice in home environment. Technical report, TRA7/09, Shool of Computing, National University of Singapore (2009)

    Google Scholar 

Download references

Acknowledgments

A. Perez-Carrillo was supported by a Beatriu de Pinos grant 2010 BP-A 00209 by the Catalan Research Agency (AGAUR) and J. Ll. Arcos was supported by ICT -2011-8-318770 and 2009-SGR-1434 projects.

Author information

Authors and Affiliations

  1. IDMIL, McGill University, Montreal, Canada

    Alfonso Perez-Carrillo

  2. IIIA-CSIC, Barcelona, Spain

    Josep-Lluis Arcos

  3. Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain

    Alfonso Perez-Carrillo & Marcelo Wanderley

Authors
  1. Alfonso Perez-Carrillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Josep-Lluis Arcos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcelo Wanderley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alfonso Perez-Carrillo .

Editor information

Editors and Affiliations

  1. CNRS - LMA, Marseille Cedex 20, France

    Richard Kronland-Martinet

  2. CNRS - LMA, Marseille, France

    Mitsuko Aramaki

  3. CNRS - LMA, Marseille, France

    Sølvi Ystad

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Perez-Carrillo, A., Arcos, JL., Wanderley, M. (2016). Estimation of Guitar Fingering and Plucking Controls Based on Multimodal Analysis of Motion, Audio and Musical Score. In: Kronland-Martinet, R., Aramaki, M., Ystad, S. (eds) Music, Mind, and Embodiment. CMMR 2015. Lecture Notes in Computer Science(), vol 9617. Springer, Cham. https://doi.org/10.1007/978-3-319-46282-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46282-0_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46281-3

  • Online ISBN: 978-3-319-46282-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation