Wrist-Worn Sensor-Based Tangible Interface for Virtual Percussion Instruments

  • Abassin Sourou Fangbemi
  • Yanxiang ZhangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10850)


Virtual musical instruments (VMIs) are Augmented Reality (AR) or Virtual Reality (VR) based applications that focus on the interaction with real or virtual musical objects through AR or VR mediums. For such purpose, different interaction tools and techniques such as fiducial markers, RGB and depth cameras, VR and AR headsets are used. Though modern AR and VR devices provide better interaction interfaces, they are more expensive than traditional ones that are unfortunately not robust as the modern devices. To provide a robust and affordable AR and VR interaction interface for real-time applications, we investigate the use of affordable body-worn inertial measurement unit (IMU), a web camera, an RGB light as a simple, cheaper and reliable solution for the interaction with virtual percussion instruments. The performance of the proposed system was evaluated through a series of feedback from fifteen users. In general, the framework was considered as intuitive and tangible.


Virtual musical instruments Percussion Augmented Reality Virtual Reality Inertial measurement unit 



The research was sponsored by CAS-TWAS President’s Fellowship.


  1. 1.
    Poupyrev, I., Berry, R., Kurumisawa, J., Nakao, K., Billinghurst, M., Airola, C., Kato, H., Yonezawa, T., Baldwin, L.: Augmented groove: collaborative jamming in augmented reality. In: ACM SIGGRAPH 2000 Conference Abstracts and Applications, p. 77 (2000)Google Scholar
  2. 2.
    Poupyrev, I., Berry, R., Billinghurst, M., Kato, H., Nakao, K., Baldwin, L., Kurumisawa, J.: Augmented reality interface for electronic music performance. In: Proceedings of HCI, pp. 805–808 (2001)Google Scholar
  3. 3.
    Jordà, S.: Interactive music systems for everyone: exploring visual feedback as a way for creating more intuitive, efficient and learnable instruments. In: Proceedings of the Stockholm Music Acoustics Conference, pp. 6–9, August 2003Google Scholar
  4. 4.
    Karjalainen, M., Maki-Patola, T.: Physics-based modeling of musical instruments for interactive virtual reality. In: 2004 IEEE 6th Workshop on Multimedia Signal Processing, pp. 223–226. IEEE (2004)Google Scholar
  5. 5.
    Wöldecke, B., Marinos, D., Pogscheba, P., Geiger, C., Herder, J., Schwirten, T.: RadarTHEREMIN-Creating musical expressions in a virtual studio environment. In: 2011 IEEE International Symposium on VR Innovation (ISVRI), pp. 345–346. IEEE (2011)Google Scholar
  6. 6.
    Olivieri, D., Conde, I.G., Vila-Sobrino, X.A.: AR-based virtual musical instruments using SMC tracking. In: 2011 6th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–7. IEEE (2011)Google Scholar
  7. 7.
    Rastogi, A., Joshi, A.: Virtual Musical Instruments (2015)Google Scholar
  8. 8.
    Lobo, N.: V-Drum: an augmented reality drum kit. Int. J. Adv. Res. Comput. Commun. Eng. 4(10), October 2015Google Scholar
  9. 9.
    Yamabe, T., Asuma, H., Kiyono, S., Nakajima, T.: Feedback design in augmented musical instruments: a case study with an ar drum kit. In: 2011 IEEE 17th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), vol. 2, pp. 126–129. IEEE (2011)Google Scholar
  10. 10.
    Nilsson, S., Vechev, V., Yeh, A., Hedler, C.: Holo beats: design and development of an AR system to teach drums. In: Proceedings of SIDeR 2016, The 12th Student Interaction Design Research Conference, Malmö (2016)Google Scholar
  11. 11.
    Fernandez, C.A.T., Paliyawan, P., Yin, C.C., Thawonmas, R.: Piano learning application with feedback provided by an AR virtual character. In: 2016 IEEE 5th Global Conference on Consumer Electronics, pp. 1–2. IEEE (2016)Google Scholar
  12. 12.
    Cabral, M., Montes, A., Roque, G., Belloc, O., Nagamura, M., Faria, R.R.A., Teubl, F., Kurashima, C., Lopes, R., Zuffo, M.: Crosscale: a 3D virtual musical instrument interface. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 199–200. IEEE (2015)Google Scholar
  13. 13.
    Fillwalk, J.: Chromachord: a virtual musical instrument. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 201–202. IEEE (2015)Google Scholar
  14. 14.
    Moore, A.G., Howell, M.J., Stiles, A.W., Herrera, N.S., McMahan, R.P.: Wedge: a musical interface for building and playing composition-appropriate immersive environments. In: 2015 IEEE Symposium on 3D User Interfaces (3DUI), pp. 205–206. IEEE (2015)Google Scholar
  15. 15.
    Hariadi, R.R., Kuswardayan, I.: Design and implementation of Virtual Indonesian Musical Instrument (VIMi) application using Leap Motion Controller. In: 2016 International Conference on Information & Communication Technology and Systems (ICTS), pp. 43–48. IEEE (2016)Google Scholar
  16. 16.
    Hsu, M.-H., Kumara, W.G.C.W., Shih, T.K., Cheng, Z.: Spider King: virtual musical instruments based on microsoft kinect. In: 2013 International Joint Conference on Awareness Science and Technology and Ubi-Media Computing (iCAST-UMEDIA), pp. 707–713. IEEE (2013)Google Scholar
  17. 17.
    Burks, N., Smith, L., Saquer, J.: A virtual xylophone for music education. In: 2016 IEEE International Symposium on Multimedia (ISM), pp. 409–410. IEEE (2016)Google Scholar
  18. 18.
    Correa, A.G.D., Ficheman, I.K., do Nascimento, M., de Deus Lopes, R.: Computer assisted music therapy: a case study of an augmented reality musical system for children with cerebral palsy rehabilitation. In: Ninth IEEE International Conference on Advanced Learning Technologies, ICALT 2009, pp. 218–220. IEEE (2009)Google Scholar
  19. 19.
    Chouvatut, V., Jindaluang, W.: Virtual piano with real-time interaction using automatic marker detection. In: 2013 International Computer Science and Engineering Conference (ICSEC), pp. 222–226. IEEE (2013)Google Scholar
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Software EngineeringUniversity of Science and Technology of ChinaHefeiChina
  2. 2.Department of Communication of Science and TechnologyUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations