“Wear it”—Wearable Robotic Musicians

  • Gil WeinbergEmail author
  • Mason Bretan
  • Guy Hoffman
  • Scott Driscoll
Part of the Automation, Collaboration, & E-Services book series (ACES, volume 8)


Recent developments in wearable technology can help people with disabilities regain their lost capabilities, merging their biological body with robotic enhancements. Myoelectric prosthetic hands, for example, allow amputees to perform basic daily-life activities by sensing and analyzing electric activity from their residual limbs, which is then used to actuate a robotic hand. These new developments not only bring back lost functionalities, but can also provide humanly impossible capabilities, turning those who were considered disabled to become super-abled. The next frontier of Robotic Musicianship research at Georgia Tech focuses on the development of wearable robotic limbs that allow not only amputees, but able-bodied people as well, to play music like no human can, with virtuosity and speed that are humanly impossible. This chapter addresses the promises and challenges of the new frontier of wearable robotic musicians, from a Robotic Prosthetic Drumming Arm that contains a drumming stick with a “mind of its own”, to a “Third Arm” that augments able-bodied drummers, to the Skywalker Piano Hand that uses deep learning predictions from ultrasound muscle data to allow amputees to play the piano using dexterous and expressive finger gestures.


  1. 1.
    Hugh, Herr, Graham Paul Whiteley, and Dudley Childress. 2003. Cyborg technology-biomimetic orthotic and prosthetic technology. SPIE Press, Bellingham, Washington.Google Scholar
  2. 2.
    Llorens-Bonilla, Baldin, Federico Parietti, and H. Harry Asada. 2012. Demonstration-based control of supernumerary robotic limbs. In 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), 3936–3942. IEEE.Google Scholar
  3. 3.
    Brian, Dellon, and Matsuoka Yoky. 2007. Prosthetics, exoskeletons, and rehabilitation [grand challenges of robotics]. Robotics & Automation Magazine, IEEE 14 (1): 30–34.CrossRefGoogle Scholar
  4. 4.
    Schirner, Gunar, Deniz Erdogmus, Kaushik Chowdhury, and Taskin Padir. 2013. The future of human-in-the-loop cyber-physical systems.Google Scholar
  5. 5.
    Li, Qinan, Weidong Chen, and Jingchuan Wang. 2011. Dynamic shared control for human-wheelchair cooperation. In 2011 IEEE international conference on robotics and automation (ICRA), 4278–4283. IEEE.Google Scholar
  6. 6.
    Nudehi, Shahin S., Ranjan Mukherjee, and Moji Ghodoussi. 2005. A shared-control approach to haptic interface design for minimally invasive telesurgical training. IEEE Transactions on Control Systems Technology 13 (4): 588–592.CrossRefGoogle Scholar
  7. 7.
    Gil, Weinberg, and Driscoll Scott. 2006. Toward robotic musicianship. Computer Music Journal 30 (4): 28–45.CrossRefGoogle Scholar
  8. 8.
    Abbink, David A., and M. Mulder. 2010. Neuromuscular analysis as a guideline in designing shared control. Advances in Haptics 109: 499–516.Google Scholar
  9. 9.
    Gentili, Rodolphe J., Hyuk Oh, Isabelle M. Shuggi, Ronald N. Goodman, Jeremy C. Rietschel, Bradley D. Hatfield, and James A. Reggia. 2013. Human-robotic collaborative intelligent control for reaching performance. In Foundations of augmented cognition, 666–675. Springer.Google Scholar
  10. 10.
    Kapur, Ajay, Michael Darling, Dimitri Diakopoulos, Jim W. Murphy, Jordan Hochenbaum, Owen Vallis, and Curtis Bahn. 2011. The machine orchestra: An ensemble of human laptop performers and robotic musical instruments. Computer Music Journal 35 (4): 49–63.CrossRefGoogle Scholar
  11. 11.
    Laura, Maes, Raes Godfried-Willem, and Rogers Troy. 2011. The man and machine robot orchestra at logos. Computer Music Journal 35 (4): 28–48.CrossRefGoogle Scholar
  12. 12.
    Cakmak, Maya, Crystal Chao, and Andrea L. Thomaz. 2010. Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental Development 2 (2): 108–118.CrossRefGoogle Scholar
  13. 13.
    Hoffman, Guy, and Gil Weinberg. 2010. Shimon: An interactive improvisational robotic marimba player. In CHI’10 extended abstracts on human factors in computing systems, 3097–3102. ACM.Google Scholar
  14. 14.
    Cipriani, Christian, Franco Zaccone, Silvestro Micera, and Maria Chiara Carrozza. 2008. On the shared control of an emg-controlled prosthetic hand: analysis of user-prosthesis interaction. IEEE Transactions on Robotics 24 (1): 170–184.CrossRefGoogle Scholar
  15. 15.
    Chris, Lake, and Dodson Robert. 2006. Progressive upper limb prosthetics. Physical Medicine and Rehabilitation Clinics of North America 17 (1): 49–72.CrossRefGoogle Scholar
  16. 16.
    Cipriani, Christian, Marco Controzzi, and M. Chiara Carrozza. 2009. Progress towards the development of the smarthand transradial prosthesis. In IEEE international conference on rehabilitation robotics, 2009. ICORR 2009, 682–687. IEEE.Google Scholar
  17. 17.
    Kim, Hyun K., J. Biggs, David W. Schloerb, Jose M. Carmena, Mikhail A. Lebedev, Miguel A.L. Nicolelis, and Mandayam A. Srinivasan. 2006. Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Transactions on Biomedical Engineering 53 (6): 1164–1173.CrossRefGoogle Scholar
  18. 18.
    Hochberg, Leigh R., Mijail D. Serruya, Gerhard M. Friehs, Jon A. Mukand, Maryam Saleh, Abraham H. Caplan, Almut Branner, David Chen, Richard D. Penn, and John P. Donoghue. 2006. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442 (7099): 164–171.CrossRefGoogle Scholar
  19. 19.
    Wu, Faye Y., and Harry Asada. 2014. Bio-artificial synergies for grasp posture control of supernumerary robotic fingers.Google Scholar
  20. 20.
    Davenport, Clark Clark Michael. 2013. Supernumerary robotic limbs: Biomechanical analysis and human-robot coordination training. PhD thesis, Massachusetts Institute of Technology.Google Scholar
  21. 21.
    Singer, Eric, Jeff Feddersen, Chad Redmon, and Bil Bowen. 2004. Lemur’s musical robots. In Proceedings of the 2004 conference on new interfaces for musical expression, 181–184. National University of Singapore.Google Scholar
  22. 22.
    Weinberg, Gil, and Scott Driscoll. 2006. Robot-human interaction with an anthropomorphic percussionist. In Proceedings of the SIGCHI conference on human factors in computing systems, 1229–1232. ACM.Google Scholar
  23. 23.
    Puckette, Miller S., Miller S. Puckette Ucsd, Theodore Apel, et al. 1998. Real-time audio analysis tools for Pd and MSP.Google Scholar
  24. 24.
    Carlson, Tom, and José del R Millán. 2013. Brain-controlled wheelchairs: A robotic architecture. IEEE Robotics and Automation Magazine 20 (EPFL-ARTICLE-181698): 65–73.CrossRefGoogle Scholar
  25. 25.
    Sun, Sisi, Trishul Mallikarjuna, and Gil Weinberg. Effect of visual cues in synchronization of rhythmic patterns.Google Scholar
  26. 26.
    Guy, Hoffman, and Weinberg Gil. 2011. Interactive improvisation with a robotic marimba player. Autonomous Robots 31 (2–3): 133–153.Google Scholar
  27. 27.
    Kapur, Ajay. 2005. A history of robotic musical instruments. In Proceedings of the international computer music conference, 21–28. Citeseer.Google Scholar
  28. 28.
    Logan-Greene, Richard. The music of Richard Johnson Logan-Greene. Accessed 5 Jan 2014.
  29. 29.
    Ort, Teddy, Faye Wu, Nicholas C. Hensel, and H. Harry Asada. 2015. Supernumerary robotic fingers as a therapeutic device for hemiparetic patients. In ASME 2015 dynamic systems and control conference, V002T27A010–V002T27A010. American Society of Mechanical Engineers.Google Scholar
  30. 30.
    Dheeraj, Nimawat, and Jailiya Pawan Raj Singh. 2015. Requirement of wearable robots in current scenario. European Journal of Advances in Engineering and Technology 2 (2): 19–23.Google Scholar
  31. 31.
    Samer, Mohammed, Amirat Yacine, and Rifai Hala. 2012. Lower-limb movement assistance through wearable robots: State of the art and challenges. Advanced Robotics 26 (1–2): 1–22.Google Scholar
  32. 32.
    Gopura, R.A.R.C., D.S.V. Bandara, Kazuo Kiguchi, and George K.I. Mann. 2016. Developments in hardware systems of active upper-limb exoskeleton robots: A review. Robotics and Autonomous Systems 75: 203–220.CrossRefGoogle Scholar
  33. 33.
    Gálvez-Zúñiga, Miguel A., and Alejandro Aceves-López. 2016. A review on compliant joint mechanisms for lower limb exoskeletons. Journal of Robotics.Google Scholar
  34. 34.
    Gopura, R.A.R.C., Kazuo Kiguchi, and D.S.V. Bandara. 2011. A brief review on upper extremity robotic exoskeleton systems. In 2011 6th international conference on industrial and information systems, 346–351. IEEE.Google Scholar
  35. 35.
    Rocon, E., A.F. Ruiz, R. Raya, A. Schiele, J.L. Pons, J.M. Belda-Lois, R. Poveda, M.J. Vivas, and J.C. Moreno. 2008. Human-robot physical interaction. In Wearable robots: Biomechatronic exoskeletons, 127–163.Google Scholar
  36. 36.
    Allan Joshua Veale and Shane Quan Xie. 2016. Towards compliant and wearable robotic orthoses: A review of current and emerging actuator technologies. Medical Engineering & Physics 38 (4): 317–325.CrossRefGoogle Scholar
  37. 37.
    Lenzi, Tommaso, Nicola Vitiello, Stefano Marco Maria De Rossi, Alessandro Persichetti, Francesco Giovacchini, Stefano Roccella, Fabrizio Vecchi, and Maria Chiara Carrozza. 2011. Measuring human-robot interaction on wearable robots: A distributed approach. Mechatronics 21 (6): 1123–1131.CrossRefGoogle Scholar
  38. 38.
    Knight, James F., Chris Baber, Anthony Schwirtz, and Huw William Bristow. 2002. The comfort assessment of wearable computers. ISWC 2: 65–74.Google Scholar
  39. 39.
    Bodine, Kerry, and Francine Gemperle. 2003. Effects of functionality on perceived comfort of wearables. In Proceedings of seventh IEEE international symposium on wearable computers, 2003, 57–60. IEEE.Google Scholar
  40. 40.
    Demers, Louise, Rhoda Weiss-Lambrou, and Bernadette Ska. 2002. The quebec user evaluation of satisfaction with assistive technology (quest 2.0): An overview and recent progress. Technology and Disability 14 (3): 101–105.Google Scholar
  41. 41.
    Nigel Corlett, E., and R.P. Bishop. 1976. A technique for assessing postural discomfort. Ergonomics 19 (2): 175–182.CrossRefGoogle Scholar
  42. 42.
    Singer, Neil C., and Warren P. Seering. 1990. Preshaping command inputs to reduce system vibration. Journal of Dynamic Systems, Measurement, and Control 112 (1): 76–82.CrossRefGoogle Scholar
  43. 43.
    Zhai, Xiaolong, Beth Jelfs, Rosa H.M. Chan, and Chung Tin. 2016. Short latency hand movement classification based on surface emg spectrogram with PCA. In 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC), 327–330. IEEE.Google Scholar
  44. 44.
    Jun-Uk, Chu, Moon Inhyuk, Lee Yun-Jung, Kim Shin-Ki, and Mun Mu-Seong. 2007. A supervised feature-projection-based real-time emg pattern recognition for multifunction myoelectric hand control. IEEE/ASME Transactions on Mechatronics 12 (3): 282–290.CrossRefGoogle Scholar
  45. 45.
    Huang, Yonghong, Kevin B. Englehart, Bernard Hudgins, and Adrian D.C. Chan. 2005. A gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses. IEEE Transactions on Biomedical Engineering 52 (11): 1801–1811.CrossRefGoogle Scholar
  46. 46.
    Silvia, Muceli, and Farina Dario. 2011. Simultaneous and proportional estimation of hand kinematics from emg during mirrored movements at multiple degrees-of-freedom. IEEE Transactions on Neural Systems and Rehabilitation Engineering 20 (3): 371–378.Google Scholar
  47. 47.
    Afshar, Pedram, and Yoky Matsuoka. 2004. Neural-based control of a robotic hand: Evidence for distinct muscle strategies. In Proceedings of IEEE international conference on robotics and automation, 2004. ICRA’04, vol. 5, 4633–4638. IEEE.Google Scholar
  48. 48.
    Hioki, Masaaki, and Haruhisa Kawasaki. 2012. Estimation of finger joint angles from sEMG using a neural network including time delay factor and recurrent structure. ISRN Rehabilitation 2012.Google Scholar
  49. 49.
    Shrirao, Nikhil A., Narender P. Reddy, and Durga R. Kosuri. 2009. Neural network committees for finger joint angle estimation from surface emg signals. Biomedical Engineering Online 8 (1): 2.CrossRefGoogle Scholar
  50. 50.
    Ngeo, Jimson G., Tomoya Tamei, and Tomohiro Shibata. 2014. Continuous and simultaneous estimation of finger kinematics using inputs from an emg-to-muscle activation model. Journal of Neuroengineering and Rehabilitation 11 (1): 122.CrossRefGoogle Scholar
  51. 51.
    Zheng, Yong-Ping, M.M.F. Chan, Jun Shi, Xin Chen, and Qing-Hua Huang. 2006. Sonomyography: Monitoring morphological changes of forearm muscles in actions with the feasibility for the control of powered prosthesis. Medical Engineering & Physics 28 (5): 405–415.CrossRefGoogle Scholar
  52. 52.
    Castellini, Claudio, and Georg Passig. 2011. Ultrasound image features of the wrist are linearly related to finger positions. In 2011 IEEE/RSJ international conference on intelligent robots and systems, 2108–2114. IEEE.Google Scholar
  53. 53.
    Castellini, Claudio and David Sierra González. 2013. Ultrasound imaging as a human-machine interface in a realistic scenario. In 2013 IEEE/RSJ international conference on intelligent robots and systems, 1486–1492. IEEE.Google Scholar
  54. 54.
    Vikram, Ravindra, and Castellini Claudio. 2014. A comparative analysis of three non-invasive human-machine interfaces for the disabled. Frontiers in Neurorobotics 8: 24.Google Scholar
  55. 55.
    Xin, Chen, Zheng Yong-Ping, Guo Jing-Yi, and Shi Jun. 2010. Sonomyography (smg) control for powered prosthetic hand: a study with normal subjects. Ultrasound in Medicine & Biology 36 (7): 1076–1088.CrossRefGoogle Scholar
  56. 56.
    Guo, Jing-Yi, Yong-Ping Zheng, Laurence P.J. Kenney, Audrey Bowen, David Howard, and Jiri J. Canderle. 2011. A comparative evaluation of sonomyography, electromyography, force, and wrist angle in a discrete tracking task. Ultrasound in Medicine & Biology 37 (6): 884–891.CrossRefGoogle Scholar
  57. 57.
    Li, Yuefeng, Keshi He, Xueli Sun, and Honghai Liu. 2016. Human-machine interface based on multi-channel single-element ultrasound transducers: A preliminary study. In 2016 IEEE 18th international conference on e-health networking, applications and services (Healthcom), 1–6. IEEE.Google Scholar
  58. 58.
    Sikdar, Siddhartha, Huzefa Rangwala, Emily B. Eastlake, Ira A. Hunt, Andrew J. Nelson, Jayanth Devanathan, Andrew Shin, and Joseph J. Pancrazio. 2013. Novel method for predicting dexterous individual finger movements by imaging muscle activity using a wearable ultrasonic system. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22 (1): 69–76.CrossRefGoogle Scholar
  59. 59.
    Hariharan, Harishwaran, Nima Aklaghi, Clayton A. Baker, Huzefa Rangwala, Jana Kosecka, and Siddhartha Sikdar. 2016. Classification of motor intent in transradial amputees using sonomyography and spatio-temporal image analysis. In Medical imaging 2016: Ultrasonic imaging and tomography, vol. 9790, 97901Q. International Society for Optics and Photonics.Google Scholar
  60. 60.
    Gordon, Claire C., Thomas Churchill, Charles E. Clauser, Bruce Bradtmiller, John T. McConville, Ilse Tebbetts, and Robert A. Walker. 1989. Anthropometric survey of us army personnel: Summary statistics, interim report for 1988. Technical report, Anthropology Research Project Inc Yellow Springs OH.Google Scholar
  61. 61.
    Szegedy, Christian, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI conference on artificial intelligence.Google Scholar
  62. 62.
    Khazendar, S., A. Sayasneh, H. Al-Assam, Du Helen, L. Jeroen Kaijser, Dirk Timmerman Ferrara, S. Jassim, and Tom Bourne. 2015. Automated characterisation of ultrasound images of ovarian tumours: The diagnostic accuracy of a support vector machine and image processing with a local binary pattern operator. Facts, Views & Vision in ObGyn 7 (1): 7.Google Scholar
  63. 63.
    Oquab, Maxime, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1717–1724.Google Scholar
  64. 64.
    Cireşan, Dan C., Ueli Meier, and Jürgen Schmidhuber. 2012. Transfer learning for latin and chinese characters with deep neural networks. In The 2012 international joint conference on neural networks (IJCNN), 1–6. IEEE.Google Scholar
  65. 65.
    Yosinski, Jason, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, 3320–3328.Google Scholar
  66. 66.
    Yu, Fisher, and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Gil Weinberg
    • 1
    Email author
  • Mason Bretan
    • 2
  • Guy Hoffman
    • 3
  • Scott Driscoll
    • 4
  1. 1.Georgia Institute of TechnologyAtlantaUSA
  2. 2.NovatoUSA
  3. 3.Cornell UniversityIthacaUSA
  4. 4.AtlantaUSA

Personalised recommendations