An Augmented Reality Environment to Provide Visual Feedback to Amputees During sEMG Data Acquisitions

  • Francesca PalermoEmail author
  • Matteo Cognolato
  • Ivan Eggel
  • Manfredo Atzori
  • Henning Müller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11650)


Myoelectric hand prostheses have the potential to improve the quality of life of hand amputees. Still, the rejection rate of functional prostheses in the adult population is high. One of the causes is the long time for fitting the prosthesis and the lack of feedback during training. Moreover, prosthesis control is often unnatural and requires mental effort during the training. Virtual and augmented reality devices can help to improve these difficulties and reduce phantom limb pain. Amputees can start training the residual limb muscles with a weightless virtual hand earlier than possible with a real prosthesis. When activating the muscles related to a specific grasp, the subjects receive a visual feedback from the virtual hand. To the best of our knowledge, this work presents one of the first portable augmented reality environment for transradial amputees that combines two devices available on the market: the Microsoft HoloLens and the Thalmic labs Myo. In the augmented environment, rendered by the HoloLens, the user can control a virtual hand with surface electromyography. By using the virtual hand, the user can move objects in augmented reality and train to activate the right muscles for each movement through visual feedback. The environment presented represents a resource for rehabilitation and for scientists. It helps hand amputees to train using prosthetic hands right after the surgery. Scientists can use the environment to develop real time control experiments, without the logistical disadvantages related to dealing with a real prosthetic hand but with the advantages of a realistic visual feedback.


Augmented reality Rehabilitation sEMG prosthesis 


  1. 1.
    Anderson, F., Bischof, W.F.: Augmented reality improves myoelectric prosthesis training. Int. J. Disabil. Hum. Dev. 13(3), 349–354 (2014)CrossRefGoogle Scholar
  2. 2.
    Atzori, M., et al.: Clinical parameter effect on the capability to control myoelectric robotic prosthetic hands. J. Rehabil. Res. Dev. 53(3), 345–358 (2016)CrossRefGoogle Scholar
  3. 3.
    Atzori, M., Gijsberts, A., Müller, H., Caputo, B.: Classification of hand movements in amputated subjects by semg and accelerometers. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3545–3549. IEEE (2014)Google Scholar
  4. 4.
    Atzori, M., Müller, H.: Control capabilities of myoelectric robotic prostheses by hand amputees: a scientific research and market overview. Front. Syst. Neurosci. 9, 162 (2015)CrossRefGoogle Scholar
  5. 5.
    Biddiss, E.A., Chau, T.T.: Upper limb prosthesis use and abandonment: a survey of the last 25 years. Prosthet. Orthot. Int. 31(3), 236–257 (2007)CrossRefGoogle Scholar
  6. 6.
    Bullock, I.M., Zheng, J.Z., De La Rosa, S., Guertler, C., Dollar, A.M.: Grasp frequency and usage in daily household and machine shop tasks. IEEE Trans. Haptics 6(3), 296–308 (2013)CrossRefGoogle Scholar
  7. 7.
    Castellini, C., Gruppioni, E., Davalli, A., Sandini, G.: Fine detection of grasp force and posture by amputees via surface electromyography. J. Physiol.-Paris 103(3–5), 255–262 (2009)CrossRefGoogle Scholar
  8. 8.
    Cipriani, C., et al.: Online myoelectric control of a dexterous hand prosthesis by transradial amputees. IEEE Trans. Neural Syst. Rehabil. Eng. 19(3), 260–270 (2011)CrossRefGoogle Scholar
  9. 9.
    Cognolato, M., et al.: Hand gesture classification in transradial amputees using the Myo armband classifier. In: 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob), pp. 156–161. IEEE (2018)Google Scholar
  10. 10.
    Davidson, J.: A survey of the satisfaction of upper limb amputees with their prostheses, their lifestyles, and their abilities. J. Hand Ther. 15(1), 62–70 (2002)CrossRefGoogle Scholar
  11. 11.
    Dupont, A.C., Morin, E.L.: A myoelectric control evaluation and trainer system. IEEE Trans. Rehabil. Eng. 2(2), 100–107 (1994)CrossRefGoogle Scholar
  12. 12.
    Farina, D., et al.: The extraction of neural information from the surface emg for the control of upper-limb prostheses: emerging avenues and challenges. IEEE Trans. Neural Syst. Rehabil. Eng. 22(4), 797–809 (2014)CrossRefGoogle Scholar
  13. 13.
    Jang, C.H., et al.: A survey on activities of daily living and occupations of upper extremity amputees. Ann. Rehabil. Med. 35(6), 907–921 (2011)CrossRefGoogle Scholar
  14. 14.
    Kuttuva, M., Burdea, G., Flint, J., Craelius, W.: Manipulation practice for upper-limb amputees using virtual reality. Presence: Teleoperators Virtual Environ. 14(2), 175–182 (2005)CrossRefGoogle Scholar
  15. 15.
    Lamounier, E., Lopes, K., Cardoso, A., Andrade, A., Soares, A.: On the use of virtual and augmented reality for upper limb prostheses training and simulation. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2451–2454. IEEE (2010)Google Scholar
  16. 16.
    Mendez, I., et al.: Evaluation of the Myo armband for the classification of hand motions. In: 2017 International Conference on Rehabilitation Robotics (ICORR), pp. 1211–1214. IEEE (2017)Google Scholar
  17. 17.
    Ortiz-Catalan, M., et al.: Phantom motor execution facilitated by machine learning and augmented reality as treatment for phantom limb pain: a single group, clinical trial in patients with chronic intractable phantom limb pain. Lancet 388(10062), 2885–2894 (2016)CrossRefGoogle Scholar
  18. 18.
    Palermo, F., Cognolato, M., Gijsberts, A., Müller, H., Caputo, B., Atzori, M.: Repeatability of grasp recognition for robotic hand prosthesis control based on sEMG data, pp. 1154–1159 (2017)Google Scholar
  19. 19.
    Peerdeman, B., et al.: Myoelectric forearm prostheses: state of the art from a user-centered perspective. J. Rehabil. Res. Dev. 48(6), 719–738 (2011)CrossRefGoogle Scholar
  20. 20.
    Pizzolato, S., Tagliapietra, L., Cognolato, M., Reggiani, M., Müller, H., Atzori, M.: Comparison of six electromyography acquisition setups on hand movement classification tasks. PloS one 12(10), e0186132 (2017)CrossRefGoogle Scholar
  21. 21.
    Roeschlein, R., Domholdt, E.: Factors related to successful upper extremity prosthetic use. Prosthet. Orthot. Int. 13(1), 14–18 (1989)Google Scholar
  22. 22.
    Silcox, D.H., Rooks, M.D., Vogel, R.R., Fleming, L.L.: Myoelectric prostheses. a long-term follow-up and a study of the use of alternate prostheses. JBJS 75(12), 1781–1789 (1993)CrossRefGoogle Scholar
  23. 23.
    Soares, A., Andrade, A., Lamounier, E., Carrijo, R.: The development of a virtual myoelectric prosthesis controlled by an emg pattern recognition system based on neural networks. J. Intell. Inf. Syst. 21(2), 127–141 (2003)CrossRefGoogle Scholar
  24. 24.
    Takeuchi, T., Wada, T., Mukobaru, M., Doi, S.: A training system for myoelectric prosthetic hand in virtual environment. In: IEEE/ICME International Conference on Complex Medical Engineering, CME 2007, pp. 1351–1356. IEEE (2007)Google Scholar
  25. 25.
    Ziegler-Graham, K., MacKenzie, E.J., Ephraim, P.L., Travison, T.G., Brookmeyer, R.: Estimating the prevalence of limb loss in the united states: 2005 to 2050. Arch. Phys. Med. Rehabil. 89(3), 422–429 (2008)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Francesca Palermo
    • 1
    • 2
    Email author
  • Matteo Cognolato
    • 1
    • 3
  • Ivan Eggel
    • 1
  • Manfredo Atzori
    • 1
  • Henning Müller
    • 1
  1. 1.University of Applied Sciences Western Switzerland (HES-SO)SierreSwitzerland
  2. 2.Queen Mary University of LondonLondonUK
  3. 3.Swiss Federal Institute of Technology Zurich (ETH)ZurichSwitzerland

Personalised recommendations