Augmented Reality to Enhance the Clinician’s Observation During Assessment of Daily Living Activities

  • M. De CeccoEmail author
  • A. FornaserEmail author
  • P. TomasinEmail author
  • M. ZanettiEmail author
  • G. Guandalini
  • P. G. Ianes
  • F. Pilla
  • G. NolloEmail author
  • M. ValenteEmail author
  • T. PisoniEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10325)


In rehabilitation medicine and in occupational therapy (OT) in particular the assessment tool is essentially the human eye observing the person performing activities of daily living to evaluate his/her level of independence, efficacy, effort, and safety, in order to design an individualized treatment program. On the contrary, in other clinical settings, diagnostics have very sophisticated technological tools such as the Computed Axial Tomography, 3D ultrasound, Functional Magnetic Resonance Imaging, Positron Emission Tomography and many others. Now it is possible to fill this gap in rehabilitation using various enabling technologies currently in a phase of real explosion, through which it will be possible to provide the rehabilitator, in addition to the evidence provided by the human eye, also a large amount of data describing the person’s motion in 3D, the interaction with the environment (forces, contact pressure maps, motion parameters related to the manipulation of objects, etc.), and the ‘internal’ parameters (heart rate, blood pressure, respiratory rate, sweating, etc.). This amount of information can be fed back to the clinician in an animation that represents the reality augmented with all the above parameters using methodologies of Augmented Reality (AR). The main benefit of this new interaction methodology is twofold: the observed scenarios depicted in animations contain all the relevant parameters simultaneously and the related data are well defined and contextualized. This new methodology is a revolution in rehabilitative evaluation methods that allow on one hand to increase the objectivity and effectiveness of clinical observation, and on the other hand to re-define more reliable assessment scales and more effective rehabilitation programs, more user-centered.


Virtual Reality Augmented Reality Human Machine Interface Visual Hull Canadian Occupational Performance Measure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This project was partially funded by the Provincia Autonoma di Trento in the framework of the AUSILIA project.


  1. Pisoni, T., Conci, N., De Natale, F., De Cecco, M., Frattari, A., Guandalini, G.: AUSILIA: assisted unit for simulating independent living activities. In: IEEE International Smart Cities Conference (2016)Google Scholar
  2. Confalonieri, M., Guandalini, G., Da Lio, M., De Cecco, M.: Force and touch make video games ‘serious’ for dexterity rehabilitation. Stud. Health Technol. Inform. 177, 139–144 (2012). H Index: 31Google Scholar
  3. Confalonieri, M., Tomasi, P., Depaul, M., Guandalini, G., Baldessari, M., Oss, D., Prada, F., Mazzalai, A., Da Lio, M., De Cecco, M.: Neuro-physical rehabilitation by means of novel touch technologies. Stud. Health Technol. Inform. 189, 158–163 (2013). ISBN 978-1-61499-267-7Google Scholar
  4. Maule, L., Fornaser, A., Leuci, M., Conci, N., Lio, M., Cecco, M.: Development of innovative HMI strategies for eye controlled wheelchairs in virtual reality. In: Paolis, L.T., Mongelli, A. (eds.) AVR 2016. LNCS, vol. 9769, pp. 358–377. Springer, Cham (2016). doi: 10.1007/978-3-319-40651-0_29 Google Scholar
  5. Cheung, K.M.G., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Proceedings 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1. IEEE (2003)Google Scholar
  6. Hernández, E.C., Schmitt, F.: Silhouette and stereo fusion for 3D object modeling. Comput. Vis. Image Underst. 96(3), 367–392 (2004)CrossRefGoogle Scholar
  7. Hodges, L.F., Kooper, R., Meyer, T.C., Rothbaum, B.O., Opdyke, D., de Graaff, J.J., Williford, J.S., North, M.M.: Virtual environments for treating the fear of heights. IEEE Comput. 28(7), 27–34 (1995)CrossRefGoogle Scholar
  8. Hodges, L.F., Watson, B.A., Kessler, G.D., Rothbaum, B.O., Opdyke, D.: Virtually conquering fear of flying. IEEE Comput. Graph. Appl. 16(6), 42–49 (1996)CrossRefGoogle Scholar
  9. Rothbaum, B.O., Hodges, L., Watson, B.A., Keller, G.D., Opdyke, D.: Virtual reality expo-sure therapy in the treatment of fear of flying: a case report. Behav. Res. Ther. 34(5–6), 477–481 (1996)CrossRefGoogle Scholar
  10. Glantz, K., Durlach, N.I., Barnett, R.C., Aviles, W.A.: Virtual reality (VR) and psychotherapy: opportunities and challenges. Presence 6(1), 87–105 (1997)CrossRefGoogle Scholar
  11. Liu, L., Miyazaki, M., Watson, B.: Norms and validity of the DriVR: a virtu-al reality driving assessment for persons with head injuries. Cyberpsychology Behav. 2(1), 53–67 (1999)CrossRefGoogle Scholar
  12. Alexander, K., et al.: Virtual gait training for children with cerebral palsy using the Lokomat gait orthosis. Stud. Health Technol. Inform. 132, 204–209 (2007)Google Scholar
  13. Andreas, M., et al.: Prospective, blinded, randomized crossover study of gait rehabilitation in stroke patients using the Lokomat gait orthosis. Neurorehabil. Neural Repair 21(4), 307–314 (2007)CrossRefGoogle Scholar
  14. Ichinose, W., Reinkensmeyer, D., Aoyagi, D., Lin, J., Ngai, K., Edgerton, V., Harkema, S., Bobrow, J.: A robotic device for measuring and controlling pelvic motion during locomotor rehabilitation. In: Proceedings of the 2003 IEEE Engineering in Medicine and Biology Society Meeting, pp. 1690–1693 (2003)Google Scholar
  15. Saiwei, Y., et al.: Improving balance skills in patients who had stroke through virtual reality treadmill training. Am. J. Phys. Med. Rehabil. 90(12), 969–978 (2011)CrossRefGoogle Scholar
  16. William, L., et al.: The development of a home-based virtual reality therapy system to promote upper extremity movement for children with hemiplegic cerebral palsy. Technol. Disabil. 21(3), 107–113 (2009)Google Scholar
  17. Sanchez, R.J., et al.: A pneumatic robot for re-training arm movement after stroke: Rationale and mechanical design. In: 9th International Conference on Rehabilitation Robotics, ICORR 2005. IEEE (2005)Google Scholar
  18. Halton, J.: Virtual rehabilitation with video games: A new frontier for occupational therapy. Occup. Ther. Now 9(6), 12–14 (2008)Google Scholar
  19. Gustavo, S., et al.: Effectiveness of virtual reality using Wii gaming technology in stroke rehabilitation a pilot randomized clinical trial and proof of principle. Stroke 41(7), 1477–1484 (2010)CrossRefGoogle Scholar
  20. Amy, H., Korner-Bitensky, N., Levin, M.: Virtual reality in stroke rehabilitation: a systematic review of its effectiveness for upper limb motor recovery. Top. Stroke Rehabil. (2014)Google Scholar
  21. Martini, R., Rios, J., Polatajko, H., Wolf, T., McEwen, S.: The performance quality rating scale (PQRS): reliability, convergent validity, and internal responsiveness for two scoring systems. Disabil. Rehabil. early online, pp. 1–8 (2014)Google Scholar
  22. Fisher, A.G., Jones, K.B.: Assessment of Motor and Process Skills: Volume II – User Manual, 7 Revised edn. Three Star Press Inc., Fort collins (2012)Google Scholar
  23. Law, M., Baptiste, S., Carswell, A., McColl, M.A., Polatajko, H.J., Pollack, N.: Canadian Occupational Performance Measure, 5th edn. CAOT Publications ACE, Ottawa (2014)Google Scholar
  24. Day, H., Jutay, J.: Measuring the psycosocial impact of assistive devices: the PIADS. Can. J. Rehabil. 9(2), 159–168 (1996)Google Scholar
  25. Demers, L., Weiss-Lambrou, R., Ska, B.: Item analysis of the quebec user evaluation of satisfaction with assistive technology (QUEST). Assistive Technol. 12(2), 96–105 (2000)CrossRefGoogle Scholar
  26. Pratt, D.R., Zyda, M., Kelleher, K.: Virtual reality: in the mind of the beholder. IEEE Comput. 28, 17–19 (1995)Google Scholar
  27. Im, D.J., Ku, J., Kim, Y.J., Cho, S., Cho, Y.K., Lim, T., Lee, H.S., Kim, H.J., Kang, Y.J.: Utility of a three-dimensional interactive augmented reality program for balance and mobility rehabilitation in the elderly: a feasibility study. Annuals Rehabil. Med. 39(3), 462–472 (2015)CrossRefGoogle Scholar
  28. Broeren, J., Björkdahl, A., Pascher, R., Rydmark, M.: Virtual reality and haptics as an assessment device in the postacute phase after stroke. CyberPsychology Behav. 5(3), 207–211 (2002)CrossRefGoogle Scholar
  29. Kim, K., Kim, J., Ku, J., Kim, D.Y., Chang, W.H., Shin, D.I., Lee, J.H., Kim, I.Y., Kim, S.I.: A virtual reality assessment and training system for unilateral neglect. CyberPsychology Behav. 7(6), 742–749 (2005)CrossRefGoogle Scholar
  30. Lee, J.H., Ku, J., Cho, W., Hahn, W.Y., Kim, I.Y., Lee, S.M., Kang, Y., Kim, D.Y., Yu, T., Wiederhold, B.K., Wiederhold, M.D., Kim, S.I.: A virtual reality system for the assessment and rehabilitation of the activities of daily living. CyberPsychology Behav. 6(4), 383–388 (2004)CrossRefGoogle Scholar
  31. Josman, N., Kizony, R., Hof, E., Goldenberg, K., Weiss, P.L., Klinger, E.: Using the virtual action planning-supermarket for evaluating executive functions in people with stroke. J. Stroke Cerebrovasc. Dis. 23(5), 879–887 (2014)CrossRefGoogle Scholar
  32. Zhang, L., Sturm, J., Cremers, D., Lee, D.: Real-time human motion tracking using multiple depth cameras. In: International Conference on Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ, pp. 2389–2395. IEEE, October 2012Google Scholar
  33. Pathirana, P.N., Li, S., Trinh, H.M., Seneviratne, A.: Robust real-time bio-kinematic movement tracking using multiple kinects for tele-rehabilitation. IEEE Trans. Ind. Electron. 63(3), 1822–1833 (2016)CrossRefGoogle Scholar
  34. Moon, S., Park, Y., Ko, D.W., Suh, I.H.: Multiple kinect sensor fusion for human skeleton tracking using Kalman filtering. Int. J. Adv. Rob. Syst. 13(2), 65 (2016)CrossRefGoogle Scholar
  35. Huang, H., Wu, S., Cohen-Or, D., Gong, M., Zhang, H., Li, G., Chen, B.: L1-medial skeleton of point cloud. ACM Trans. Graph. 32(4), 65:1 (2013)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of TrentoTrentoItaly
  2. 2.Apss (Apss.Tn.It)TrentoItaly

Personalised recommendations