Time-of-Flight Camera Based Virtual Reality Interaction for Balance Rehabilitation Purposes

  • Danilo Avola
  • Luigi Cinque
  • Stefano Levialdi
  • Andrea Petracca
  • Giuseppe Placidi
  • Matteo Spezialetti
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8641)


The 3D Human Body Models (3D HBMs) and the 3D Virtual Reality Environments (3D VREs) enable users to interact with simulated scenarios in an engaging and natural way. The Computer Vision (CV) based Motion Capture (MoCap) systems allow us to obtain user models (i.e., self-avatars) without using cumbersome and uncomfortable physical tools (e.g., sensor suites) which could adversely affect user experience. This last point is of great importance in developing interactive applications for balance rehabilitation purposes where the recovery of lost skills is related to different factors (e.g., patient motivation) including spontaneity of the interaction during the virtual rehabilitative exercises. This paper presents an overview of the Customized Rehabilitation Framework (CRF), a single range imaging sensor based system oriented to patients who experienced with brain strokes, head traumas or neurodegenerative disorders. In particular, the paper is focused on the implementation of two new ad-hoc virtual exercises (i.e., Surfboard and Swing) supporting patients in recovering physical and functional balance. Observations on accuracy of user body models and their real-time interaction ability within rehabilitative simulated environments are presented. In addition, basic experiments concerning usefulness of the proposed exercises to support balance rehabilitation purposes are also reported.


3D human body models 3D virtual reality environments motion capture Time-of-Flight camera self-avatars 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Computer Vision and Image Understanding 104(2), 90–126 (2006)CrossRefGoogle Scholar
  2. 2.
    Berman, S., Stern, H.: Sensors for gesture recognition systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C 42(3), 277–290 (2012)CrossRefGoogle Scholar
  3. 3.
    Chen, L., Wei, H., Ferryman, J.: A survey of human motion analysis using depth imagery. Pattern Recognition Letters 34(15), 1995–2006 (2013)CrossRefGoogle Scholar
  4. 4.
    Zannatha, J.M.I., Tamayo, A.J.M., Sanchez, A.D.G., Delgado, J.E.L., Cheu, L.E.R., Arevalo, W.A.S.: Development of a system based on 3d vision, interactive virtual environments, ergonometric signals and a humanoid for stroke rehabilitation. Computer Methods and Programs in Biomedicine 112(2), 239–249 (2013)CrossRefGoogle Scholar
  5. 5.
    Chang, Y.J., Han, W.Y., Tsai, Y.C.: A kinect-based upper limb rehabilitation system to assist people with cerebral palsy. Research in Developmental Disabilities 34(11), 3654–3659 (2013)CrossRefGoogle Scholar
  6. 6.
    Bohil, C.J., Alicea, B., Biocca, F.A.: Virtual reality in neuroscience research and therapy. Nature Reviews Neuroscience 12, 752–762 (2011)Google Scholar
  7. 7.
    Chang, Y.J., Chen, S.F., Huang, J.D.: A kinect-based system for physical rehabilitation: A pilot study for young adults with motor disabilities. Research in Developmental Disabilities 32(6), 2566–2570 (2011)CrossRefGoogle Scholar
  8. 8.
    Suau, X., Ruiz-Hidalgo, J., Casas, J.R.: Detecting end-effectors on 2. 5d data using geometric deformable models: Application to human pose estimation. Computer Vision and Image Understanding 117(3), 281–288 (2013)Google Scholar
  9. 9.
    Shen, J., Yang, W., Liao, Q.: Part template: 3d representation for multiview human pose estimation. Pattern Recognition 46(7), 1920–1932 (2013)CrossRefGoogle Scholar
  10. 10.
    Westfeld, P., Maas, H.G., Bringmann, O., Grollich, D., Schmauder, M.: Automatic techniques for 3d reconstruction of critical workplace body postures from range imaging data. Journal of Photogrammetry and Remote Sensing 85, 56–65 (2013)CrossRefGoogle Scholar
  11. 11.
    Mohr, R., Bajcsy, R.: Packing volumes by spheres. IEEE Transactions on Pattern Analysis and Machine Intelligence 5(1), 111–116 (1983)CrossRefGoogle Scholar
  12. 12.
    Badler, N., O’Rourke, J., Toltzis, H.: A spherical representation of a human body for visualizing movement. Proceedings of the IEEE 67(10), 1397–1403 (1979)CrossRefGoogle Scholar
  13. 13.
    Avola, D., Spezialetti, M., Placidi, G.: Design of an efficient framework for fast prototyping of customized human-computer interfaces and virtual environments for rehabilitation. Computer Methods and Programs in Biomedicine 110(3), 490–502 (2013)CrossRefGoogle Scholar
  14. 14.
    Spezialetti, M., Avola, D., Placidi, G., De Gasperis, G.: Movement analysis based on virtual reality and 3d depth sensing camera for whole body rehabilitation. In: Computational Modelling of Objects Represented in Images. Fundamentals, Methods and Applications, pp. 367–372. CRC Press, Taylor & Francis Group (2012)Google Scholar
  15. 15.
    Placidi, G.: A smart virtual glove for the hand telerehabilitation. Computers in Biology and Medicine 37(8), 1100–1107 (2007)CrossRefGoogle Scholar
  16. 16.
    Placidi, G., Avola, D., Iacoviello, D., Cinque, L.: Overall design and implementation of the virtual glove. Computers in Biology and Medicine 43(11), 1927–1940 (2013)CrossRefGoogle Scholar
  17. 17.
  18. 18.
  19. 19.
    Vicon (2014),
  20. 20.
  21. 21.
  22. 22.
    Clark, R.A., Pua, Y.H., Fortin, K., Ritchie, C., Webster, K.E., Denehy, L., Bryant, A.L.: Validity of the microsoft kinect for assessment of postural control. Gait & Posture 36(3), 372–377 (2012)CrossRefGoogle Scholar
  23. 23.
  24. 24.
  25. 25.
    Reyes, M., Clapés, A., Ramírez, J., Revilla, J.R., Escalera, S.: Automatic digital biometry analysis based on depth maps. Computers in Industry 64(9), 1316–1325 (2013)CrossRefGoogle Scholar
  26. 26.
    Hernández-López, J.J., Quintanilla-Olvera, A.L., López-Ramírez, J.L., Rangel-Butanda, F.J., Ibarra-Manzano, M.A., Almanza-Ojeda, D.L.: Detecting objects using color and depth segmentation with kinect sensor. Procedia Technology 3, 196–204 (2012)CrossRefGoogle Scholar
  27. 27.
    Camplani, M., Salgado, L.: Background foreground segmentation with rgb-d kinect data: An efficient combination of classifiers. Journal of Visual Communication and Image Representation 25(1), 122–136 (2014)CrossRefGoogle Scholar
  28. 28.
    Diraco, G., Leone, A., Siciliano, P.: In-home hierarchical posture classification with a time-of-flight 3d sensor. Gait & Posture 39(1), 182–187 (2014)CrossRefGoogle Scholar
  29. 29.
    Schonauer, C., Pintaric, T., Kaufmann, H., Jansen Kosterink, S., Vollenbroek-Hutten, M.: Chronic pain rehabilitation with a serious game using multimodal input. In: International Conference on Virtual Rehabilitation (ICVR), pp. 1–8 (2011)Google Scholar
  30. 30.
    IOTracker (2014),
  31. 31.
    Zhu, Y., Dariush, B., Fujimura, K.: Kinematic self retargeting: A framework for human pose estimation. Computer Vision and Image Understanding 114(12), 1362–1375 (2010)CrossRefGoogle Scholar
  32. 32.
    Shum, H.P.H., Ho, E.S.L., Jiang, Y., Takagi, S.: Real-time posture reconstruction for microsoft kinect. IEEE Transaction on Cybernetics 43(5), 1357–1369 (2013)CrossRefGoogle Scholar
  33. 33.
    Metcalf, C., Robinson, R., Malpass, A., Bogle, T., Dell, T., Harris, C., Demain, S.: Markerless motion capture and measurement of hand kinematics: Validation and application to home-based upper limb rehabilitation. IEEE Transactions on Biomedical Engineering 60(8), 2184–2192 (2013)CrossRefGoogle Scholar
  34. 34.
    Rose, F.D., Brooks, B.M., Virtual, A.R.: reality in brain damage rehabilitation: Review. CyberPsychology & Behavior 8(3), 241–262 (2005)CrossRefGoogle Scholar
  35. 35.
    Cho, S., Ku, J., Cho, Y.K., Kim, I.Y., Kang, Y.J., Jang, D.P., Kim, S.I.: Development of virtual reality proprioceptive rehabilitation system for stroke patients. Computer Methods and Programs in Biomedicine 113(1), 258–265 (2014)CrossRefGoogle Scholar
  36. 36.
    Yue, H., Chen, W., Wu, X., Liu, J.: Fast 3d modeling in complex environments using a single kinect sensor. Optics and Lasers in Engineering 53, 104–111 (2014)CrossRefGoogle Scholar
  37. 37.
  38. 38.
  39. 39.
    OpenGL (2014),
  40. 40.
  41. 41.
    Moro, S.B., Bisconti, S., Muthalib, M., Spezialetti, M., Cutini, S., Ferrari, M., Placidi, G., Quaresima, V.: A semi-immersive virtual reality incremental swing balance task activates prefrontal cortex: A functional near-infrared spectroscopy study. NeuroImage 85(pt. 1), 451–460 (2014)CrossRefGoogle Scholar
  42. 42.
    Ferrari, M., Bisconti, S., Spezialetti, M., Basso Moro, S., Palo, C., Placidi, G., Quaresima, V.: Prefrontal cortex activated bilaterally by a tilt board balance task: A functional near-infrared spectroscopy study in a semi-immersive virtual reality environment. Brain Topography, 1–13 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Danilo Avola
    • 1
  • Luigi Cinque
    • 2
  • Stefano Levialdi
    • 2
  • Andrea Petracca
    • 1
  • Giuseppe Placidi
    • 1
  • Matteo Spezialetti
    • 1
  1. 1.Department of Life, Health and Environmental SciencesUniversity of L’AquilaL’AquilaItaly
  2. 2.Department of Computer ScienceSapienza UniversityRomeItaly

Personalised recommendations