3D Dynamic Pose Estimation from Markerless Optical Data

Reference work entry

Abstract

This chapter provides an overview of three-dimensional (3D) dynamic Pose (position and orientation) estimation of human movement without the use of markers or sensors, more commonly known as Markerless Motion Capture (Markerless Mocap). As with Marker-based Motion Capture (Marker-based Mocap), the methods presented estimate the Pose of an underlying multibody subject-specific model comprising rigid segments with anatomically defined local reference frames and joint constraints. In addition, the model has an overlying surface representing the skin, or clothing, depending on the context.

The focus of this chapter is on Markerless Mocap algorithms best suited to biomechanical analyses of human movement. In other words, those techniques appropriate for estimating 3D Pose directly, and accurately, from recorded data. Of all the approaches to Markerless Mocap, 3D-to-3D Pose estimation is most similar to Marker-based Mocap techniques because it requires arrays of multiple, time synchronous, video cameras encircling the capture volume. In addition to the underlying multibody skeletal model that marker-based and markerless techniques have in common, during Markerless Mocap, the subject is identified by a surface model overlying the skeleton. In each frame of motion data, a pixelated surface, comprised of a dense collection of points lying on the surface, is extracted from the scene and registered to the model.

Neither marker-based nor 3D-to-3D Markerless Mocap is typically accurate enough to record the Pose of the bones at a resolution for studying joint dynamics. An alternative markerless approach to joint level biomechanics has emerged. Biplanar videogradiography (or Dynamic Stereo X-ray) uses a 3D-to-2D approach to Markerless Mocap, whereby only two views of the subject are acquired because of space limitations and to minimize radiation exposure. A brief introduction to 3D-to-2D registration will be presented because this is covered in more detail in another chapter.

Keywords

Markerless Mocap Marker-based Mocap Multibody 3D Pose estimation Articulated registration Space carving Stereo reconstruction Biplanar videoradiography 3D-to-3D registration 3D-to-2D registration Visual hull 

References

  1. Anderst WJ, Vaidya R, Tashman S (2008) A technique to measure three-dimensional in vivo rotation of fused and adjacent lumbar vertebrae. Spine J 8:991–997CrossRefGoogle Scholar
  2. Anderst WJ, Donaldson WF, Lee JY, Kang JD (2014) In vivo cervical facet joint capsule deformation during flexion-extension. Spine J 39(8):514–520CrossRefGoogle Scholar
  3. Anderst W, Donaldson W, Lee J, Kang J (2013) Cervical disc deformation during flexion-extension in asymptomatic controls and single-level arthrodesis patients. J Orthop Res 31(12):1881–1889CrossRefGoogle Scholar
  4. Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J, (2005) SCAPE: shape completion and animation of people. In: ACM transactions on graphics (TOG), vol 24, no 3. ACM, pp 408–416Google Scholar
  5. Balan AO, Sigal L, Black MJ, Davis JE, Haussecker HW (2007) Detailed human shape and pose from images. In: 2007 IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8Google Scholar
  6. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: European conference on computer vision. Springer, Berlin/Heidelberg, pp 404–417Google Scholar
  7. Besl PJ, McKay ND (1992) Method for registration of 3-D shapes. In: Robotics-DL tentative. International Society for Optics and Photonics, pp 586–606Google Scholar
  8. Bey MJ, Zauel R, Brock SK, Tashman S (2006) Validation of a new model-based tracking technique for measuring three-dimensional, in vivo glenohumeral joint kinematics. J Biomech Eng 128:604–609CrossRefGoogle Scholar
  9. Bey MJ, Kline SK, Tashman S, Zauel R (2008) Accuracy of biplane X-ray imaging combined with model-based tracking for measuring in-vivo patellofemoral joint motion. J Orthop Surg Res 3:38CrossRefGoogle Scholar
  10. Bey MJ, Peltz CD, Ciarelli K, Kline SK, Divine GW, van Holsbeeck M, Muh S, Kolowich P, Lock T, Moutzouros V (2011) In vivo shoulder function after surgical repair of a torn rotator cuff: Glenohumeral joint mechanics, shoulder strength, clinical outcomes, and their interaction. Am J Sports Med 10:2117–2129CrossRefGoogle Scholar
  11. Bogo F, Romero J, Loper M, Black MJ (2014) FAUST: Dataset and evaluation for 3D mesh registration In: Proceedings IEEE Conference on computer vision and pattern recognition (CVPR), pp 3794–3801Google Scholar
  12. Bogo F, Black MJ, Loper M, Romero J (2015) Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In: International conference on computer vision (ICCV), pp 2300–2308Google Scholar
  13. Brainerd EL, Baier DB, Gatesy SM, Hedrick TL, Metzger KA, Crisco JJ (2010) X-ray reconstruction of moving morphology (XROMM): precision, accuracy and applications in comparative biomechanics research. J Exp Zool 313A:262–279Google Scholar
  14. Cappozzo A, Catani F, Leardini A, Benedetti MG, Della Croce U (1996) Position and orientation in space of bones during movement: experimental artefacts. Clin Biomech 11(2):90–100Google Scholar
  15. Cheung KMG, Baker S, Kanade T (2003) Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Computer vision and pattern recognition, 2003. Proceedings. 2003 I.E. computer society conference on, vol 1, IEEE, pp 1–77Google Scholar
  16. Corazza S, Mündermann L, Gambaretto E, Ferrigno G, Andriacchi TP (2010) Markerless motion capture through visual hull, articulated icp and subject specific model generation. Int J Comput Vis 87(1–2):156–169CrossRefGoogle Scholar
  17. Della Croce U, Leardini A, Chiari L, Cappozzo A (2005) Human movement analysis using stereophotogrammetry. Part 4: assessment of anatomical landmark misplacement and its effects on joint kinematics. Gait Posture 21(2):226–237Google Scholar
  18. Fernandez JW, Akbarshahi M, Kim HJ, Pandy MG (2008) Integrating modelling, motion capture and x-ray fluoroscopy to investigate patellofemoral function during dynamic activity. Comput Methods Biomech Biomed Engin 11(1):41–53CrossRefGoogle Scholar
  19. Gill TJ, Van de Velde SK, Wing DW, Oh LS, Hosseini A, Li G (2009) Tibiofemoral and Patellofemoral kinematics following reconstruction of an isolated posterior cruciate ligament injury: in vivo analysis during lunge. Am J Sports Med 37(12):2388–2385CrossRefGoogle Scholar
  20. Giphart JE, Zirker C, Myers C, Pennington WW, LaPrade RF (2012) Accuracy of a contour-based biplane fluoroscopy technique for tracking knee joint kinematics of different speeds. J Biomech 45:2935–2938CrossRefGoogle Scholar
  21. Gower JC (1975) Generalized procrustes analysis. Psychometrika 40(1):33–51MathSciNetCrossRefMATHGoogle Scholar
  22. Goyal K, Tashman S, Wang JH, Li K, Zhang X, Harner C (2012) In vivo analysis of the isolated posterior cruciate ligament-deficient knee during functional activities. Am J Sports Med 40(4):777–785CrossRefGoogle Scholar
  23. Haque MA, Anderst W, Tashman S, Mari GE (2013) Hierarchical model-based tracking of cervical vertebrae from dynamic biplane radiographs. Med Eng Phys 35(7):994–1004CrossRefGoogle Scholar
  24. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference, vol 15, p 50Google Scholar
  25. Higginson JS, Neptune RR, Anderson FC (2005) Simulated parallel annealing within a neighborhood for optimization of biomechanical systems. J Biomech 38:1938–1942CrossRefGoogle Scholar
  26. Hoshino Y, Fu FH, Irrgang JJ, Tashman S (2013) Can joint contact dynamics be restored by anterior cruciate ligament reconstruction? Clin Orthop Relat Res 471(9):2924–2931CrossRefGoogle Scholar
  27. Ingber L (2012) Adaptive simulated annealing. In: Oliveira H, Petraglia A, Ingber L, Machado M, Petraglia M (eds) Stochastic global optimization and its applications with fuzzy adaptive simulated annealing. Springer, New York, pp 33–61Google Scholar
  28. KaewTraKulPong P, Bowden R (2002) An improved adaptive background mixture model for real-time tracking with shadow detection. In: Video-based surveillance systems. Springer, New York, pp 135–144CrossRefGoogle Scholar
  29. Kapron AL, Aoki SK, Peters CL, Maas SA, Bey MJ, Zauel R, Andersen A (2014) Accuracy and feasibility of dual fluoroscopy and model-based tracking to quantify in vivo hip kinematics during clinical exams. J Appl Biomech 30(3):461–470CrossRefGoogle Scholar
  30. Keskin C, Kıraç F, Kara YE, Akarun L (2013) Real time hand pose estimation using depth sensors. In: Consumer depth cameras for computer vision. Springer, London, pp 119–137CrossRefGoogle Scholar
  31. Kutulakos KN, Seitz SM (2000) A theory of shape by space carving. Int J Comput Vis 38(3):199–218CrossRefMATHGoogle Scholar
  32. Li W, Wang S, Xia Q, Passias P, Kozanek M, Wood K (2011) Lumbar facet joint motion in patients with degenerative disc disease at affected and adjacent levels: an in vivo biomechanical study. Spine 36(10):629–637CrossRefGoogle Scholar
  33. Loper M, Mahmood N, Black MJ (2014) MoSh: motion and shape capture from sparse markers. ACM Trans Graph 33(6):220:1–220:13CrossRefGoogle Scholar
  34. Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3D surface construction algorithm. In: ACM siggraph computer graphics, vol 21, no 4. ACM, pp 163–169Google Scholar
  35. Lourakis MI, Argyros AA (2009) SBA: a software package for generic sparse bundle adjustment. ACM Trans Mathemat Software 36(1):2MathSciNetMATHGoogle Scholar
  36. Lowe DG (1999) Object recognition from local scale-invariant features. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference on, vol 2, IEEE, pp 1150–1157Google Scholar
  37. Marsh C, Martin DE, Harner C, Tashman S (2014) Effect of posterior horn medial meniscus root tear on in vivo knee kinematics. Orthop J Sports Med 2(7):1–7Google Scholar
  38. Martin DE, Greco NJ, Klatt BA, Wright VJ, Anderst WJ, Tashman S (2011) Model-based tracking of the hip: implications for novel analyses of hip pathology. J Arthroplast 26(1):88–97CrossRefGoogle Scholar
  39. Massimini DF, Li G, Warner JP (2010) Glenohumeral contact kinematics in patients after total shoulder arthroplasty. J Bone Joint Surg Am 92(4):916–926CrossRefGoogle Scholar
  40. Nakamura T, Matsumoto J, Nishimaru H, Bretas RV, Takamura Y, Hori E, Ono T, Nishijo H (2016) A markerless 3D computerized motion capture system incorporating a skeleton model for monkeys. PLoS One 11(11):e0166154CrossRefGoogle Scholar
  41. Ohnishi T, Suzuki M, Nawata A, Naomoto S, Iwasaki T, Haneishi H (2010) Three-dimensional motion study of femur, tibia, and patella at the knee joint from bi-plane fluoroscopy and CT images. Radiol Phys Technol 3:151–158CrossRefGoogle Scholar
  42. Piccardi M (2004) Background subtraction techniques: a review. In: Systems, man and cybernetics, 2004 I.E. international conference on, vol 4, IEEE, pp 3099–3104Google Scholar
  43. Marcard T, Pons-Moll G, Rosenhahn B (2016) Human pose estimation efrom video and IMUs. Trans Patt Anal Mach Intellig 38:1533–1547CrossRefGoogle Scholar
  44. Pons-Moll G, Romero J, Mahmood N, Black M (2015) Dyna: a model of dynamic human shape in motion. ACM Trans Graph 34(4):120:1–120:14CrossRefGoogle Scholar
  45. Rosten E, Drummond T (2006, May) Machine learning for high-speed corner detection. In: European conference on computer vision. Springer, Berlin/Heidelberg, pp 430–443Google Scholar
  46. Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: An efficient alternative to SIFT or SURF. In: 2011 international conference on computer vision, IEEE, pp 2564–2571Google Scholar
  47. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D. and Szeliski, R., 2006, June. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: 2006 I.E. computer society conference on computer vision and pattern recognition (CVPR’06), vol 1, IEEE, pp 519–528Google Scholar
  48. Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56(1):116–124CrossRefGoogle Scholar
  49. Siddon RL (1985) Fast calculation of the exact radiological path for a three-dimensional CT array. Med Phy 12:252–255CrossRefGoogle Scholar
  50. Sigal L, Balan AO, Black MJ (2010) Humaneva: synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int J Comput Vis 87(1–2):4–27CrossRefGoogle Scholar
  51. Stoll C, Hasler N, Gall J, Seidel HP, Theobalt C (2011) Fast articulated motion tracking using a sums of gaussians body model. In: 2011 international conference on computer vision, IEEE, pp 951–958Google Scholar
  52. Tashman S, Collon D, Anderson K, Kolowich P, Anderst W (2004) Abnormal rotational knee motion during running after anterior cruciate ligament reconstruction. Am J Sports Med 32(4):975–983CrossRefGoogle Scholar
  53. Tashman S, Princehorn J, Penatto S, Andherst W (2017) Intelligent algorithms for tracking three-dimensional skeletal movement from radiographic image sequences. US patent # 9538940 B2Google Scholar
  54. Toshev A, Szegedy C (2014) Deeppose: human pose estimation via deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1653–1660Google Scholar
  55. Triggs B, McLauchlan PF, Hartley RI, Fitzgibbon AW (1999) Bundle adjustment – a modern synthesis. In: International workshop on vision algorithms. Springer, Berlin/Heidelberg, pp 298–372Google Scholar
  56. Van de Velde SK, Gill TJ, Li G (2009) Evaluation of kinematics of anterior cruciate ligament-deficient knees with use of advanced imaging techniques, three-dimensional modeling techniques, and robotics. J Bone Joint Surg Am 91(Suppl 1):108–114CrossRefGoogle Scholar
  57. Wandt B, Ackermann H, Rosenhahn B (2016) 3d reconstruction of human motion from monocular image sequences. Trans Pattern Analy Mach Intellig 38(8):1505–1516CrossRefGoogle Scholar
  58. Wang C, Wang Y, Lin Z, Yuille A, Gao W (2014). Robust estimation of 3d human poses from a single image. In: Conference on computer vision and pattern recognition (CVPR)Google Scholar
  59. Weiss A, Hirshberg D, Blanc MJ (2011) Home 3D body scans from noisy image and range data. In: ICCV ’11 proceedings of the 2011 international conference on computer vision, pp 1951–1958Google Scholar
  60. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1912–1920Google Scholar
  61. Zhao H, Reader AJ (2003). Fast ray-tracing technique to calculate line integral paths in voxel arrays. In: Proceedings of the IEEE nuclear science symposium and medical imaging conference, pp 2808–2812Google Scholar
  62. Zivkovic Z (2004) Improved adaptive Gaussian mixture model for background subtraction. In: Pattern recognition, 2004. ICPR 2004. Proceedings of the 17th international conference on, vol 2, IEEE, pp 28–31Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.KinaTrax Inc.Palm BeachUSA
  2. 2.HAS-Motion IncKingstonCanada
  3. 3.C-Motion Inc.GermantownUSA

Section editors and affiliations

  • William Scott Selbie
    • 1
  1. 1.Has-Motion Inc.KingstonCanada

Personalised recommendations