Skip to main content
Log in

A Survey on Human Performance Capture and Animation

  • Survey
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

With the rapid development of computing technology, three-dimensional (3D) human body models and their dynamic motions are widely used in the digital entertainment industry. Human performance mainly involves human body shapes and motions. Key research problems in human performance animation include how to capture and analyze static geometric appearance and dynamic movement of human bodies, and how to simulate human body motions with physical effects. In this survey, according to the main research directions of human body performance capture and animation, we summarize recent advances in key research topics, namely human body surface reconstruction, motion capture and synthesis, as well as physics-based motion simulation, and further discuss future research problems and directions. We hope this will be helpful for readers to have a comprehensive understanding of human performance capture and animation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Robinette K M, Daanen H, Paquet E. The CAESAR project: A 3D surface anthropometry survey. In Proc. the 2nd International Conference on 3-D Digital Imaging and Modeling, Oct. 1999, pp.380-386.

  2. Wang C C, Chang T K, Yuen M M. From laser-scanned data to feature human model: A system based on fuzzy logic concept. Computer-Aided Design, 2003, 35(3): 241-253.

    Article  Google Scholar 

  3. Woodham R J. Shape from shading. In Shape from Shading, Horm B K P, Brooks M J (eds.), Cambridge, USA: MIT Press, 1989, pp.513-531.

  4. Vlasic D, Peers P, Baran I, Debevec P, Popović J, Rusinkiewicz S, Matusik W. Dynamic shape capture using multi-view photometric stereo. ACM Transactions on Graphics, 2009, 28(5): Article No. 174.

  5. Wu C, Varanasi K, Liu Y, Seidel H P, Theobalt C. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. the IEEE International Conference on Computer Vision, Nov. 2011, pp.1108-1115.

  6. Stoll C, Gall J, de Aguiar E, Thrun S, Theobalt C. Video-based reconstruction of animatable human characters. ACM Transactions on Graphics, 2010, 29(6): Article No. 139.

  7. Zhu H, Liu Y, Fan J, Dai Q, Cao X. Video-based outdoor human reconstruction. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 27(4): 760-770.

    Article  Google Scholar 

  8. Newcombe R A, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison A J, Kohi P, Shotton J, Hodges S, Fitzgibbon A. KinectFusion: Real-time dense surface mapping and tracking. In Proc. IEEE International Symposium on Mixed and Augmented Reality, Oct. 2011, pp.127-136.

  9. Li H, Vouga E, Gudym A, Luo L, Barron J T, Gusev G. 3D self-portraits. ACM Transactions on Graphics, 2013, 32(6): Article No. 187.

  10. Newcombe R A, Fox D, Seitz S M. DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.343-352.

  11. Dou M, Taylor J, Fuchs H, Fitzgibbon A, Izadi S. 3D scanning deformable objects with a single RGB-D sensor. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.493-501.

  12. Butler D A, Izadi S, Hilliges O, Molyneaux D, Hodges S, Kim D. Shake‘n’sense: Reducing interference for overlapping structured light depth cameras. In Proc. the ACM Annual Conference on Human Factors in Computing Systems, May 2012, pp.1933-1936.

  13. Tong J, Zhou J, Liu L, Pan Z, Yan H. Scanning 3D full human bodies using kinects. IEEE Transactions on Visualization and Computer Graphics, 2012, 18(4): 643-650.

    Article  Google Scholar 

  14. Lin S, Chen Y, Lai Y K, Martin R R, Cheng Z Q. Fast capture of textured fullbody avatar with RGB-D cameras. The Visual Computer, 2016, 32(6/7/8): 681-691.

  15. Ye G, Liu Y, Hasler N, Ji X, Dai Q, Theobalt C. Performance capture of interacting characters with handheld kinects. In Proc. the 12th European Conference on Computer Vision, Volume Part II, Oct. 2012, pp.828-841.

  16. Wang C, Liu Y, Guo X, Zhong Z, Le B, Deng Z. Spectral animation compression. Journal of Computer Science and Technology, 2015, 30(3): 540-552.

    Article  Google Scholar 

  17. Anguelov D, Srinivasan P, Koller D, Thrun S, Rodgers J, Davis J. SCAPE: Shape completion and animation of people. ACM Transactions on Graphics, 2005, 24(3): 408-416.

    Article  Google Scholar 

  18. Weiss A, Hirshberg D, Black M J. Home 3D body scans from noisy image and range data. In Proc. the IEEE International Conference on Computer Vision, Nov. 2011, pp. 1951-1958.

  19. Bogo F, Black M J, Loper M, Romero J. Detailed full-body reconstructions of moving people from monocular RGB-D sequences. In Proc. the IEEE International Conference on Computer Vision, Dec. 2015, pp.2300-2308.

  20. Chen Y, Liu Z, Zhang Z. Tensor-based human body modeling. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2013, pp.105-112.

  21. Hasler N, Stoll C, Sunkel M, Rosenhahn B, Seidel H P. A statistical model of human pose and body shape. Computer Graphics Forum, 2009, 28(2): 337-346.

    Article  Google Scholar 

  22. Cheng K L, Tong R F, Tang M, Qian J Y, Sarkis M. Parametric human body reconstruction based on sparse key points. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2467-2479.

    Article  Google Scholar 

  23. Zeng M, Zheng J, Cheng X, Liu X. Templateless quasi-rigid shape modeling with implicit loop-closure. In Proc. the Conference on Computer Vision and Pattern Recognition, Jun. 2013, pp.145-152.

  24. Chen Y, Cheng Z Q, Lai C, Martin R R, Dang G. Real-time reconstruction of an animating human body from a single depth camera. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(8): 2000-2011.

    Article  Google Scholar 

  25. Guan P, Reiss L, Hirshberg D A, Weiss A, Black M J. DRAPE: DRessing any PErson. ACM Transactions on Graphics, 2012, 31(4): Article No. 35.

  26. Tsoli A, Mahmood N, Black M J. Breathing life into shape: Capturing, modeling and animating 3D human breathing. ACM Transactions on Graphics, 2014, 33(4): Article No. 52.

  27. Zheng J, Zeng M, Cheng X, Liu X. Scape-based human performance reconstruction. Computers & Graphics, 2014, 38: 191-198.

    Article  Google Scholar 

  28. Ye M, Wang H, Deng N, Yang X, Yang R. Real-time human pose and shape estimation for virtual try-on using a single commodity depth camera. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(4): 550-559.

    Google Scholar 

  29. Pons-Moll G, Romero J, Mahmood N, Black M J. Dyna: A model of dynamic human shape in motion. ACM Transactions on Graphics, 2015, 34(4): Article No. 120.

  30. Zuffi S, Black M J. The stitched puppet: A graphical model of 3D human shape and pose. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2015.

  31. Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.

  32. Bogo F, Romero J, Pons-Moll G, Black M. Dynamic faust: Registering human bodies in motion. In Proc. the Conference on Computer Vision and Pattern Recognition, July 2017.

  33. Brigante C, Abbate N, Basile A, Faulisi A, Sessa S. Towards miniaturization of a mems-based wearable motion capture system. IEEE Transactions on Industrial Electronics, 2011, 58(8): 3234-3241.

    Article  Google Scholar 

  34. Andrews S, Huerta I, Komura T, Sigal L, Mitchell K. Real-time physics-based motion capture with sparse sensors. In Proc. the 13th European Conference on Visual Media Production, Dec. 2016.

  35. Hou J, Chau L P, Magnenat-Thalmann N, He Y. Human motion capture data tailored transform coding. IEEE Trans actions on Visualization and Computer Graphics, 2015, 21(7): 848-859.

    Article  Google Scholar 

  36. Vlasic D, Baran I, Matusik W, Popović J. Articulated mesh animation from multiview silhouettes. ACM Transactions on Graphics, 2008, 27(3): Article No. 97.

  37. Liu Y, Stoll C, Gall J, Seidel H P, Theobalt C. Marker-less motion capture of interacting characters using multi-view image segmentation. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2011, pp.1249-1256.

  38. Hasler N, Rosenhahn B, Thormahlen T, Wand M, Gall J, Seidel H P. Markerless motion capture with unsynchronized moving cameras. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, pp.224-231.

  39. Shiratori T, Park H S, Sigal L, Sheikh Y, Hodgins J K. Motion capture from bodymounted cameras. ACM Transactions on Graphics, 2011, 30(4): Article No. 31.

  40. Elhayek A, Aguiar E, Jain A, Tompson J, Pishchulin L, Andriluka M, Bregler C, Schiele B, Theobalt C. Efficient convnetbased marker-less motion capture in general scenes with a low number of cameras. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2015, pp.3810-3818.

  41. Ionescu C, Papava D, Olaru V, Sminchisescu C. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1325-1339.

    Article  Google Scholar 

  42. Sigal L, Balan A O, Black M J. HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision, 2010, 87: 4-27.

  43. Dantone M, Gall J, Leistner C, van Gool L. Body parts dependent joint regressors for human pose estimation in still images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(11): 2131-2143.

    Article  Google Scholar 

  44. Wei S E, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp. 4724-4732.

  45. Wei X, Chai J. Videomocap: Modeling physically realistic human motion from monocular video sequences. ACM Transactions on Graphics, 2010, 29(4): Article No. 42.

  46. Insafutdinov E, Pishchulin L, Andres B, Andriluka M, Schiele B. DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. In Proc. the European Conference on Computer Vision, Oct. 2016, pp.34-50.

  47. Andriluka M, Pishchulin L, Gehler P, Schiele B. 2D human pose estimation: New benchmark and state of the art analysis. In Proc. the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3686-3693.

  48. Baak A, Müller M, Bharaj G, Seidel H P, Theobalt C. A data-driven approach for real-time full body pose reconstruction from a depth camera. In Consumer Depth Cameras for Computer Vision, Fossati A, Gall J, Grabrier H et al. (eds.), Springer, 2013, pp.71-98.

  49. Ye M, Wang X, Yang R, Ren L, Pollefeys M. Accurate 3D pose estimation from a single depth image. In Proc. the International Conference on Computer Vision, Nov. 2011, pp. 731-738.

  50. Liu Z, Zhou L, Leung H, Shum H P. Kinect posture reconstruction based on a local mixture of gaussian process models. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2437-2450.

    Article  Google Scholar 

  51. Wei X, Zhang P, Chai J. Accurate realtime full-body motion capture using a single depth camera. ACM Transactions on Graphics, 2012, 31(6): Article No. 188.

  52. Zhang P, Siu K, Zhang J, Liu C K, Chai J. Leveraging depth cameras and wearable pressure sensors for fullbody kinematics and dynamics capture. ACM Transactions on Graphics, 2014, 33(6): Article No. 221.

  53. von Marcard T, Ponsmoll G, Rosenhahn B. Human pose estimation from video and IMUs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(8): 1533-1547.

    Article  Google Scholar 

  54. Arikan O, Forsyth D. Interactive motion generation from examples. ACM Transactions on Graphics, 2002, 21(3): 483-490.

    Article  MATH  Google Scholar 

  55. Kovar L, Gleicher M, Pighin F. Motion graphs. ACM Transactions on Graphics, 2002, 21(3): 473-482.

    Article  Google Scholar 

  56. Ikemoto L, Forsyth D A. Enriching a motion collection by transplanting limbs. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aug. 2004, pp. 99-108.

  57. Gleicher M. Motion path editing. In Proc. the Symposium on Interactive 3D Graphics, Mar. 2001, pp.195-202.

  58. Huang Y, Kallmann M. Planning motions and placements for virtual demonstrators. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(5): 1568-1579.

    Article  Google Scholar 

  59. Kim Y, Park H, Bang S, Lee S H. Retargeting human-object interaction to virtual avatars. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2405-2412.

    Article  Google Scholar 

  60. Mukai T, Kuriyama S. Geostatistical motion interpolation. ACM Transactions on Graphics, 2005, 24(3): 1062-1070.

    Article  Google Scholar 

  61. Kovar L, Gleicher M. Automated extraction and parameterization of motions in large data sets. ACM Transactions on Graphics, 2004, 23(3): 559-568.

    Article  Google Scholar 

  62. Wang H, Ho E S, Komura T. An energydriven motion planning method for two distant postures. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(1): 18-30.

    Article  Google Scholar 

  63. Tanco L M, Hilton A. Realistic synthesis of novel human movements from a database of motion capture examples. In Proc. Workshop on Human Motion, Dec. 2000, pp.137-142.

  64. Pullen K, Bregler C. Animating by multilevel sampling. In Proc. Computer Animation, May 2000, pp.36-42.

  65. Hsu E, Pulli K, Popović J. Style translation for human motion. ACM Transactions on Graphics, 2005, 24(3): 1082-1089.

    Article  Google Scholar 

  66. Chai J, Hodgins J K. Constraint-based motion optimization using a statistical dynamic model. ACM Transactions on Graphics, 2007, 26(3): Article No. 8.

  67. Lau M, Bar-Joseph Z, Kuffner J. Modeling spatial and temporal variation in motion data. ACM Transactions on Graphics, 2009, 28(5): Article No. 171.

  68. Min J, Chai J. Motion graphs++: A compact generative model for semantic motion analysis and synthesis. ACM Transactions on Graphics, 2012, 31(6): Article No. 153.

  69. Holden D, Saito J, Komura T, Joyce T. Learning motion manifolds with convolutional autoencoders. In Proc. SIG-GRAPH Asia 2015 Technical Briefs, Nov. 2015.

  70. Holden D, Saito J, Komura T. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics, 2016, 35(4): Article No. 138.

  71. Ikemoto L, Arikan O, Forsyth D. Generalizing motion edits with Gaussian processes. ACM Transactions on Graphics, 2009, 28(1): Article No. 1.

  72. Xia S, Wang C, Chai J, Hodgins J K. Realtime style transfer for unlabeled heterogeneous human motion. ACM Transactions on Graphics, 2015, 34(4): Article No. 119.

  73. Brand M, Hertzmann A. Style machines. In Proc. the 27th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 2000, pp.183-192.

  74. Wang J M, Fleet D J, Hertzmann A. Multifactor Gaussian process models for style-content separation. In Proc. the 24th International Conference on Machine Learning, Jun. 2007, pp.975-982.

  75. Min J, Liu H, Chai J. Synthesis and editing of personalized stylistic human motion. In Proc. the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Feb. 2010, pp.39-46.

  76. Ma W, Xia S, Hodgins J K, Yang X, Li C, Wang Z. Modeling style and variation in human motion. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.21-30.

  77. Drumwright E. A fast and stable penalty method for rigid body simulation. IEEE Transactions on Visualization and Computer Graphics, 2008, 14(1): 231-240.

    Article  Google Scholar 

  78. Wieber P B. On the stability of walking systems. In Proc. the International Workshop on Humanoid and Human Friendly Robotics, Dec. 2002.

  79. Yin K, Loken K, Panne M. SIMBICON: Simple biped locomotion control. ACM Transactions on Graphics, 2007, 26(3): Article No. 105.

  80. Coros S, Beaudoin P, Panne M. Generalized biped walking control. ACM Transactions on Graphics, 2010, 29(4): Article No. 130.

  81. Wang J M, Fleet D J, Hertzmann A. Optimizing walking controllers. ACM Transactions on Graphics, 2009, 28(5): Article No. 168.

  82. Wang J, Hamner S R, Delp S L, Koltun V. Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Transactions on Graphics, 2012, 31(4): Article No. 25.

  83. Liu L, Yin K, de Panne M V, Guo B. Terrain runner: Control, parameterization, composition, and planning for highly dynamic motions. ACM Transactions on Graphics, 2012, 31(6): Article No. 154.

  84. Liu L, de Panne M V, Yin K. Guided learning of control graphs for physics-based characters. ACM Transactions on Graphics, 2016, 35(3): Article No. 29.

  85. Zordan V B, Hodgins J K. Motion capture-driven simulations that hit and react. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2002, pp.89-96.

  86. Zordan V B, Majkowska A, Chiu B, Fast M. Dynamic response for motion capture animation. ACM Transactions on Graphics, 2005, 24(3): 697-701.

    Article  Google Scholar 

  87. Sharon D, Panne M. Synthesis of controllers for stylized planar bipedal walking. In Proc. the IEEE International Conference on Robotics and Automation, Apr. 2005, pp. 2387-2392.

  88. Sok K W, Kim M, Lee J. Simulating biped behaviors from human motion data. ACM Transactions on Graphics, 2007, 26(3): Article No. 107.

  89. Silva M, Abe Y, Popović J. Interactive simulation of stylized human locomotion. ACM Transactions on Graphics, 2008, 27(3): Article No. 82.

  90. Geijtenbeek T, Pronost N, Stappen A F. Simple data-driven control for simulated bipeds. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2012, pp.211-219.

  91. Hamalainen P, Rajamaki J, Liu C K. Online control of simulated humanoids using particle belief propagation. ACM Transactions on Graphics, 2015, 34(4): Article No. 81.

  92. Agrawal S, de Panne M V. Taskbased locomotion. ACM Transactions on Graphics, 2016, 35(4): Article No. 82.

  93. Yeadon M R. The simulation of aerial movement-II. A mathematical inertia model of the human body. Journal of Biomechanics, 1990, 23(1): 67-74.

    Article  Google Scholar 

  94. Sheets A, Abrams G D, Corazza S, Safran M R, Andriacchi T P. Kinematics differences between the flat, kick, and slice serves measured using a markerless motion capture method. Annals of Biomedical Engineering, 2011, 39(12): 3011-3020.

    Article  Google Scholar 

  95. Lv X, Chai J, Xia S. Data-driven inverse dynamics for human motion. ACM Transactions on Graphics, 2016, 35(6): Article No. 163.

  96. Witkin A, Kass M. Spacetime constraints. ACM SIGGRAPH Computer Graphics, 1988, 22(4): 159-168.

    Article  Google Scholar 

  97. Liu C K, Hertzmann A, Popović Z. Composition of complex optimal multicharacter motions. In Proc. the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Sept. 2006, pp.215-222.

  98. Borno M A, de Lasa M, Hertzmann A. Trajectory optimization for full-body movements with complex contacts. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(8): 1405-1414.

    Article  Google Scholar 

  99. Park C, Park J S, Tonneau S, Mansard N, Multon F, Pettre J, Manocha D. Dynamically balanced and plausible trajectory planning for human-like characters. In Proc. the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Feb., 2016, pp.39-48.

  100. Geijtenbeek T, Pronost N. Interactive character animation using simulated physics: A state-of-the-art review. Computer Graphics Forum, 2012, 31(8): 2492-2515.

    Article  Google Scholar 

  101. Abe Y, da Silva M, Popović J. Multiobjective control with frictional contacts. In Proc. the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aug. 2007, pp.249-258.

  102. Wu C C, Zordan V. Goal-directed stepping with momentum control. In Proc. the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.113-118.

  103. de Lasa M, Mordatch I, Hertzmann A. Feature-based locomotion controllers. ACM Transactions on Graphics, 2010, 29(4): Article No. 131.

  104. Muico U, Lee Y, Popović J, Popović Z. Contact-aware nonlinear control of dynamic characters. ACM Transactions on Graphics, 2009, 28(3): Article No. 81.

  105. Muico U, Popović J, Popović Z. Composite control of physically simulated characters. ACM Transactions on Graphics, 2011, 30(3): Article No. 16.

  106. Wu J C, Popović Z. Terrain-adaptive bipedal locomotion control. ACM Transactions on Graphics, 2010, 29(4): Article No. 72.

  107. Kwon T, Hodgins J. Control systems for human running using an inverted pendulum model and a reference motion capture sequence. In Proc. the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Jul. 2010, pp.129-138.

  108. Mordatch I, de Lasa M, Hertzmann A. Robust physics-based locomotion using low-dimensional planning. ACM Transactions on Graphics, 2010, 29(4): Article No. 71.

  109. Han D, Noh J, Jin X, Shin J S, Shin S Y. On-line real-time physics-based predictive motion control with balance recovery. Computer Graphics Forum, 2014, 33(2): 245-254.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lin Gao.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 177 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, S., Gao, L., Lai, YK. et al. A Survey on Human Performance Capture and Animation. J. Comput. Sci. Technol. 32, 536–554 (2017). https://doi.org/10.1007/s11390-017-1742-y

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-017-1742-y

Keywords

Navigation