Skip to main content

Advertisement

SpringerLink
Log in

Real-Time Avatar Pose Transfer and Motion Generation Using Locally Encoded Laplacian Offsets

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

We propose a human avatar representation scheme based on intrinsic coordinates, which are invariant to isometry and insensitive to human pose changes, and an efficient pose transfer algorithm that can utilize this representation to reconstruct a human body geometry following a given pose. Such a pose transfer algorithm can be used to control the movement of an avatar model in virtual reality environments following a user’s motion in real time. Our proposed algorithm consists of three main steps. First, we recognize the user’s pose and select a template model from the database who has a similar pose; then, the intrinsic Laplacian offsets encoded in local coordinates are used to reconstruct the human body geometry following the template pose; finally, the morphing between the two poses is generated using a linear interpolation. We perform experiments to evaluate the accuracy and efficiency of our algorithm. We believe our proposed system is a promising human modeling tool that can be used in general virtual reality applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Robitaille N, Jackson P L, Hébert L J, Mercier C et al. A virtual reality avatar interaction (VRAI) platform to assess residual executive dysfunction in active military personnel with previous mild traumatic brain injury: Proof of concept. Disability and Rehabilitation: Assistive Technology, 2017, 12(7): 758-764.

    Google Scholar 

  2. Lifkooee M Z, Liu C L, Li M Q, Li X. Image-based human character modeling and reconstruction for virtual reality exposure therapy. In Proc. the 13th International Conference on Computer Science & Education, Aug. 2018, Article No. 149.

  3. Urella N, Hughes J, Conrad E, Zhang J S, Li X. A VR scene modelling platform for PTSD treatment. In Proc. the 12th International Conference on Computer Science and Education, Aug. 2017, pp.257-262.

  4. Pishchulin L, Wuhrer S, Helten T, Theobalt C, Schiele B. Building statistical shape spaces for 3D human modeling. Pattern Recognition, 2017, 67: 276-286.

    Article  Google Scholar 

  5. Orts-Escolano S, Rhemann C, Fanello S et al. Holoportation: Virtual 3D teleportation in real-time. In Proc. the 29th Annual Symposium on User Interface Software and Technology, Oct. 2016, pp.741-754.

  6. Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Rössl C, Seidel H P. Laplacian surface editing. In Proc. the 2nd Eurographics Symposium on Geometry processing, Jul. 2004, pp.175-184.

  7. Shi L, Yu Y Z, Bell N, Feng W W. A fast multigrid algorithm for mesh deformation. ACM Trans. Graph., 2006, 25(3): 1108-1117.

    Article  Google Scholar 

  8. Huang J, Shi X H, Liu X U, Zhou K, Wei L Y, Teng S H, Bao H J, Guo B N, Shum H Y. Subspace gradient domain mesh deformation. In Proc. ACM SIGGRAPH 2006 Papers, Jul. 2006, pp.1126-1134.

  9. Le B H, Hodgins J K. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics, 2016, 35(4): Article No. 37.

  10. De Aguiar E, Theobalt C, Thrun S, Seidel H P. Automatic conversion of mesh animations into skeleton-based animations. Computer Graphics Forum, 2008, 27(2): 389-397.

    Article  Google Scholar 

  11. Kim M, Pons-Moll G, Pujades S, Bang S, Kim J, Black M J, Lee S H. Data-driven physics for human soft tissue animation. ACM Transactions on Graphics, 2017, 36(4): Article No. 54.

  12. Shi X H, Zhou K, Tong Y Y, Desbrun M, Bao H J, Guo B N. Mesh puppetry: Cascading optimization of mesh deformation with inverse kinematics. ACM Trans. Graph., 2007, 26(3): Article No. 81.

  13. Yasin H, Iqbal U, Kruger B, Weber A, Gall J. A dual-source approach for 3D pose estimation from a single image. In Proc. the 2006 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.4948-4956.

  14. Liu Z, Zhu J K, Bu J J, Chen C. A survey of human pose estimation: The body parts parsing based methods. Journal of Visual Communication and Image Representation, 2015, 32: 10-19.

    Article  Google Scholar 

  15. Martinez J, Hossain R, Romero J, Little J J. A simple yet effective baseline for 3D human pose estimation. In Proc. the 2007 IEEE International Conference on Computer Vision, Oct. 2017, pp.2659-2668.

  16. Moreno-Noguer F. 3D human pose estimation from a single image via distance matrix regression. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1561-1570.

  17. Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H P, Xu W P, Casas D, Theobalt C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, 2017, 36(4): Article No. 44.

  18. Remondino F. 3-D reconstruction of static human body shape from image sequence. Computer Vision and Image Understanding, 2004, 93(1): 65-85.

    Article  Google Scholar 

  19. Seo H, Yeo Y I, Wohn K. 3D body reconstruction from photos based on range scan. In Proc. the 1st International Conference on Technologies for E-Learning and Digital Entertainment, Apr. 2006, pp.849-860.

  20. Cheng K L, Tong R F, Tang M, Qian J Y, Sarkis M. Parametric human body reconstruction based on sparse key points. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2467-2479.

    Article  Google Scholar 

  21. Lifkooee M Z, Soysal ÖM, Sekeroglu K. Video mining for facial action unit classification using statistical spatial-temporal feature image and LoG deep convolutional neural network. Machine Vision and Applications, 2018. https://doi.org/10.1007/s00138-018-0967-2, Jan. 2019.

  22. Tagliasacchi A, Delame T, Spagnuolo M, Amenta N, Telea A. 3D skeletons: A state-of-the-art report. Comput. Graph. Forum, 2016, 35(2): 573-597.

    Article  Google Scholar 

  23. Saha P K, Borgefors G, di Baja G S. A survey on skeletonization algorithms and their applications. Pattern Recognition Letters, 2016, 76(1): 3-12.

    Article  Google Scholar 

  24. Cornea N D, Silver D, Min P. Curve-skeleton properties, applications, and algorithms. IEEE Transactions on Visualization and Computer Graphics, 2007, 13(3): 530-548.

    Article  Google Scholar 

  25. Tagliasacchi A, Zhang H, Cohen-Or D. Curve skeleton extraction from incomplete point cloud. ACM Trans. Graph., 2009, 28(3): Article No. 71.

  26. Zheng Q, Sharf A, Tagliasacchi A, Chen B Q, Zhang H, Sheffer A, Cohen-Or D. Consensus skeleton for non-rigid space-time registration. Computer Graphics Forum, 2010, 29(2): 635-644.

    Article  Google Scholar 

  27. James D L, Twigg C D. Skinning mesh animations. ACM Transactions on Graphics, 2005, 24(3): 399-407.

    Article  Google Scholar 

  28. Colaianni M, Zöllhoefer M, Süßmuth J, Seider B, Greiner G. A pose invariant statistical shape model for human bodies. In Proc. the 5th International Conference on 3D Body Scanning Technologies, Oct. 2014, pp.327-336.

  29. Wuhrer S, Shu C, Xi P C. Posture-invariant statistical shape analysis using Laplace operator. Computers & Graphics, 2012, 36(5): 410-416.

    Article  Google Scholar 

  30. Davis T A, Hager W W. Dynamic supernodes in sparse cholesky update/downdate and triangular solves. ACM Trans. Math. Software, 2009, 35(4): Article No. 27.

  31. Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.

  32. Xu Z, Zhang Q, Cheng S. Multilevel active registration for Kinect human body scans: From low quality to high quality. Multimedia Systems, 2018, 24(3): 257-270.

    Article  Google Scholar 

  33. Li X, Iyengar S. On computing mapping of 3D objects: A survey. ACM Computing Surveys, 2015, 47(2): Article No. 34.

  34. van Kaick O, Zhang H, Hamarneh G, Cohen-Or D. A survey on shape correspondence. Computer Graphics Forum, 2011, 30(6): 1681-1707.

    Article  Google Scholar 

  35. Loper M, Mahmood N, Romero J, Pons-Moll G, Black M J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 2015, 34(6): Article No. 248.

  36. Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black M J. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.561-578.

  37. Alexa M. Recent advances in mesh morphing. Computer Graphics Forum, 2002, 21(2): 173-198.

    Article  Google Scholar 

  38. Liu Z G, Zhou L Y, Leung H, Multon F, Shum H P. High quality compatible triangulations for planar shape animation. In Proc. the 2017 ACM SIGGRAPH ASIA Workshop on Data-Driven Animation Techniques, Nov. 2017, pp.1-8.

Download references

Acknowledgements

We thank anonymous reviewers for their constructive comments.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Louisiana State University, Baton Rouge, 70803, U.S.A.

    Masoud Zadghorban Lifkooee, Celong Liu, Yongqing Liang & Xin Li

  2. Department of Construction Management, Louisiana State University, Baton Rouge, 70803, U.S.A.

    Yimin Zhu

Authors
  1. Masoud Zadghorban Lifkooee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Celong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongqing Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yimin Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Li.

Electronic supplementary material

ESM 1

(PDF 419 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lifkooee, M.Z., Liu, C., Liang, Y. et al. Real-Time Avatar Pose Transfer and Motion Generation Using Locally Encoded Laplacian Offsets. J. Comput. Sci. Technol. 34, 256–271 (2019). https://doi.org/10.1007/s11390-019-1909-9

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-019-1909-9

Keywords

Search

Navigation