Advertisement

Journal of Computer Science and Technology

, Volume 34, Issue 2, pp 256–271 | Cite as

Real-Time Avatar Pose Transfer and Motion Generation Using Locally Encoded Laplacian Offsets

  • Masoud Zadghorban Lifkooee
  • Celong Liu
  • Yongqing Liang
  • Yimin Zhu
  • Xin LiEmail author
Regular Paper
  • 20 Downloads

Abstract

We propose a human avatar representation scheme based on intrinsic coordinates, which are invariant to isometry and insensitive to human pose changes, and an efficient pose transfer algorithm that can utilize this representation to reconstruct a human body geometry following a given pose. Such a pose transfer algorithm can be used to control the movement of an avatar model in virtual reality environments following a user’s motion in real time. Our proposed algorithm consists of three main steps. First, we recognize the user’s pose and select a template model from the database who has a similar pose; then, the intrinsic Laplacian offsets encoded in local coordinates are used to reconstruct the human body geometry following the template pose; finally, the morphing between the two poses is generated using a linear interpolation. We perform experiments to evaluate the accuracy and efficiency of our algorithm. We believe our proposed system is a promising human modeling tool that can be used in general virtual reality applications.

Keywords

human body pose transfer local intrinsic coordinates avatar control in virtual reality 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgements

We thank anonymous reviewers for their constructive comments.

Supplementary material

11390_2019_1909_MOESM1_ESM.pdf (420 kb)
ESM 1 (PDF 419 kb)

References

  1. [1]
    Robitaille N, Jackson P L, Hébert L J, Mercier C et al. A virtual reality avatar interaction (VRAI) platform to assess residual executive dysfunction in active military personnel with previous mild traumatic brain injury: Proof of concept. Disability and Rehabilitation: Assistive Technology, 2017, 12(7): 758-764.Google Scholar
  2. [2]
    Lifkooee M Z, Liu C L, Li M Q, Li X. Image-based human character modeling and reconstruction for virtual reality exposure therapy. In Proc. the 13th International Conference on Computer Science & Education, Aug. 2018, Article No. 149.Google Scholar
  3. [3]
    Urella N, Hughes J, Conrad E, Zhang J S, Li X. A VR scene modelling platform for PTSD treatment. In Proc. the 12th International Conference on Computer Science and Education, Aug. 2017, pp.257-262.Google Scholar
  4. [4]
    Pishchulin L, Wuhrer S, Helten T, Theobalt C, Schiele B. Building statistical shape spaces for 3D human modeling. Pattern Recognition, 2017, 67: 276-286.CrossRefGoogle Scholar
  5. [5]
    Orts-Escolano S, Rhemann C, Fanello S et al. Holoportation: Virtual 3D teleportation in real-time. In Proc. the 29th Annual Symposium on User Interface Software and Technology, Oct. 2016, pp.741-754.Google Scholar
  6. [6]
    Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Rössl C, Seidel H P. Laplacian surface editing. In Proc. the 2nd Eurographics Symposium on Geometry processing, Jul. 2004, pp.175-184.Google Scholar
  7. [7]
    Shi L, Yu Y Z, Bell N, Feng W W. A fast multigrid algorithm for mesh deformation. ACM Trans. Graph., 2006, 25(3): 1108-1117.CrossRefGoogle Scholar
  8. [8]
    Huang J, Shi X H, Liu X U, Zhou K, Wei L Y, Teng S H, Bao H J, Guo B N, Shum H Y. Subspace gradient domain mesh deformation. In Proc. ACM SIGGRAPH 2006 Papers, Jul. 2006, pp.1126-1134.Google Scholar
  9. [9]
    Le B H, Hodgins J K. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics, 2016, 35(4): Article No. 37.Google Scholar
  10. [10]
    De Aguiar E, Theobalt C, Thrun S, Seidel H P. Automatic conversion of mesh animations into skeleton-based animations. Computer Graphics Forum, 2008, 27(2): 389-397.CrossRefGoogle Scholar
  11. [11]
    Kim M, Pons-Moll G, Pujades S, Bang S, Kim J, Black M J, Lee S H. Data-driven physics for human soft tissue animation. ACM Transactions on Graphics, 2017, 36(4): Article No. 54.Google Scholar
  12. [12]
    Shi X H, Zhou K, Tong Y Y, Desbrun M, Bao H J, Guo B N. Mesh puppetry: Cascading optimization of mesh deformation with inverse kinematics. ACM Trans. Graph., 2007, 26(3): Article No. 81.Google Scholar
  13. [13]
    Yasin H, Iqbal U, Kruger B, Weber A, Gall J. A dual-source approach for 3D pose estimation from a single image. In Proc. the 2006 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.4948-4956.Google Scholar
  14. [14]
    Liu Z, Zhu J K, Bu J J, Chen C. A survey of human pose estimation: The body parts parsing based methods. Journal of Visual Communication and Image Representation, 2015, 32: 10-19.CrossRefGoogle Scholar
  15. [15]
    Martinez J, Hossain R, Romero J, Little J J. A simple yet effective baseline for 3D human pose estimation. In Proc. the 2007 IEEE International Conference on Computer Vision, Oct. 2017, pp.2659-2668.Google Scholar
  16. [16]
    Moreno-Noguer F. 3D human pose estimation from a single image via distance matrix regression. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1561-1570.Google Scholar
  17. [17]
    Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H P, Xu W P, Casas D, Theobalt C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, 2017, 36(4): Article No. 44.Google Scholar
  18. [18]
    Remondino F. 3-D reconstruction of static human body shape from image sequence. Computer Vision and Image Understanding, 2004, 93(1): 65-85.CrossRefGoogle Scholar
  19. [19]
    Seo H, Yeo Y I, Wohn K. 3D body reconstruction from photos based on range scan. In Proc. the 1st International Conference on Technologies for E-Learning and Digital Entertainment, Apr. 2006, pp.849-860.Google Scholar
  20. [20]
    Cheng K L, Tong R F, Tang M, Qian J Y, Sarkis M. Parametric human body reconstruction based on sparse key points. IEEE Transactions on Visualization and Computer Graphics, 2016, 22(11): 2467-2479.CrossRefGoogle Scholar
  21. [21]
    Lifkooee M Z, Soysal ÖM, Sekeroglu K. Video mining for facial action unit classification using statistical spatial-temporal feature image and LoG deep convolutional neural network. Machine Vision and Applications, 2018.  https://doi.org/10.1007/s00138-018-0967-2, Jan. 2019.
  22. [22]
    Tagliasacchi A, Delame T, Spagnuolo M, Amenta N, Telea A. 3D skeletons: A state-of-the-art report. Comput. Graph. Forum, 2016, 35(2): 573-597.CrossRefGoogle Scholar
  23. [23]
    Saha P K, Borgefors G, di Baja G S. A survey on skeletonization algorithms and their applications. Pattern Recognition Letters, 2016, 76(1): 3-12.CrossRefGoogle Scholar
  24. [24]
    Cornea N D, Silver D, Min P. Curve-skeleton properties, applications, and algorithms. IEEE Transactions on Visualization and Computer Graphics, 2007, 13(3): 530-548.CrossRefGoogle Scholar
  25. [25]
    Tagliasacchi A, Zhang H, Cohen-Or D. Curve skeleton extraction from incomplete point cloud. ACM Trans. Graph., 2009, 28(3): Article No. 71.Google Scholar
  26. [26]
    Zheng Q, Sharf A, Tagliasacchi A, Chen B Q, Zhang H, Sheffer A, Cohen-Or D. Consensus skeleton for non-rigid space-time registration. Computer Graphics Forum, 2010, 29(2): 635-644.CrossRefGoogle Scholar
  27. [27]
    James D L, Twigg C D. Skinning mesh animations. ACM Transactions on Graphics, 2005, 24(3): 399-407.CrossRefGoogle Scholar
  28. [28]
    Colaianni M, Zöllhoefer M, Süßmuth J, Seider B, Greiner G. A pose invariant statistical shape model for human bodies. In Proc. the 5th International Conference on 3D Body Scanning Technologies, Oct. 2014, pp.327-336.Google Scholar
  29. [29]
    Wuhrer S, Shu C, Xi P C. Posture-invariant statistical shape analysis using Laplace operator. Computers & Graphics, 2012, 36(5): 410-416.CrossRefGoogle Scholar
  30. [30]
    Davis T A, Hager W W. Dynamic supernodes in sparse cholesky update/downdate and triangular solves. ACM Trans. Math. Software, 2009, 35(4): Article No. 27.Google Scholar
  31. [31]
    Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.Google Scholar
  32. [32]
    Xu Z, Zhang Q, Cheng S. Multilevel active registration for Kinect human body scans: From low quality to high quality. Multimedia Systems, 2018, 24(3): 257-270.CrossRefGoogle Scholar
  33. [33]
    Li X, Iyengar S. On computing mapping of 3D objects: A survey. ACM Computing Surveys, 2015, 47(2): Article No. 34.Google Scholar
  34. [34]
    van Kaick O, Zhang H, Hamarneh G, Cohen-Or D. A survey on shape correspondence. Computer Graphics Forum, 2011, 30(6): 1681-1707.CrossRefGoogle Scholar
  35. [35]
    Loper M, Mahmood N, Romero J, Pons-Moll G, Black M J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 2015, 34(6): Article No. 248.Google Scholar
  36. [36]
    Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black M J. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.561-578.Google Scholar
  37. [37]
    Alexa M. Recent advances in mesh morphing. Computer Graphics Forum, 2002, 21(2): 173-198.CrossRefGoogle Scholar
  38. [38]
    Liu Z G, Zhou L Y, Leung H, Multon F, Shum H P. High quality compatible triangulations for planar shape animation. In Proc. the 2017 ACM SIGGRAPH ASIA Workshop on Data-Driven Animation Techniques, Nov. 2017, pp.1-8.Google Scholar

Copyright information

© Springer Science+Business Media, LLC & Science Press, China 2019

Authors and Affiliations

  • Masoud Zadghorban Lifkooee
    • 1
  • Celong Liu
    • 1
  • Yongqing Liang
    • 1
  • Yimin Zhu
    • 2
  • Xin Li
    • 1
    Email author
  1. 1.Department of Electrical and Computer EngineeringLouisiana State UniversityBaton RougeU.S.A.
  2. 2.Department of Construction ManagementLouisiana State UniversityBaton RougeU.S.A.

Personalised recommendations