Real-Time Avatar Pose Transfer and Motion Generation Using Locally Encoded Laplacian Offsets
- 20 Downloads
We propose a human avatar representation scheme based on intrinsic coordinates, which are invariant to isometry and insensitive to human pose changes, and an efficient pose transfer algorithm that can utilize this representation to reconstruct a human body geometry following a given pose. Such a pose transfer algorithm can be used to control the movement of an avatar model in virtual reality environments following a user’s motion in real time. Our proposed algorithm consists of three main steps. First, we recognize the user’s pose and select a template model from the database who has a similar pose; then, the intrinsic Laplacian offsets encoded in local coordinates are used to reconstruct the human body geometry following the template pose; finally, the morphing between the two poses is generated using a linear interpolation. We perform experiments to evaluate the accuracy and efficiency of our algorithm. We believe our proposed system is a promising human modeling tool that can be used in general virtual reality applications.
Keywordshuman body pose transfer local intrinsic coordinates avatar control in virtual reality
Unable to display preview. Download preview PDF.
We thank anonymous reviewers for their constructive comments.
- Robitaille N, Jackson P L, Hébert L J, Mercier C et al. A virtual reality avatar interaction (VRAI) platform to assess residual executive dysfunction in active military personnel with previous mild traumatic brain injury: Proof of concept. Disability and Rehabilitation: Assistive Technology, 2017, 12(7): 758-764.Google Scholar
- Lifkooee M Z, Liu C L, Li M Q, Li X. Image-based human character modeling and reconstruction for virtual reality exposure therapy. In Proc. the 13th International Conference on Computer Science & Education, Aug. 2018, Article No. 149.Google Scholar
- Urella N, Hughes J, Conrad E, Zhang J S, Li X. A VR scene modelling platform for PTSD treatment. In Proc. the 12th International Conference on Computer Science and Education, Aug. 2017, pp.257-262.Google Scholar
- Orts-Escolano S, Rhemann C, Fanello S et al. Holoportation: Virtual 3D teleportation in real-time. In Proc. the 29th Annual Symposium on User Interface Software and Technology, Oct. 2016, pp.741-754.Google Scholar
- Sorkine O, Cohen-Or D, Lipman Y, Alexa M, Rössl C, Seidel H P. Laplacian surface editing. In Proc. the 2nd Eurographics Symposium on Geometry processing, Jul. 2004, pp.175-184.Google Scholar
- Huang J, Shi X H, Liu X U, Zhou K, Wei L Y, Teng S H, Bao H J, Guo B N, Shum H Y. Subspace gradient domain mesh deformation. In Proc. ACM SIGGRAPH 2006 Papers, Jul. 2006, pp.1126-1134.Google Scholar
- Le B H, Hodgins J K. Real-time skeletal skinning with optimized centers of rotation. ACM Transactions on Graphics, 2016, 35(4): Article No. 37.Google Scholar
- Kim M, Pons-Moll G, Pujades S, Bang S, Kim J, Black M J, Lee S H. Data-driven physics for human soft tissue animation. ACM Transactions on Graphics, 2017, 36(4): Article No. 54.Google Scholar
- Shi X H, Zhou K, Tong Y Y, Desbrun M, Bao H J, Guo B N. Mesh puppetry: Cascading optimization of mesh deformation with inverse kinematics. ACM Trans. Graph., 2007, 26(3): Article No. 81.Google Scholar
- Yasin H, Iqbal U, Kruger B, Weber A, Gall J. A dual-source approach for 3D pose estimation from a single image. In Proc. the 2006 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.4948-4956.Google Scholar
- Martinez J, Hossain R, Romero J, Little J J. A simple yet effective baseline for 3D human pose estimation. In Proc. the 2007 IEEE International Conference on Computer Vision, Oct. 2017, pp.2659-2668.Google Scholar
- Moreno-Noguer F. 3D human pose estimation from a single image via distance matrix regression. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.1561-1570.Google Scholar
- Mehta D, Sridhar S, Sotnychenko O, Rhodin H, Shafiei M, Seidel H P, Xu W P, Casas D, Theobalt C. VNect: Realtime 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, 2017, 36(4): Article No. 44.Google Scholar
- Seo H, Yeo Y I, Wohn K. 3D body reconstruction from photos based on range scan. In Proc. the 1st International Conference on Technologies for E-Learning and Digital Entertainment, Apr. 2006, pp.849-860.Google Scholar
- Lifkooee M Z, Soysal ÖM, Sekeroglu K. Video mining for facial action unit classification using statistical spatial-temporal feature image and LoG deep convolutional neural network. Machine Vision and Applications, 2018. https://doi.org/10.1007/s00138-018-0967-2, Jan. 2019.
- Tagliasacchi A, Zhang H, Cohen-Or D. Curve skeleton extraction from incomplete point cloud. ACM Trans. Graph., 2009, 28(3): Article No. 71.Google Scholar
- Colaianni M, Zöllhoefer M, Süßmuth J, Seider B, Greiner G. A pose invariant statistical shape model for human bodies. In Proc. the 5th International Conference on 3D Body Scanning Technologies, Oct. 2014, pp.327-336.Google Scholar
- Davis T A, Hager W W. Dynamic supernodes in sparse cholesky update/downdate and triangular solves. ACM Trans. Math. Software, 2009, 35(4): Article No. 27.Google Scholar
- Bogo F, Romero J, Loper M, Black M J. FAUST: Dataset and evaluation for 3D mesh registration. In Proc. the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp.3794-3801.Google Scholar
- Li X, Iyengar S. On computing mapping of 3D objects: A survey. ACM Computing Surveys, 2015, 47(2): Article No. 34.Google Scholar
- Loper M, Mahmood N, Romero J, Pons-Moll G, Black M J. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, 2015, 34(6): Article No. 248.Google Scholar
- Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black M J. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Proc. the 14th European Conference on Computer Vision, Oct. 2016, pp.561-578.Google Scholar
- Liu Z G, Zhou L Y, Leung H, Multon F, Shum H P. High quality compatible triangulations for planar shape animation. In Proc. the 2017 ACM SIGGRAPH ASIA Workshop on Data-Driven Animation Techniques, Nov. 2017, pp.1-8.Google Scholar