Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation

  • Dafni AntotsiouEmail author
  • Guillermo Garcia-Hernando
  • Tae-Kyun Kim
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11134)


Human hand actions are quite complex, especially when they involve object manipulation, mainly due to the high dimensionality of the hand and the vast action space that entails. Imitating those actions with dexterous hand models involves different important and challenging steps: acquiring human hand information, retargeting it to a hand model, and learning a policy from acquired data. In this work, we capture the hand information by using a state-of-the-art hand pose estimator. We tackle the retargeting problem from the hand pose to a 29 DoF hand model by combining inverse kinematics and PSO with a task objective optimisation. This objective encourages the virtual hand to accomplish the manipulation task, relieving the effect of the estimator’s noise and the domain gap. Our approach leads to a better success rate in the grasping task compared to our inverse kinematics baseline, allowing us to record successful human demonstrations. Furthermore, we used these demonstrations to learn a policy network using generative adversarial imitation learning (GAIL) that is able to autonomously grasp an object in the virtual space.


Hand pose estimation Motion retargeting PSO Anthropomorphic hand model Imitation learning GAIL 



This work is part of Imperial College London-Samsung Research project, supported by Samsung Electronics.


  1. 1.
    Borst, C.W., Indugula, A.P.: Realistic virtual grasping. In: VR (2005)Google Scholar
  2. 2.
    Buchmann, V., Violich, S., Billinghurst, M., Cockburn, A.: FingARtips: gesture based direct manipulation in augmented reality. In: GRAPHITE (2004)Google Scholar
  3. 3.
    Devin, C., Gupta, A., Darrell, T., Abbeel, P., Levine, S.: Learning modular neural network policies for multi-task and multi-robot transfer. In: ICRA (2017)Google Scholar
  4. 4.
    Dhariwal, P., et al.: Openai baselines (2017).
  5. 5.
    Duan, Y., et al.: One-shot imitation learning. In: ICRA (2018)Google Scholar
  6. 6.
    Finn, C., Yu, T., Zhang, T., Abbeel, P., Levine, S.: One-shot visual imitation learning via meta-learning. In: CoRL (2017)Google Scholar
  7. 7.
    Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In: CVPR (2018)Google Scholar
  8. 8.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)Google Scholar
  9. 9.
    Hamer, H., Gall, J., Weise, T., Van Gool, L.: An object-dependent hand pose prior from sparse training data. In: CVPR (2010)Google Scholar
  10. 10.
    Ho, J., Ermon, S.: Generative adversarial imitation learning. In: NIPS (2016)Google Scholar
  11. 11.
    Höll, M., Oberweger, M., Arth, C., Lepetit, V.: Efficient physics-based implementation for realistic hand-object interaction in virtual reality. In: 2018 IEEEVR (2018)Google Scholar
  12. 12.
    Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. ACM CSUR 50, 21 (2017)Google Scholar
  13. 13.
    Jang, Y., Noh, S.T., Chang, H.J., Kim, T.K., Woo, W.: 3D finger cape: clicking action and position estimation under self-occlusions in egocentric viewpoint. TVCG 21, 501–510 (2015)Google Scholar
  14. 14.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: ICANN (1995)Google Scholar
  15. 15.
    Kim, J.S., Park, J.M.: Physics-based hand interaction with virtual objects. In: ICRA (2015)Google Scholar
  16. 16.
    Kumar, V., Gupta, A., Todorov, E., Levine, S.: Learning dexterous manipulation policies from experience and imitation. IJRR (2018)Google Scholar
  17. 17.
    Kumar, V., Todorov, E.: MuJoCo HAPTIX: a virtual reality system for hand manipulation. In: IEEE-RAS (2015)Google Scholar
  18. 18.
    Li, Y., Song, J., Ermon, S.: Infogail: interpretable imitation learning from visual demonstrations. In: NIPS (2017)Google Scholar
  19. 19.
    Li, Z., Hsu, P., Sastry, S.: Grasping and coordinated manipulation by a multifingered robot hand. IJRR 8, 33–50 (1989)Google Scholar
  20. 20.
    Makris, A., Argyros, A.A.: Model-based 3D hand tracking with on-line shape adaptation. In: BMVC (2015)Google Scholar
  21. 21.
    McGee, T.G., Para, M.P., Katyal, K.D., Johannes, M.S.: Demonstration of force feedback control on the modular prosthetic limb. In: SMC (2014)Google Scholar
  22. 22.
    Mehta, D., et al.: VNect: real-time 3D human pose estimation with a single RGB camera. TOG 36, 44 (2017)CrossRefGoogle Scholar
  23. 23.
    Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., Theobalt, C.: Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In: ICCV (2017)Google Scholar
  24. 24.
    Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., Abbeel, P.: Overcoming exploration in reinforcement learning with demonstrations. In: ICRA (2018)Google Scholar
  25. 25.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In: ICCV (2011)Google Scholar
  26. 26.
    Peng, X.B., Abbeel, P., Levine, S., van de Panne, M.: DeepMimic: example-guided deep reinforcement learning of physics-based character skills. In: SIGGRAPH (2018)Google Scholar
  27. 27.
    Rogez, G., Khademi, M., Supančič III, J.S., Montiel, J.M.M., Ramanan, D.: 3D hand pose detection in egocentric RGB-D images. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8925, pp. 356–371. Springer, Cham (2015). Scholar
  28. 28.
    Romero, J., Kjellström, H., Kragic, D.: Hands in action: real-time 3D reconstruction of hands in interaction with objects. In: ICRA (2010)Google Scholar
  29. 29.
    Schaal, S.: Learning from demonstration. In: NIPS (1997)Google Scholar
  30. 30.
    Shi, Y., Eberhart, R.: A modified particle swarm optimizer. In: WCCI (1998)Google Scholar
  31. 31.
    Tompson, J., Stein, M., Lecun, Y., Perlin, K.: Real-time continuous pose recovery of human hands using convolutional networks. ToG 33, 169 (2014)CrossRefGoogle Scholar
  32. 32.
    Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. IJCV 118, 172–193 (2016)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Villegas, R., Yang, J., Ceylan, D., Lee, H.: Neural kinematic networks for unsupervised motion retargetting. In: CVPR (2018)Google Scholar
  34. 34.
    Yang, Y., Hospedales, T.M.: A unified perspective on multi-domain and multi-task learning. In: ICLR (2014)Google Scholar
  35. 35.
    Ye, Q., Kim, T.-K.: Occlusion-aware hand pose estimation using hierarchical mixture density network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 817–834. Springer, Cham (2018). Scholar
  36. 36.
    Ye, Q., Yuan, S., Kim, T.-K.: Spatial attention deep net with partial PSO for hierarchical hybrid hand pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 346–361. Springer, Cham (2016). Scholar
  37. 37.
    Yuan, S., et al.: Depth-based 3D hand pose estimation: from current achievements to future goals. In: CVPR (2018)Google Scholar
  38. 38.
    Yuan, S., Ye, Q., Stenger, B., Jain, S., Kim, T.K.: Bighand2. 2m benchmark: hand pose dataset and state of the art analysis. In: CVPR (2017)Google Scholar
  39. 39.
    Zhang, T., McCarthy, Z., Jow, O., Lee, D., Goldberg, K., Abbeel, P.: Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In: ICRA (2018)Google Scholar
  40. 40.
    Zhao, W., Zhang, J., Min, J., Chai, J.: Robust realtime physics-based motion control for human grasping. TOG 36, 207 (2013)Google Scholar
  41. 41.
    Zhu, Y., et al.: Reinforcement and imitation learning for diverse visuomotor skills. In: RSS (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Dafni Antotsiou
    • 1
    Email author
  • Guillermo Garcia-Hernando
    • 1
  • Tae-Kyun Kim
    • 1
  1. 1.Imperial College LondonLondonUK

Personalised recommendations