Online Human In-Hand Manipulation Skill Recognition and Learning

  • Disi Chen
  • Zhaojie JuEmail author
  • Dalin Zhou
  • Gongfa Li
  • Honghai Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11650)


This work intends to contribute to transfer human in-hand manipulation skills to a dexterous prosthetic hand. We proposed a probabilistic framework for both human skill representation and high efficient recognition. Gaussian Mixture Model (GMM) as a probabilistic model, is highly applicable in clustering, data fitting and classification. The human in-hand motions were perceived by a wearable data glove, CyberGlove, the motion trajectory data proposed and represented by GMMs. Firstly, only a certain amount of motion data were used for batch learning the parameters of GMMs. Then, the newly coming data of human motions will help to update the parameters of the GMMs without observation of the historical training data, through our proposed incremental parameter estimation framework. Recognition in the research takes full advantages of the probabilistic model, when the GMMs were trained, the log-likelihood of a candidate trajectory can be used as a measurement to achieve human in-hand manipulation skill recognition. The recognition results of the online trained GMMs show a steady increase in accuracy, which proved that the incremental learning process improved the performance of human in-hand manipulation skill recognition.


In-hand manipulation skills GMMs Online learning 


  1. 1.
    Yousef, H., Boukallel, M., Althoefer, K.: Tactile sensing for dexterous in-hand manipulation in robotics—a review. Sens. Actuators A Phys. 167(2), 171–187 (2011)CrossRefGoogle Scholar
  2. 2.
    Schaal, S.: Learning from demonstration. In: Advances in Neural Information Processing Systems, pp. 1040–1046 (1997)Google Scholar
  3. 3.
    Ju, Z., Liu, H.: Human hand motion analysis with multisensory information. IEEE/ASME Trans. Mechatron. 19(2), 456–466 (2014)CrossRefGoogle Scholar
  4. 4.
    Chen, D., et al.: An interactive image segmentation method in hand gesture recognition. Sensors 17(2), 253 (2017)CrossRefGoogle Scholar
  5. 5.
    Zhang, Y., Chen, L., Ran, X.: Online incremental EM training of GMM and its application to speech processing applications. In: Signal Processing (ICSP), 2010 IEEE 10th International Conference on Proceeding, pp. 1309–1312. IEEE (2010)Google Scholar
  6. 6.
    Sudsang, A., Srinivasa, N.: Grasping and in-hand manipulation: geometry and algorithms. Algorithmica 26(3–4), 466–493 (2000)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Kondo, M., Ueda, J., Ogasawara, T.: Recognition of in-hand manipulation using contact state transition for multifingered robot hand control. Robot. Auton. Syst. 56(1), 66–81 (2008)CrossRefGoogle Scholar
  8. 8.
    Kumar, P., Verma, J., Prasad, S.: Hand data glove: a wearable real-time device for human-computer interaction. Int. J. Adv. Sci. Technol. 43 (2012)Google Scholar
  9. 9.
    Bendels, G. H., Kahlesz, F., Klein, R.: Towards the next generation of 3D content creation. In: Proceedings of the working conference on Advanced visual interfaces, pp. 283–289. ACM (2004)Google Scholar
  10. 10.
    de La Gorce, M., Paragios, N., Fleet, D. J.: Model-based hand tracking with texture, shading and self-occlusions. In: Computer Vision and Pattern Recognition, 2008, CVPR 2008, IEEE Conference On Proceeding, pp. 1–8. IEEE (2008)Google Scholar
  11. 11.
    Fukuda, O., Tsuji, T., Kaneko, M., Otsuka, A.: A human-assisting manipulator teleoperated by EMG signals and arm motions. IEEE Trans. Robot. Autom. 19(2), 210–222 (2003)CrossRefGoogle Scholar
  12. 12.
    Reddy, N.P., Gupta, V.: Toward direct biocontrol using surface EMG signals: control of finger and wrist joint models. Med. Eng. Phys. 29(3), 398–403 (2007)CrossRefGoogle Scholar
  13. 13.
    Kuroda, T., Tabata, Y., Goto, A., Ikuta, H., Murakami, M.: Consumer price data-glove for sign language recognition. In: Proceedings of 5th International Conference on Disability, Virtual Reality Associated Technologies, Oxford, UK, pp. 253–258 (2004)Google Scholar
  14. 14.
    Fahn, C.S., Sun, H.: Development of a data glove with reducing sensors based on magnetic induction. IEEE Trans. Ind. Electron. 52(2), 585–594 (2005)CrossRefGoogle Scholar
  15. 15.
    Calinon, S., Guenter, F., Billard, A.: On learning, representing, and generalizing a task in a humanoid robot. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 37(2), 286–298 (2007)CrossRefGoogle Scholar
  16. 16.
    Kalgaonkar, K., Raj, B.: One-handed gesture recognition using ultrasonic Doppler sonar. In: Acoustics, Speech and Signal Processing, 2009, ICASSP 2009, IEEE International Conference on Proceeding, pp. 1889–1892. IEEE (2009)Google Scholar
  17. 17.
    Hall, P., Marshall, D., Martin, R.: Merging and splitting eigenspace models. IEEE Trans. Pattern Anal. Mach. Intell. 22(9), 1042–1049 (2000)CrossRefGoogle Scholar
  18. 18.
    Song, M., Wang, H.: Highly efficient incremental estimation of Gaussian mixture models for online data stream clustering. In: Intelligent Computing: Theory and Applications III, vol. 5803, pp. 174–184. International Society for Optics and Photonics (2005)Google Scholar
  19. 19.
    Hicks, Y., Hall, P.M., Marshall, D.: A method to add Hidden Markov Models with application to learning articulated motion. In: BMVC, pp. 1–10 (2003)Google Scholar
  20. 20.
    Ju, Z., Liu, H.: A unified fuzzy framework for human-hand motion recognition. IEEE Trans. Fuzzy Syst. 19(5), 901–913 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of PortsmouthPortsmouthUK
  2. 2.Wuhan University of Science and TechnologyWuhanChina
  3. 3.Shanghai Jiao Tong UniversityShanghaiChina

Personalised recommendations