Science China Technological Sciences

, Volume 60, Issue 3, pp 374–384 | Cite as

Human stochastic closed-loop behavior for master-slave teleoperation using multi-leap-motion sensor

  • TianJian Hu
  • XiaoJun Zhu
  • XueQian Wang
  • TianShu Wang
  • JunFeng Li
  • WeiPing Qian


Teleoperation has a wide range of applications that have been under development over the past two decades. Previous researches have focused on the control design of teleoperation machine systems to deal with obstacles such as time-delayed stability and transparency. Recent researches have shown that the inclusion of human closed-loop dynamics in control design can improve the performance of robot telemanipulation. The complexity of human behavior arises from the uncertainty of both human physiology and psychology; hence, the investigation can benefit from empirical studies. This study develops a type of statistical learning method to model and evaluate human stochastic closed-loop behavior, which is considered as a hand motion during the direct incremental control process of master-slave teleoperation. The hand trajectory is empirically considered as having a binary linear regression relationship with the error and error rate between the demanded and simulated teleoperator trajectories, while random movements with zero error and error rate are discovered. Hand movement tracking is achieved using a multi-leap-motion sensor (MLM), which is a markerless and natural infrared vision-based manner for motion capture. The established behavior model and statistical learning results reveal certain human properties of operational activities including visual perception, decision making, and robot telemanipulation. The properties indicate some probable system enhancements for future work.


human behavior teleoperation statistical learning leap motion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Iborra A, Pastor J A, Alvarez B, et al. Robots in radioactive environments. IEEE Robot Autom Mag, 2003, 10: 12–22CrossRefGoogle Scholar
  2. 2.
    El-Fakdi A, Cufí X, Hurtós N, et al. Team-based building of a remotely operated underwater robot, an innovative method of teaching engineering. J Intell Robot Syst, 2016, 81: 51–61CrossRefGoogle Scholar
  3. 3.
    Imaida T, Yokokohji Y, Doi T, et al. Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition. IEEE T Robotic Autom, 2004, 20: 499–511CrossRefGoogle Scholar
  4. 4.
    Norris J S, Powell M W, Vona M A, et al. Mars exploration rover operations with the science activity planner. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, 2005. Barcelona: IEEE, 2005. 4618–4623Google Scholar
  5. 5.
    Ding L, Gao H B, Deng Z Q, et al. Three-layer intelligence of planetary exploration wheeled mobile robots: Robint, virtint, and humint. Sci China Tech Sci, 2015, 58: 1299–1317CrossRefGoogle Scholar
  6. 6.
    Rembold U, Burghart C R. Surgical robotics: An introduction. J Intell Robot Syst, 2001, 30: 1–28CrossRefMATHGoogle Scholar
  7. 7.
    Koizumi N, Warisawa S, Nagoshi M, et al. Construction methodology for a remote ultrasound diagnostic system. IEEE Trans Robot, 2009, 25: 522–538CrossRefGoogle Scholar
  8. 8.
    Enloe C L, Pakula W A, Finney G A, et al. Teleoperation in the undergraduate physics laboratory-teaching an old dog new tricks. IEEE Trans Educ, 1999, 42: 174–179CrossRefGoogle Scholar
  9. 9.
    Baier H, Schmidt G. Transparency and stability of bilateral kinesthetic teleoperation with time-delayed communication. J Intell Robot Syst, 2004, 40: 1–22CrossRefGoogle Scholar
  10. 10.
    Kron A, Schmidt G. Stability and performance analysis of kinesthetic control architectures for bimanual telepresence systems. J Intell Robot Syst, 2006, 46: 1–26CrossRefGoogle Scholar
  11. 11.
    Lii N Y, Chen Z P, Pleintinger B, et al. Toward understanding the effects of visual- and force-feedback on robotic hand grasping performance for space teleoperation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. Taipei: IEEE, 2010. 3745–3752CrossRefGoogle Scholar
  12. 12.
    Thompson R L, Daniel R W, Murray D W. Experiments on operator matching in visual teleoperation. In: 2000 IEEE International Conference on Systems, Man, and Cybernetics, 2000. Nashville: IEEE, 2000. 931–936Google Scholar
  13. 13.
    Cong S, Wang J N. Internet-based and visual feedback networked robot arm teleoperation system. In: International Conference on Networking, Sensing and Control. Chicago: IEEE, 2010. 452–457Google Scholar
  14. 14.
    Hu T, Huang X, Tan Q. Active disturbance rejection controller for space teleoperation. In: International Conference on Automatic Control and Artificial Intelligence (ACAI 2000). Wuhan: IEEE, 2012. 334–337CrossRefGoogle Scholar
  15. 15.
    Anderson R J, Spong M W. Bilateral control of teleoperators with time delay. IEEE T Automat Contr, 1989, 34: 494–501MathSciNetCrossRefGoogle Scholar
  16. 16.
    Salcudean S E, Hashtrudi-Zaad K, Tafazoli S, et al. Bilateral matched impedance teleoperation with application to excavator control. IEEE Contr Syst Mag, 1999, 19: 29–37CrossRefGoogle Scholar
  17. 17.
    Prokopiou P A, Tzafestas S G, Harwin W S. A novel scheme for human-friendly and time-delays robust neuropredictive teleoperation. J Intell Robot Syst, 1999, 25: 311–340CrossRefGoogle Scholar
  18. 18.
    Hirche S, Buss M. Human-oriented control for haptic teleoperation. Proc IEEE, 2012, 100: 623–647CrossRefGoogle Scholar
  19. 19.
    Ou Y, Qian H, Xu Y. Support vector machine based approach for abstracting human control strategy in controlling dynamically stable robots. J Intell Robot Syst, 2009, 55: 39–54CrossRefMATHGoogle Scholar
  20. 20.
    Medina J R, Lorenz T, Hirche S. Synthesizing anticipatory haptic assistance considering human behavior uncertainty. IEEE Trans Robot, 2015, 31: 180–190CrossRefGoogle Scholar
  21. 21.
    Du G, Zhang P. A markerless human-robot interface using particle filter and kalman filter for dual robots. IEEE Trans Ind Electron, 2015, 62: 2257–2264CrossRefGoogle Scholar
  22. 22.
    Lawrence D A. Stability and transparency in bilateral teleoperation. IEEE Trans Rob Aut, 1993, 9: 624–637CrossRefGoogle Scholar
  23. 23.
    Lee D, Spong M W. Passive bilateral teleoperation with constant time delay. IEEE Trans Robot, 2006, 22: 269–281CrossRefGoogle Scholar
  24. 24.
    Chopra N, Spong M W, Hirche S, et al. Bilateral teleoperation over the internet: the time varying delay problem. In: Proceedings of the 2003 American Control Conference. Denver, Colorado, USA. 2003. 155–160Google Scholar
  25. 25.
    Guo S P, Li D X, Meng Y H, et al. Task space control of free-floating space robots using constrained adaptive RBF-NTSM. Sci China Tech Sci, 2014, 57: 828–837CrossRefGoogle Scholar
  26. 26.
    Li Z, Xia Y. Adaptive neural network control of bilateral teleoperation with unsymmetrical stochastic delays and unmodeled dynamics. Int J Robust Nonlin, 2014, 24: 1628–1652MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Li Z, Xia Y, Wang D, et al. Neural network-based control of networked trilateral teleoperation with geometrically unknown constraints. IEEE Trans Cybern, 2016, 46: 1051–1064CrossRefGoogle Scholar
  28. 28.
    Ando T, Tsukahara R, Seki M, et al. A haptic interface “force blinker 2” for navigation of the visually impaired. IEEE Trans Ind Electron, 2012, 59: 4112–4119CrossRefGoogle Scholar
  29. 29.
    Du G, Zhang P, Li D. Human-manipulator interface based on multisensory process via Kalman filters. IEEE Trans Ind Electron, 2014, 61: 5411–5418CrossRefGoogle Scholar
  30. 30.
    Bachmann E R, McGhee R B, Yun Z, et al. Inertial and magnetic posture tracking for inserting humans into networked virtual environments. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. Banff, Canada, 2001. 9–16Google Scholar
  31. 31.
    Rebelo J, Sednaoui T, den Exter E B, et al. Bilateral robot teleoperation: A wearable arm exoskeleton featuring an intuitive user interface. IEEE Robot Autom Mag, 2014, 21: 62–69CrossRefGoogle Scholar
  32. 32.
    Ma Z, Ben-Tzvi P. RML glove—An exoskeleton glove mechanism with haptics feedback. IEEE/ASME Trans Mech, 2015, 20: 641–652CrossRefGoogle Scholar
  33. 33.
    Kofman J, Verma S, Wu X H. Robot-manipulator teleoperation by markerless vision-based hand-arm tracking. Int J Optomechatron, 2007, 1: 331–357CrossRefGoogle Scholar
  34. 34.
    Kofman J, Wu X, Luu T J, et al. Teleoperation of a robot manipulator using a vision-based human-robot interface. IEEE Trans Ind Electron, 2005, 52: 1206–1219CrossRefGoogle Scholar
  35. 35.
    Du G, Zhang P. Markerless human-robot interface for dual robot manipulators using Kinect sensor. Robot Comput Integr Manuf, 2014, 30: 150–159CrossRefGoogle Scholar
  36. 36.
    Weichert F, Bachmann D, Rudak B, et al. Analysis of the accuracy and robustness of the leap motion controller. Sensors, 2013, 13: 6380–6393CrossRefGoogle Scholar
  37. 37.
    James G, Witten D, Hastie T, et al. An introduction to statistical learning. 2014, Scholar
  38. 38.
    Xu W, Liang B, Xu Y, et al. A ground experiment system of freefloating robot for capturing space target. J Intell Robot Syst, 2007, 48: 187–208CrossRefGoogle Scholar
  39. 39.
    Matlin M W. Cognitive psychology. 2013, Scholar

Copyright information

© Science China Press and Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • TianJian Hu
    • 1
    • 2
  • XiaoJun Zhu
    • 3
  • XueQian Wang
    • 3
    • 4
  • TianShu Wang
    • 2
  • JunFeng Li
    • 2
  • WeiPing Qian
    • 1
  1. 1.Beijing Institute of Tracking and Telecommunication TechnologyBeijingChina
  2. 2.School of Aerospace EngineeringTsinghua UniversityBeijingChina
  3. 3.Shenzhen Laboratory of Space Robotics and TelescienceShenzhenChina
  4. 4.The Graduate School of ShenzhenTsinghua UniversityShenzhenChina

Personalised recommendations