Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Inference of Manipulation Intent in Teleoperation for Robotic Assistance

  • 23 Accesses


In teleoperation, predicting an operator’s intent and providing subsequent assistance have demonstrated great advantages in reducing an operator’s workload and a task’s difficulty as well as enhancing the task performance. Current research aims to tackle target-approaching intent, while our work focus on inferring manipulation (task) intent after the user grasps the object. We model how an object is grasped when being utilized in different manipulation tasks (intents) and then adopt this object grasping model in teleoperation for the intent inference. Our paper focuses on determining if direct interaction models can be used for indirect interaction. As the nature of one’s grasping pose may satisfy multiple tasks (intents), we explore a form of classification modeling known as multi-label classification for multiple broad categories of tasks and objects. We also comprehensively compare classification techniques to determine the most suitable method for determining manipulation intent. With knowing the manipulation intent, future robot control algorithms can provide more helpful and appropriate assistance to facilitate task accomplishment.

This is a preview of subscription content, log in to check access.


  1. 1.

    Lum, M.J., Friedman, D.C., Sankaranarayanan, G., King, H., Fodero, K., Leuschke, R., Hannaford, B., Rosen, J., Sinanan, M.N.: The raven: Design and validation of a telesurgery system. Int. J. Robot. Res. 28(9), 1183–1197 (2009)

  2. 2.

    Hochberg, L.R., Bacher, D., Jarosiewicz, B., Masse, N.Y., Simeral, J.D., Vogel, J., Haddadin, S., Liu, J., Cash, S.S., van der Smagt, P., et al.: Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398), 372 (2012)

  3. 3.

    Bodner, J., Wykypiel, H., Wetscher, G., Schmid, T.: First experiences with the da vinci™operating robot in thoracic surgery. Europ. J. Cardio-thoracic Surg. 25(5), 844–851 (2004)

  4. 4.

    Rybarczyk, Y., Colle, E., Hoppenot, P.: Contribution of neuroscience to the teleoperation of rehabilitation robot. In: 2002 IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 6–pp. IEEE (2002)

  5. 5.

    Healey, A.N.: Speculation on the neuropsychology of teleoperation: Implications for presence research and minimally invasive surgery. Presence 17(2), 199–211 (2008)

  6. 6.

    Li, Y., Tee, K.P., Chan, W.L., Yan, R., Chua, Y., Limbu, D.K.: Continuous role adaptation for human–robot shared control. IEEE Trans. Robot. 31(3), 672–681 (2015)

  7. 7.

    Webb, J.D., Li, S., Zhang, X.: Using visuomotor tendencies to increase control performance in teleoperation. In: American Control Conference (ACC), 2016, pp. 7110–7116. IEEE (2016)

  8. 8.

    Dragan, A.D., Srinivasa, S.S.: A policy-blending formalism for shared control. Int. J. Robot. Res. 32(7), 790–805 (2013)

  9. 9.

    Javdani, S., Srinivasa, S.S., Bagnell, J.A.: Shared autonomy via hindsight optimization. arXiv:1503.07619 (2015)

  10. 10.

    Mylonas, G.P., Kwok, K.-W., James, D.R., Leff, D., Orihuela-Espina, F., Darzi, A., Yang, G.-Z.: Gaze-contingent motor channelling, haptic constraints and associated cognitive demand for robotic mis. Medi. Image Anal. 16(3), 612–631 (2012)

  11. 11.

    Ren, J., Patel, R.V., McIsaac, K.A., Guiraudon, G., Peters, T.M.: Dynamic 3-d virtual fixtures for minimally invasive beating heart procedures. IEEE Trans. Med. Imag. 27(8), 1061–1070 (2008)

  12. 12.

    Muelling, K., Venkatraman, A., Valois, J.-S., Downey, J., Weiss, J., Javdani, S., Hebert, M., Schwartz, A.B., Collinger, J.L., Bagnell, J.A.: Autonomy infused teleoperation with application to bci manipulation. arXiv:1503.05451 (2015)

  13. 13.

    Kim, H.K., Biggs, J., Schloerb, W., Carmena, M., Lebedev, M.A., Nicolelis, M.A., Srinivasan, M.A.: Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Trans. Biomed. Eng. 53(6), 1164–1173 (2006)

  14. 14.

    Li, S., Zhang, X., Kim, F.J., da Silva, R.D., Gustafson, D., Molina, W.R.: Attention-aware robotic laparoscope based on fuzzy interpretation of eye-gaze patterns. J. Med. Dev. 9(4), 041007 (2015)

  15. 15.

    Nikolaidis, S., Zhu, Y.X., Hsu, D., Srinivasa, S.: Human-robot mutual adaptation in shared autonomy. arXiv:1701.07851 (2017)

  16. 16.

    Romano, J.M., Hsiao, K., Niemeyer, G., Chitta, S., Kuchenbecker, K.J.: Human-inspired robotic grasp control with tactile sensing. IEEE Trans. Robot. 27(6), 1067–1079 (2011)

  17. 17.

    Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34(4-5), 705–724 (2015)

  18. 18.

    Montesano, L., Lopes, M., Bernardino, A., Santos-Victor, J.: Learning object affordances: From sensory–motor coordination to imitation. IEEE Trans. Robot. 24(1), 15–26 (2008)

  19. 19.

    Fischinger, D., Vincze, M.: Empty the basket-a shape based learning approach for grasping piles of unknown objects. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2051–2057. IEEE (2012)

  20. 20.

    Trinkle, J.C.: On the stability and instantaneous velocity of grasped frictionless objects. IEEE Trans. Robot. Autom. 8(5), 560–572 (1992)

  21. 21.

    Song, D., Huebner, K., Kyrki, V., Kragic, D.: Learning task constraints for robot grasping using graphical models. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1579–1585. IEEE (2010)

  22. 22.

    Balasubramanian, R., Xu, L., Brook, P.D., Smith, J.R., Matsuoka, Y.: Physical human interactive guidance: Identifying grasping principles from human-planned grasps. IEEE Trans. Robot. 28(4), 899–910 (2012)

  23. 23.

    Huaman Quispe, A., Ben Amor, H., Christensen, H., Stilman, M.: Grasping for a purpose: Using task goals for efficient manipulation planning, arXiv:1603.04338 (2016)

  24. 24.

    Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. In: Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, ser. NIPS’01, pp. 841–848. MIT Press, Cambridge (2001). [Online]. Available: http://dl.acm.org/citation.cfm?id=2980539.2980648

  25. 25.

    Kohonen, T.: The self-organizing map. Neurocomputing 21(1-3), 1–6 (1998)

  26. 26.

    Wehrens, R., Buydens, L.M., et al.: Self-and super-organizing maps in r: The kohonen package. J Stat Softw 21(5), 1–19 (2007)

  27. 27.

    Aliferis, C.F., Tsamardinos, I., Statnikov, A.: Hiton: A novel Markov blanket algorithm for optimal variable selection. In: AMIA Annual Symposium Proceedings, vol. 2003, p 21. American Medical Informatics Association (2003)

  28. 28.

    Scutari, M.: Learning bayesian networks with the bnlearn R package. J. Stat. Softw. 35(3), 1–22 (2010)

  29. 29.

    Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proc. Am. Math. Soc. 7(1), 48–50 (1956)

  30. 30.

    Cooper, G.F., Herskovits, E.: A bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992)

  31. 31.

    Tsoumakas, G., Katakis, I.: Multi-label classification: An overview. Int. J. Data Warehousing Mining 3, 3 (2006)

  32. 32.

    Alvares-Cherman, E., Metz, J., Monard, M.C.: Incorporating label dependency into the binary relevance framework for multi-label classification. Expert Syst. Appl. 39(2), 1647–1655 (2012)

  33. 33.

    Elisseeff, A., Weston, J.: A kernel method for multi-labelled classification. In: Advances in Neural Information Processing Systems (2002)

  34. 34.

    Cooper, G.F.: The computational complexity of probabilistic inference using Bayesian belief networks. Artif. Intell. 42(2–3), 393–405 (1990)

  35. 35.

    Abdiansah, A., Wardoyo, R.: Time complexity analysis of support vector machines (svm) in libsvm. International Journal Computer and Application (2015)

  36. 36.

    Rojas, R.: Neural Networks: A Systematic Introduction. Springer Science and Business Media (2013)

  37. 37.

    Baranitha, R., Mohajerpoor, R., Rakkiyappan, R.: Bilateral teleoperation of single-master multislave systems with semi-Markovian jump stochastic interval time-varying delayed communication channels. IEEE Trans. Cybern., 1–11 (2019)

  38. 38.

    Mohajerpoor, R., Sharifi, I., Talebi, H.A., Rezaei, S.M.: Adaptive bilateral teleoperation of an unknown object handled by multiple robots under unknown communication delay. In: 2013 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 1158–1163 (2013)

Download references


This material is based on work supported by the US NSF under grant 1652454. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the National Science Foundation.

Author information

Correspondence to Xiaoli Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, S., Bowman, M., Nobarani, H. et al. Inference of Manipulation Intent in Teleoperation for Robotic Assistance. J Intell Robot Syst (2020). https://doi.org/10.1007/s10846-019-01125-8

Download citation


  • Object manipulation
  • Human intent
  • Teleoperation
  • Robotic assistant