Force Feedback and Sensory Substitution for Robot-Assisted Surgery

  • Allison M. Okamura
  • Lawton N. Verner
  • Tomonori Yamamoto
  • James C. Gwilliam
  • Paul G. Griffiths


It is hypothesized that the lack of haptic (force and tactile) feedback presented to the surgeon is a limiting factor in the performance of teleoperated robot-assisted minimally invasive surgery. This chapter reviews the technical challenges of creating force feedback in robot-assisted surgical systems and describes recent results in creating and evaluating the effectiveness of this feedback in mock surgical tasks. In the design of a force-feedback teleoperator, the importance of hardware design choices and their relationship to controller design are emphasized. In addition, the practicality and necessity of force feedback in all degrees of freedom of the teleoperator are considered in the context of surgical tasks and the operating room environment. An alternative to direct force feedback to the surgeon’s hands is sensory substitution/augmented reality, in which graphical displays are used to convey information about the forces between the surgical instrument and the patient, or about the mechanical properties of the patient’s tissue. Experimental results demonstrate that the effectiveness of direct and graphical force feedback depend on the nature of the surgical task and the experience level of the surgeon.


Minimally Invasive Surgery Force Feedback Haptic Feedback Haptic Device Recursive Little Square 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported in part by Johns Hopkins University, National Science Foundation grants 0347464, 9731478, and 0722943, and National Institutes of Health grant EB002004. The authors thank Dr. David Yuh, Dr. Li-Ming Su, Dr. Mohsen Mahvash, Carol Reiley, Balazs Vagvolgyi, Masaya Kitagawa and Wagahta Semere for their contributions to this work, and Intuitive Surgical, Inc. for access to surgical robotics hardware.


  1. 1.
    Akinbiyi, T., Reiley, C.E., Saha, S., Burschka, D., Hasser, C.J., Yuh, D.D., Okamura, A.M.: Dynamic augmented reality for sensory substitution in robot-assisted surgical systems. In: 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 567–570 (2006)Google Scholar
  2. 2.
    Anderson, R.J., Spong, M.W.: Bilateral control of teleoperators with time delay. IEEE Trans. Automat. Contr. 34(5), 494–501 (1989)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Barbagli, F., Salisbury, K.: The effect of sensor/actuator asymmetries in haptic interfaces. In: Proceedings of 11th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, pp. 140–147 (2003)Google Scholar
  4. 4.
    Barbagli, F., Salisbury, K., Ho, C., Spence, C., Tan, H.: Haptic discrimination of force direction and the influence of visual information. ACM Trans. Appl. Percept. 3(2), 125–135 (2006)CrossRefGoogle Scholar
  5. 5.
    Bethea, B.T., Okamura, A.M., Kitagawa, M., Fitton, T.P., Cattaneo, S.M., Gott, V.L., Baumgartner, W.A., Yuh, D.D.: Application of haptic feedback to robotic surgery. J. Laparoendosc. Adv. Surg. Tech. 14(3), 191–195 (2004)CrossRefGoogle Scholar
  6. 6.
    Cavusoglu, M.C., Sherman, A., Tendick, F.: Design of bilateral teleoperation controllers for haptic exploration and telemanipulation of soft environments. IEEE Trans. Rob. Autom. 18(4), 641–647 (2002)CrossRefGoogle Scholar
  7. 7.
    Daniel, R.W., McAree, P.R.: Fundamental limits of performance for force reflecting teleoperation. Int. J. Rob. Res. 17(8), 811–830 (1998)CrossRefGoogle Scholar
  8. 8.
    Diolaiti, N., Melchiorri, C., Stramigioli, S.: Contact impedance estimation for robotic systems. IEEE Trans. Robot. 21(5), 925–935 (2005)CrossRefGoogle Scholar
  9. 9.
    Gersem, G.D., Brussel, H.V., Tendick, F.: Reliable and enhanced stiffness perception in soft-tissue telemanipulation. Int. J. Rob. Res. 24(10), 805–822 (2005)CrossRefGoogle Scholar
  10. 10.
    Griffiths, P.G., Gillespie, R.B.: Characterizing teleoperator behavior for feedback design and performance analysis. In: Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 273–280 (2008)Google Scholar
  11. 11.
    Griffiths, P.G., Gillespie, R.B., Freudenberg, J.S.: A fundamental tradeoff between performance and sensitivity in haptic rendering. IEEE Trans. Robot. 24(3), 537–548 (2008)CrossRefGoogle Scholar
  12. 12.
    Gwilliam, J., Mahvash, M., Vagvolgyi, B., Vacharat, A., Yuh, D.D., Okamura, A.M.: Effects of haptic and graphical force feedback for teleoperated palpation. In: IEEE International Conference on Robotics and Automation, pp. 677–682 (2009)Google Scholar
  13. 13.
    Hannaford, B.: A design framework for teleoperators with kinesthetic feedback. IEEE Trans. Rob. Autom. 5(4), 426–434 (1989)CrossRefGoogle Scholar
  14. 14.
    Hashtrudi-Zaad, K., Salcudean, S.E.: Analysis of control architectures for teleoperation systems with impedance/admittance master and slave manipulators. Int. J. Rob. Res. 20(6), 419–445 (2001)CrossRefGoogle Scholar
  15. 15.
    Haykin, S.S.: Active Network Theory. Addison-Wesley, London (1970)MATHGoogle Scholar
  16. 16.
    Hoyt, K., Castaneda, B., Zhang, M., Nigwekar, P., di Sant’agnese, P.A., Joseph, J.V., Strang, J., Rubens, D.J., Parker, K.J.: Tissue elasticity properties as biomarkers for prostate cancer. Cancer Biomark. 4(4–5), 213–225 (2008)Google Scholar
  17. 17.
    Khalil, H.K.: Nonlinear Systems, 3rd edn. Prentice Hall, Upper Saddle River, NJ (2002)MATHGoogle Scholar
  18. 18.
    Kim, K., Cavusoglu, M.C., Chung, W.K.: Quantitative comparison of bilateral teleoperation systems using μ-synthesis. IEEE Trans. Robot. 23(4), 776–789 (2007)CrossRefGoogle Scholar
  19. 19.
    King, C.H., Culjat, M.O., Franco, M.L., Lewis, C.E., Dutson, E.P., Grundfest, W.S., Bisley, J.W.: Tactile feedback induces reduced grasping force in robot-assisted surgery. IEEE Trans. Haptics 2(2), 103–110 (2009)CrossRefGoogle Scholar
  20. 20.
    Kitagawa, M., Bethea, B.T., Gott, V.L., Okamura, A.M.: Analysis of suture manipulation forces for teleoperation with force feedback. In: T. Dohi, R. Kikinis (eds.) Medical Image Computing and Computer Assisted Intervention. Lecture Notes in Computer Science, vol. 2488, pp. 155–162. Springer, Berlin (2002)Google Scholar
  21. 21.
    Krouskop, T.A., Wheeler, T.M., Kallel, F., Garra, B.S., Hall, T.: Elastic moduli of breast and prostate tissues under compression. Ultrasonic imaging 20(4), 260–274 (1998)Google Scholar
  22. 22.
    Lawrence, D.: Stability and transparency in bilateral teleoperation. IEEE Trans. Rob. Autom. 9(5), 624–637 (1993)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Liu, H., Noonan, D.P., Challacombe, B.J., Dasgupta, P., Seneviratne, L.D., Althoefer, K.: Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery. IEEE Trans. Biomed. Eng. 57(2), 404–14 (2010)Google Scholar
  24. 24.
    Mahvash, M., Gwilliam, J., Agarwal, R., Vagvolgyi, B., Su, L.M., Yuh, D.D., Okamura, A.M.: Force-feedback surgical teleoperator: Controller design and palpation experiments. In: 16th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, pp. 465–471 (2008)Google Scholar
  25. 25.
    Miller, A.P., Peine, W.J., Son, J.S., Hammoud, Z.T.: Tactile imaging system for localizing lung nodules during video assisted thoracoscopic surgery. In: IEEE International Conference on Robotics and Automation, pp. 2996–3001 (2007)Google Scholar
  26. 26.
    Niemeyer, G., Slotine, J.E.: Telemanipulation with time delays. Int. J. Rob. Res. 23(9), 873–890 (2004)CrossRefGoogle Scholar
  27. 27.
    Okamura, A.M.: Methods for haptic feedback in teleoperated robot-assisted surgery. Ind. Rob. 31(6), 499–508 (2004)CrossRefGoogle Scholar
  28. 28.
    Ortmaier, T., Deml, B., Kuebler, B., Passig, G., Reintsema, D., Seibold, U.: Robot assisted force feedback surgery. In: Ferre, M., Buss, M., Aracil, R., Melchiorri, C., Balaguer, C. (eds.) Advances in Telerobotics, Springer Tracts on Advanced Robotics STAR 31, pp. 361–379. Springer, Berlin (2007)Google Scholar
  29. 29.
    Reiley, C.E., Akinbiyi, T., Burschka, D., Chang, D.C., Okamura, A.M., Yuh, D.D.: Effects of visual force feedback on robot-assisted surgical task performance. J. Thorac. Cardiovasc. Surg. 135(1), 196–202 (2008)CrossRefGoogle Scholar
  30. 30.
    Saha, S.: Appropriate degrees of freedom of force sensing in robot-assisted minimally invasive surgery. M.S. thesis, Department of Biomedical Engineering, The Johns Hopkins University (2006)Google Scholar
  31. 31.
    Semere, W., Kitagawa, M., Okamura, A.M.: Teleoperation with sensor/actuator asymmetry: Task performance with partial force feedback. In: Proceedings of 12th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, pp. 121–127 (2004)Google Scholar
  32. 32.
    Skogestad, S., Postlethwaite, I.: Multivariable Feedback Control: Analysis and Design. Wiley, New York (1997)MATHGoogle Scholar
  33. 33.
    Tan, H., Barbagli, F., Salisbury, K., Ho, C., Spence, C.: Force-direction discrimination is not influenced by reference force direction. Haptics-e, Electron J. Haptics Res. ( 4(1) (2006)
  34. 34.
    Tavakoli, M., Aziminejad, A., Patel, R., Moallem, M.: High-fidelity bilateral teleoperation systems and the effect of multimodal haptics. IEEE Trans. Syst. Man Cybern. B Cybern. 37(6), 1512–1528 (2007)CrossRefGoogle Scholar
  35. 35.
    Tavakoli, M., Howe, R.D.: Haptic effects of surgical teleoperator flexibility. Int. J. Rob. Res. 28(10), 1289–1301 (2004)CrossRefGoogle Scholar
  36. 36.
    Taylor, R., Jensen, P., Whitcomb, L., Barnes, A., Kumar, R., Stoianovici, D., Gupta, P., Wang, Z., Dejuan, E., Kavoussi, L.: Steady-hand robotic system for microsurgical augmentation. Int. J. Rob. Res. 18(12), 1201–1210 (1999)CrossRefGoogle Scholar
  37. 37.
    Tobergte, A., Konietschke, R., Hirzinger, G.: Planning and control of a teleoperation system for research in minimally invasive robotic surgery. In: IEEE International Conference on Robotics and Automation, pp. 4225–4232 (2009)Google Scholar
  38. 38.
    Trejos, A.L., Jayender, J., Perri, M.T., Naish, M.D., Patel, R.V., Malthaner, R.A.: Robot-assisted Tactile Sensing for Minimally Invasive Tumor Localization. Int. J. Rob. Res. 28(9), 1118–1133 (2009)CrossRefGoogle Scholar
  39. 39.
    Verner, L.N., Okamura, A.M.: Force & torque feedback vs force only feedback. In: Third Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (World Haptics), pp. 406–410 (2009)Google Scholar
  40. 40.
    Verner, L.N.: Sensor/actuator asymmetries in telemanipulators. Ph.D. in Mechanical Engineering, The Johns Hopkins University (2009)Google Scholar
  41. 41.
    Verner, L.N., Okamura, A.M.: Effects of translational and gripping force feedback are decoupled in a 4-degree-of-freedom telemanipulator. In: Second Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (World Haptics), pp. 286–291 (2007)Google Scholar
  42. 42.
    Verner, L.N., Okamura, A.M.: Telemanipulators with sensor/actuator asymmetries fail the robustness criterion. In: Proceedings of 16th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, pp. 267–271 (2008)Google Scholar
  43. 43.
    Wagner, C.R., Stylopoulos, N., Jackson, P.G., Howe, R.D.: The benefit of force feedback in surgery: Examination of blunt dissection. Presence: Teleoperators and Virtual Environments 16(3), 252–262 (2007)CrossRefGoogle Scholar
  44. 44.
    Xu, K., Simaan, N.: An investigation of the intrinsic force sensing capabilities of continuum robots. IEEE Trans. Robot. 24(3), 576–587 (2008)CrossRefGoogle Scholar
  45. 45.
    Yamamoto, T., Bernhardt, M., Peer, A., Buss, M., Okamura, A.M.: Multi-estimator technique for environment parameter estimation during telemanipulation. In: IEEE International Conference on Biomedical Robotics and Biomechatronics, pp. 217–223 (2008)Google Scholar
  46. 46.
    Yamamoto, T., Vagvolgyi, B., Balaji, K., Whitcomb, L., Okamura, A.M.: Tissue property estimation and graphical display for teleoperated robot-assisted surgery. In: IEEE International Conference on Robotics and Automation, pp. 4239–4245 (2009)Google Scholar
  47. 47.
    Yokokohji, Y., Yoshikawa, T.: Bilateral control of master-slave manipulators for ideal kinesthetic coupling–formulation and experiment. IEEE Trans. Rob. Autom. 10(5), 605–20 (1994)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Allison M. Okamura
    • 1
  • Lawton N. Verner
  • Tomonori Yamamoto
  • James C. Gwilliam
  • Paul G. Griffiths
  1. 1.Department of Mechanical Engineering, Laboratory for Computional Sensing and RoboticsJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations