Haptics-Enabled Surgical Training System with Guidance Using Deep Learning

  • Ehren Biglari
  • Marie Feng
  • John Quarles
  • Edward Sako
  • John Calhoon
  • Ronald Rodriguez
  • Yusheng FengEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9177)


In this paper, we present a haptics-enabled surgical training system integrated with deep learning for characterization of particular procedures of experienced surgeons to guide medical residents-in-training with quantifiable patterns. The prototype of virtual reality surgical system is built for open-heart surgery with specific steps and biopsy operation. Two abstract surgical scenarios are designed to emulate incision and biopsy surgical procedures. Using deep learning algorithm (autoencoder), the two surgical procedures were trained and characterized. Results show that a vector with 30 real-valued components can quantify both surgical patterns. These values can be used to compare how a resident-in-training performs differently as opposed to an experienced surgeon so that quantifiable corrective training guidance can be provided.


Virtual surgical training system Haptic device Machine learning Deep learning algorithm Autoencoder Motion tracking and quantification 



This project is sponsored by San Antonio Life Science Institute (SALSI) as part of the Medical Data Analytics and Visualization Cluster grant. We also appreciate the assistance of Sam Newman from Multi-media Lab at University of Texas Health Science Center at San Antonio, who generated digital representation of the virtual surgical room and the patient.


  1. 1.
  2. 2.
    Valverde, H.H.: A review of flight simulator transfer of training studies. Hum. Factors: J. Hum. Factors Ergon. Soc. 15(6), 510–522 (1973)Google Scholar
  3. 3.
    Yan, J.K.: Advances in computer-generated imagery for flight simulation. Comput. Graph. and Appl.w IEEE 5(8), 37–51 (1985)CrossRefGoogle Scholar
  4. 4.
    R. N. Haber, “Flight Simulation,” Scientific American, 255(1), 1986Google Scholar
  5. 5.
    Hays, R.T., Jacobs, J.W., Prince, C., Salas, E.: Flight simulator training effectiveness: A meta-analysis. Mil. Psychol. 4(2), 63–74 (1992)CrossRefGoogle Scholar
  6. 6.
    Hays, R.T., Jacobs, J.W., Prince, C., Salas, E.: Requirements for future research in flight simulation training: Guidance based on a meta-analytic review. Int. J. Aviat. Psychol. 2(2), 143–158 (1992)CrossRefGoogle Scholar
  7. 7.
    Satava, R.M.: Virtual reality surgical simulator. Surg. Endosc. 7(3), 203–205 (1993)CrossRefGoogle Scholar
  8. 8.
    Tekkis, P.P., et al.: Evaluation of the learning curve in laparoscopic colorectal surgery: comparison of right-sided and left-sided resections. Ann. Surg. 242(1), 83–91 (2005)CrossRefGoogle Scholar
  9. 9.
    Liu, A., Tendick, F., Cleary, K., Kaufmann, C.: A survey of surgical simulation: applications, technology, and education. Presence: Teleoperators Virtual Environ. 12(6), 599–614 (2003)CrossRefGoogle Scholar
  10. 10.
    Aggarwal, R., Grantcharov, T.P., Eriksen, J.R., Blirup, D., Kristiansen, V.B., Funch-Jensen, P., Darzi, A.: An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann. Surg. 244(2), 310 (2006)CrossRefGoogle Scholar
  11. 11.
    Grantcharov, T.P.: Is virtual reality simulation an effective training method in surgery? Nat. Clin. Pract. Gastroenterol. Hepatol. 5(5), 232–233 (2008)CrossRefGoogle Scholar
  12. 12.
    Hinton, G.E., Osindero, S., The, Y.-W.: A fast learning algorithm for deep belief nets. J. Neural Comput. 18(7), 1527–1534 (2006)zbMATHCrossRefGoogle Scholar
  13. 13.
    Seymour, N.E.: VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J. Surg. 32(2), 182–188 (2008)CrossRefGoogle Scholar
  14. 14.
    Larsen, C. R., Soerensen, J. L., Grantcharov, T. P., Dalsgaard, T., Schouenborg, L., Ottosen, C., Schroeder, T. V., Ottesen, B. S.: Effect of virtual reality training on laparoscopic surgery: randomised controlled trial, BMJ: British Medical Journal, 338, 2009Google Scholar
  15. 15.
    Entwistle, N.: Promoting deep learning through teaching and assessment: conceptual frameworks and educational contexts. In: Proceeding of TLRP Conference, Leicester, November 2000Google Scholar
  16. 16.
    Biglari, E., Feng, Y.: Interactive virtual reality driven learning framework for engineering and science education. In: Proceedings of American Society for Engineering Education Gulf-Southwest Conference, New Orleans, LA 2014Google Scholar
  17. 17.
    Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)Google Scholar
  18. 18.
    Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  19. 19.
    Schmidhuber, J.: deep learning in neural networks: an overview. Neural Networks 61, 85–117 (2015)CrossRefGoogle Scholar
  20. 20.
    Yasmin, S., Du, N., Chen, J., Feng, Y.: A Haptic-enabled novel approach to cardiovascular visualization. Comput. Animation and Virtual Worlds 25(3–4), 255–269 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Ehren Biglari
    • 1
    • 2
  • Marie Feng
    • 1
  • John Quarles
    • 1
    • 2
  • Edward Sako
    • 3
  • John Calhoon
    • 3
  • Ronald Rodriguez
    • 4
  • Yusheng Feng
    • 1
    • 5
    Email author
  1. 1.Center for Simulation, Visualization and Real-Time PredictionSan AntonioUSA
  2. 2.Department of Computer ScienceThe University of Texas at San AntonioSan AntonioUSA
  3. 3.Department of Cardiothoracic SurgeryThe University of Texas Health Science Center at San AntonioSan AntonioUSA
  4. 4.Department of UrologyThe University of Texas Health Science Center at San AntonioSan AntonioUSA
  5. 5.Department of Mechanical EngineeringThe University of Texas at San AntonioSan AntonioUSA

Personalised recommendations