Proof of Concept: Wearable Augmented Reality Video See-Through Display for Neuro-Endoscopy

  • Marina CarboneEmail author
  • Sara CondinoEmail author
  • Fabrizio Cutolo
  • Rosanna Maria Viglialoro
  • Oliver Kaschke
  • Ulrich W. Thomale
  • Vincenzo Ferrari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10851)


In mini-invasive surgery and in endoscopic procedures, the surgeon operates without a direct visualization of the patient’s anatomy. In image-guided surgery, solutions based on wearable augmented reality (AR) represent the most promising ones. The authors describe the characteristics that an ideal Head Mounted Display (HMD) must have to guarantee safety and accuracy in AR-guided neurosurgical interventions and design the ideal virtual content for guiding crucial task in neuro endoscopic surgery. The selected sequence of AR content to obtain an effective guidance during surgery is tested in a Microsoft Hololens based app.


Minimally invasive surgery Augmented reality and visualization Computer assisted intervention Neuroendoscopy 



Funded BY THE HORIZON2020 Project VOSTARS, Project ID: 731974. Call: ICT-29-2016 - Photonics KET 2016.


  1. 1.
    Inoue, D., Cho, B., Mori, M., Kikkawa, Y., Amano, T., Nakamizo, A., Yoshimoto, K., Mizoguchi, M., Tomikawa, M., Hong, J., Hashizume, M., Sasaki, T.: Preliminary study on the clinical application of augmented reality neuronavigation. J. Neurol. Surg. A Cent. Eur. Neurosurg. 74, 71–76 (2013)CrossRefGoogle Scholar
  2. 2.
    Kockro, R.A., Tsai, Y.T., Ng, I., Hwang, P., Zhu, C., Agusanto, K., Hong, L.X., Serra, L.: Dex-ray: augmented reality neurosurgical navigation with a handheld video probe. Neurosurgery 65, 795–807 (2009). discussion 807-798CrossRefGoogle Scholar
  3. 3.
    Schulz, M., Bohner, G., Knaus, H., Haberl, H., Thomale, U.-W.: Navigated endoscopic surgery for multiloculated hydrocephalus in children. J. Neurosurg. Pediatr. 5, 434–442 (2010)CrossRefGoogle Scholar
  4. 4.
    King, A.P., Edwards, P.J., Maurer Jr., C.R., de Cunha, D.A., Hawkes, D.J., Hill, D.L., Gaston, R.P., Fenlon, M.R., Strong, A.J., Chandler, C.L., Richards, A., Gleeson, M.J.: A system for microscope-assisted guided interventions. Stereotact. Funct. Neurosurg. 72, 107–111 (1999)CrossRefGoogle Scholar
  5. 5.
    Edwards, P.J., King, A.P., Maurer Jr., C.R., de Cunha, D.A., Hawkes, D.J., Hill, D.L., Gaston, R.P., Fenlon, M.R., Jusczyzck, A., Strong, A.J., Chandler, C.L., Gleeson, M.J.: Design and evaluation of a system for microscope-assisted guided interventions (MAGI). IEEE Trans. Med. Imaging 19, 1082–1093 (2000)CrossRefGoogle Scholar
  6. 6.
    Stadie, A.T., Reisch, R., Kockro, R.A., Fischer, G., Schwandt, E., Boor, S., Stoeter, P.: Minimally invasive cerebral cavernoma surgery using keyhole approaches - solutions for technique-related limitations. Minim. Invasive Neurosurg. 52, 9–16 (2009)CrossRefGoogle Scholar
  7. 7.
    Cabrilo, I., Bijlenga, P., Schaller, K.: Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir. (Wien) 156, 1769–1774 (2014)CrossRefGoogle Scholar
  8. 8.
    Deng, W.W., Li, F., Wang, M.N., Song, Z.J.: Easy-to-Use augmented reality neuronavigation using a wireless tablet PC. Stereot. Funct. Neuros. 92, 17–24 (2014)CrossRefGoogle Scholar
  9. 9.
    Besharati Tabrizi, L., Mahvash, M.: Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique. J. Neurosurg. 123, 206–211 (2015)CrossRefGoogle Scholar
  10. 10.
    Citardi, M.J., Agbetoba, A., Bigcas, J.L., Luong, A.: Augmented reality for endoscopic sinus surgery with surgical navigation: a cadaver study. Int. Forum Allergy Rhinol. 6, 523–528 (2016)CrossRefGoogle Scholar
  11. 11.
    Cutolo, F., Meola, A., Carbone, M., Sinceri, S., Cagnazzo, F., Denaro, E., Esposito, N., Ferrari, M., Ferrari, V.: A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom. Comput. Assist. Surg. 22, 39–53 (2017)CrossRefGoogle Scholar
  12. 12.
    Kawamata, T., Iseki, H., Shibasaki, T., Hori, T.: Endoscopic augmented reality navigation system for endonasal transsphenoidal surgery to treat pituitary tumors: technical note. Neurosurgery 50, 1393–1397 (2002)Google Scholar
  13. 13.
    Meola, A., Cutolo, F., Carbone, M., Cagnazzo, F., Ferrari, M., Ferrari, V.: Augmented reality in neurosurgery: a systematic review. Neurosurg. Rev. 40, 537–548 (2017)CrossRefGoogle Scholar
  14. 14.
    Finger, T., Schaumann, A., Schulz, M., Thomale, U.W.: Augmented reality in intraventricular neuroendoscopy. Acta Neurochir. (Wien) 159, 1033–1041 (2017)CrossRefGoogle Scholar
  15. 15.
    Cutolo, F.: Augmented Reality in Image-Guided Surgery. In: Lee, N. (ed.) Encyclopedia of computer graphics and games, pp. 1–11. Springer, Cham (2017)Google Scholar
  16. 16.
    Rolland, J.P., Fuchs, H.: Optical versus video see-through head-mounted displays in medical visualization. Presence Teleoper. Virtual Environ. 9, 287–309 (2000)CrossRefGoogle Scholar
  17. 17.
    Cutolo, F., Fontana, U., Carbone, M., D’Amato, R., Ferrari, V.: Hybrid video/optical see-through HMD. Adjunct. In: Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality (Ismar-Adjunct), pp. 52–57 (2017)Google Scholar
  18. 18.
  19. 19.
    Kersten-Oertel, M., Jannin, P., Collins, D.L.: The state of the art of visualization in mixed reality image guided surgery. Comput. Med. Imag. Grap. 37, 98–112 (2013)CrossRefGoogle Scholar
  20. 20.
    Bichlmeier, C., Wimme, F., Heining, S.M., Navab, N.: Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, 2007, ISMAR 2007, pp. 129–138 (2007)Google Scholar
  21. 21.
    Kersten-Oertel, M., Chen, S.J.S., Collins, D.L.: An evaluation of depth enhancing perceptual cues for vascular volume visualization in neurosurgery. IEEE Trans. Vis. Comput. Graph. 20, 391–403 (2014)CrossRefGoogle Scholar
  22. 22.
    Badiali, G., Ferrari, V., Cutolo, F., Freschi, C., Caramella, D., Bianchi, A., Marchetti, C.: Augmented reality as an aid in maxillofacial surgery: Validation of a wearable system allowing maxillary repositioning. J. Cranio. Maxill. Surg. 42, 1970–1976 (2014)CrossRefGoogle Scholar
  23. 23.
    Cutolo, F., Parchi, P.D., Ferrari, V.: Video see through AR head-mounted display for medical procedures. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2014, pp. 393–396. IEEE (2014)Google Scholar
  24. 24.
    Parrini, S., Cutolo, F., Freschi, C., Ferrari, M., Ferrari, V.: Augmented reality system for freehand guide of magnetic endovascular devices. In: 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 490–493. IEEE (2014)Google Scholar
  25. 25.
    Ferrari, V., Viglialoro, R.M., Nicoli, P., Cutolo, F., Condino, S., Carbone, M., Siesto, M., Ferrari, M.: Augmented reality visualization of deformable tubular structures for surgical simulation. Int. J. Med. Robot. Comput. Assist. Surg. 12(2), 231–240 (2015)CrossRefGoogle Scholar
  26. 26.
    Cutolo, F., Badiali, G., Ferrari, V.: Human-PnP: ergonomic AR interaction paradigm for manual placement of rigid bodies. In: Linte, Cristian A., Yaniv, Z., Fallavollita, P. (eds.) AE-CAI 2015. LNCS, vol. 9365, pp. 50–60. Springer, Cham (2015). Scholar
  27. 27.
    Evans, G., Miller, J., Pena, M.I., MacAllister, A., Winer, E.: Evaluating the Microsoft HoloLens through an augmented reality assembly application. Degrad. Environ. Sens. Process. Display 2017, 10197 (2017)Google Scholar
  28. 28.
    Ferrari, V., Carbone, M., Cappelli, C., Boni, L., Melfi, F., Ferrari, M., Mosca, F., Pietrabissa, A.: Value of multidetector computed tomography image segmentation for preoperative planning in general surgery. Surg. Endosc. 26, 616–626 (2012)CrossRefGoogle Scholar
  29. 29.
  30. 30.
    Badiali, G., Roncari, A., Bianchi, A., Taddei, F., Marchetti, C., Schileo, E.: Navigation in orthognathic surgery: 3D accuracy. Facial Plast. Surg. FPS 31, 463–473 (2015)CrossRefGoogle Scholar
  31. 31.
    Volonte, F., Pugin, F., Bucher, P., Sugimoto, M., Ratib, O., Morel, P.: Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat. Sci. 18, 506–509 (2011)CrossRefGoogle Scholar
  32. 32.
    Zheng, G., Nolte, L.P.: Computer-assisted orthopedic surgery: current state and future perspective. Front. Surg. 2, 66 (2015)CrossRefGoogle Scholar
  33. 33.
    Luebbers, H.T., Messmer, P., Obwegeser, J.A., Zwahlen, R.A., Kikinis, R., Graetz, K.W., Matthews, F.: Comparison of different registration methods for surgical navigation in cranio-maxillofacial surgery. J. Cranio-Maxillo-Facial Surg. 36, 109–116 (2008). Official publication of the European Association for Cranio-Maxillo-Facial SurgeryCrossRefGoogle Scholar
  34. 34.
    Condino, S., Calabro, E.M., Alberti, A., Parrini, S., Cioni, R., Berchiolli, R.N., Gesi, M., Ferrari, V., Ferrari, M.: Simultaneous tracking of catheters and guidewires: comparison to standard fluoroscopic guidance for arterial cannulation. Eur. J. Vasc. Endovasc. Surg. 47, 53–60 (2014). The official journal of the European Society for Vascular SurgeryCrossRefGoogle Scholar
  35. 35.
    Parrini, S., Zhang, L., Condino, S., Ferrari, V., Caramella, D., Ferrari, M.: Automatic carotid centerline extraction from three-dimensional ultrasound Doppler images. In: 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2014, pp. 5089–5092 (2014)Google Scholar
  36. 36.
    Condino, S., Ferrari, V., Freschi, C., Alberti, A., Berchiolli, R., Mosca, F., Ferrari, M.: Electromagnetic navigation platform for endovascular surgery: how to develop sensorized catheters and guidewires. Int. J. Med. Robot. + Comput. Assist. Surg. MRCAS 8, 300–310 (2012)CrossRefGoogle Scholar
  37. 37.
    Ukimura, O., Gill, I.S.: Image-fusion, augmented reality, and predictive surgical navigation. Urol. Clin. North Am. 36, 115–123, vii (2009)CrossRefGoogle Scholar
  38. 38.
    Lamata, P., Ali, W., Cano, A., Cornella, J., Declerck, J., Elle, O.J., Freudenthal, A., Furtado, H., Kalkofen, D., Naerum, E., Samset, E., Sánchez-Gonzalez, P., Sánchez-Margallo, F.M., Schmalstieg, D., Sette, M., Stüdeli, T., Sloten, J.V., Gómez, E.J.: Augmented Reality for Minimally Invasive Surgery: Overview and Some Recent Advances (2010)Google Scholar
  39. 39.
    Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopic surgical oncology. Surg. Oncol. 20, 189–201 (2011)CrossRefGoogle Scholar
  40. 40.
    Rankin, T.M., Slepian, M.J., Armstrong, D.G.: Augmented reality in surgery. In: Latifi, R., Rhee, P., Gruessner, W.G.R. (eds.) Technological Advances in Surgery, Trauma and Critical Care, pp. 59–71. Springer, New York (2015)CrossRefGoogle Scholar
  41. 41.
    Ferrari, V., Viglialoro, R.M., Nicoli, P., Cutolo, F., Condino, S., Carbone, M., Siesto, M., Ferrari, M.: Augmented reality visualization of deformable tubular structures for surgical simulation. Int. J. Med. Rob. + Comput. Assist. Surg. MRCAS 12, 231–240 (2016)CrossRefGoogle Scholar
  42. 42.
    Viglialoro, R., Ferrari, V., Carbone, M.C.M., Condino, S., Porcelli, F., Puccio, F.D., Ferrari, M., Mosca, F.: A physical patient specific simulator for cholecystectomy training. In: CARS Proceedings of the 25th International Congress and Exhibition, Pisa, Italy, June 27–30 (2012)Google Scholar
  43. 43.
    Francesconi, M., Freschi, C., Sinceri, S., Carbone, M., Cappelli, C., Morelli, L., Ferrari, V., Ferrari, M.: New training methods based on mixed reality for interventional ultrasound: design and validation. In: Engineering in Medicine and Biology Society (EMBC), 2015, 37th Annual International Conference of the IEEE, pp. 5098–5101. IEEE (2015)Google Scholar
  44. 44.
    Freschi, C., Parrini, S., Dinelli, N., Ferrari, M., Ferrari, V.: Hybrid simulation using mixed reality for interventional ultrasound imaging training. Int. J. Comput. Assist. Radiol. Surg. 10(7), 1109–1115 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Marina Carbone
    • 1
    • 2
    Email author
  • Sara Condino
    • 1
    Email author
  • Fabrizio Cutolo
    • 1
    • 2
  • Rosanna Maria Viglialoro
    • 1
  • Oliver Kaschke
    • 3
  • Ulrich W. Thomale
    • 4
  • Vincenzo Ferrari
    • 1
    • 2
  1. 1.EndoCAS CenterDepartment of Translational Research and New Technologies in Medicine and Surgery, University of PisaPisaItaly
  2. 2.Information Engineering DepartmentUniversity of PisaPisaItaly
  3. 3.Ear-Nose-Throat DepartmentSankt Gertrauden-Krankenhaus GmbHBerlinGermany
  4. 4.Department of Neurosurgery with Pediatric NeurosurgeryCharité UniversitätsmedizinBerlinGermany

Personalised recommendations