Definition
Augmented reality visualization in image-guided surgery provides the surgeon with the ability to access the radiological images and surgical planning contextually to the anatomy of the real patient. It aims to integrate surgical navigation with virtual planning.
Introduction
The general ability to see into a living human system and to transfer the three-dimensional complexity of the human body into a comprehensive and useful visual representation has historically been considered of utmost importance by physicians in their will to pass on the acquired knowledge and experience to future generations (Fig. 1). In more recent times, the growing availability of new medical imaging modalities together with the need to reduce the invasiveness of the surgical procedures has encouraged the research for new 3D visualization modalities of patient-specific virtual reconstructions of the anatomy. Those...
References
Abe, Y., Sato, S., Kato, K., Hyakumachi, T., Yanagibashi, Y., Ito, M., Abumi, K.: A novel 3D guidance system using augmented reality for percutaneous vertebroplasty. J. Neurosurg. Spine. 19(4), 492–501 (2013). https://doi.org/10.3171/2013.7.Spine12917
Badiali, G., Ferrari, V., Cutolo, F., Freschi, C., Caramella, D., Bianchi, A., Marchetti, C.: Augmented reality as an aid in maxillofacial surgery: validation of a wearable system allowing maxillary repositioning. J. Cranio-Maxillofac. Surg. 42(8), 1970–1976 (2014). https://doi.org/10.1016/j.jcms.2014.09.001
Baumhauer, M., Simpfendoerfer, T., Schwarz, R., Seitel, M., Müller-Stich, B., Gutt, C., Rassweiler, J., Meinzer, H.-P., Wolf, I.: Soft tissue navigation for laparoscopic prostatectomy: evaluation of camera pose estimation for enhanced visualization. In: Medical Imaging 2007, pp. 650911-650911-650912. Society for Optics and Photonics (2007)
Benton, S.A.: Selected Papers on Three-dimensional Displays. SPIE Optical Engineering Press, Bellingham (2001)
Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., Homolka, P., Ewers, R., Bergmann, H.: A head-mounted operating binocular for augmented reality visualization in medicine – design and initial evaluation. IEEE Trans. Med. Imaging. 21(8), 991–997 (2002). https://doi.org/10.1109/Tmi.2002.803099
Blackwelll, M., Nikou, C., DiGioia, A.M., Kanade, T.: An image overlay system for medical data visualization. Med. Image Comput. Comput. Assist. Interv. Miccai’98. 1496, 232–240 (1998)
Cabrilo, I., Bijlenga, P., Schaller, K.: Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir. 156(9), 1769–1774 (2014). https://doi.org/10.1007/s00701-014-2183-9
Cabrilo, I., Schaller, K., Bijlenga, P.: Augmented reality-assisted bypass surgery: embracing minimal invasiveness. World Neurosurg. 83(4), 596–602 (2015). https://doi.org/10.1016/j.wneu.2014.12.020
Caversaccio, M., Giraldez, J.G., Thoranaghatte, R., Zheng, G., Eggli, P., Nolte, L.P., Ballester, M.A.G.: Augmented reality endoscopic system (ARES): preliminary results. Rhinology. 46(2), 156–158 (2008)
Cutolo, F., Parchi, P.D., Ferrari, V.: Video see through AR head-mounted display for medical procedures. Int. Sym. Mix Augment, 393–396 (2014)
Cutolo, F., Badiali, G., Ferrari, V.: Human-PnP: ergonomic AR interaction paradigm for manual placement of rigid bodies. Augmented Environ. Comput. Assist. Interv. Ae-Cai. 9365, 50–60 (2015). https://doi.org/10.1007/978-3-319-24601-7_6
Cutolo, F., Carbone, M., Parchi, P.D., Ferrari, V., Lisanti, M., Ferrari, M.: Application of a new wearable augmented reality video see-through display to aid percutaneous procedures in spine surgery. In: De Paolis, L.T., Mongelli, A. (eds.) Augmented Reality, Virtual Reality, and Computer Graphics: Third International Conference, AVR 2016, Lecce, Italy, June 15–18, 2016. Proceedings, Part II, pp. 43–54. Springer International Publishing, Cham (2016a)
Cutolo, F., Freschi, C., Mascioli, S., Parchi, P., Ferrari, M., Ferrari, V.: Robust and accurate algorithm for wearable stereoscopic augmented reality with three indistinguishable markers. Electronics. 5(3), 59 (2016b)
Cutolo, F., Meola, A., Carbone, M., Sinceri, S., Cagnazzo, F., Denaro, E., Esposito, N., Ferrari, M., Ferrari, V.: A new head-mounted display-based augmented reality system in neurosurgical oncology: a study on phantom. Comput. Assist. Surg. 22(1), 39–53 (2017). https://doi.org/10.1080/24699322.2017.1358400
Deng, W., Li, F., Wang, M., Song, Z.: Easy-to-use augmented reality neuronavigation using a wireless tablet PC. Stereotact. Funct. Neurosurg. 92(1), 17–24 (2014). https://doi.org/10.1159/000354816
Devernay, F., Mourgues, F., Coste-Maniere, E.: Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery. In: International Workshop on Medical Imaging and Augmented Reality, Proceedings, pp. 16–20. (2001)
Edwards, P.J., Johnson, L.G., Hawkes, D.J., Fenlon, M.R.,Strong, A., Gleeson, M.: Clinical experience and perception in stereo augmented reality surgical navigation. In: Yang, G.Z., Jiang, T. (eds.) MIAR, pp. 369–376. Springer, Berlin/Heidelberg (2004). https://doi.org/10.1007/978-3-540-28626-4_45
Elmi-Terander, A., Skulason, H., Soderman, M., Racadio, J., Homan, R., Babic, D., van der Vaart, N., Nachabe, R.: Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. Spine. 41(21), E1303–E1311 (2016). https://doi.org/10.1097/BRS.0000000000001830
Falk, V., Mourgues, F., Adhami, L., Jacobs, S., Thiele, H., Nitzsche, S., Mohr, F.W., Coste-Maniere, T.: Cardio navigation: planning, simulation, and augmented reality in robotic assisted endoscopic bypass grafting. Ann. Thorac. Surg. 79(6), 2040–2048 (2005). https://doi.org/10.1016/j/athorascur.2004.11.060
Feuerstein, M., Mussack, T., Heining, S.M., Navab, N.: Intraoperative laparoscope augmentation for port placement and resection planning in minimally invasive liver resection. IEEE Trans. Med. Imaging. 27(3), 355–369 (2008). https://doi.org/10.1109/Tmi.2007.907327
Freysinger, W., Gunkel, A.R., Thumfart, W.F.: Image-guided endoscopic ENT surgery. Eur. Arch. Otorhinolaryngol. 254(7), 343–346 (1997). https://doi.org/10.1007/Bf02630726
Fritz, J., U-Thainual, P., Ungi, T., Flammang, A.J., Kathuria, S., Fichtinger, G., Iordachita, I.I., Carrino, J.A.: MR-guided vertebroplasty with augmented reality image overlay navigation. Cardiovasc. Intervent. Radiol. 37(6), 1589–1596 (2014). https://doi.org/10.1007/s00270-014-0885-2
Genc, Y., Tuceryan, M., Navab, N.: Practical solutions for calibration of optical see-through devices. In: International Symposium on Mixed and Augmented Reality, Proceedings, pp. 169–175 (2002)
Gilson, S.J., Fitzgibbon, A.W., Glennerster, A.: Spatial calibration of an optical see-through head-mounted display. J. Neurosci. Methods. 173(1), 140–146 (2008). https://doi.org/10.1016/j.jneumeth.2008.05.015
Grimson, W.E.L., Ettinger, G.J., White, S.J., LozanoPerez, T., Wells, W.M., Kikinis, R.: An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization. IEEE Trans. Med. Imaging. 15(2), 129–140 (1996). https://doi.org/10.1109/42.491415
Holliman, N.S., Dodgson, N.A., Favalora, G.E., Pockett, L.: Three-dimensional displays: a review and applications analysis. IEEE Trans. Broadcast. 57(2), 362–371 (2011). https://doi.org/10.1109/Tbc.2011.2130930
Inoue, D., Cho, B., Mori, M., Kikkawa, Y., Amano, T., Nakamizo, A., Yoshimoto, K., Mizoguchi, M., Tomikawa, M., Hong, J., Hashizume, M., Sasaki, T.: Preliminary study on the clinical application of augmented reality neuronavigation. J. Neurol. Surg. A. Cent. Eur. Neurosurg. 74(2), 71–76 (2013). https://doi.org/10.1055/s-0032-1333415
Iseki, H., Masutani, Y., Iwahara, M., Tanikawa, T., Muragaki, Y., Taira, T., Dohi, T., Takakura, K.: Volumegraph (overlaid three-dimensional image-guided navigation). Clinical application of augmented reality in neurosurgery. Stereotact. Funct. Neurosurg. 68(1–4 Pt 1), 18–24 (1997)
Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., Steinicke, F., Lappe, M., Koch, R.: Geometric calibration of head-mounted displays and its effects on distance estimation. IEEE Trans. Vis. Comput. Graph. 18(4), 589–596 (2012)
Kersten-Oertel, M., Jannin, P., Collins, D.L.: DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans. Vis. Comput. Graph. 18(2), 332–352 (2012). https://doi.org/10.1109/TVCG.2011.50
Kersten-Oertel, M., Jannin, P., Collins, D.L.: The state of the art of visualization in mixed reality image guided surgery. Comput. Med. Imaging Graph. 37(2), 98–112 (2013). https://doi.org/10.1016/j.compmedimag.2013.01.009
Kersten-Oertel, M., Gerard, I., Drouin, S., Mok, K., Sirhan, D., Sinclair, D.S., Collins, D.L.: Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int. J. Comput. Assist. Radiol. Surg. 10(11), 1823–1836 (2015). https://doi.org/10.1007/s11548-015-1163-8
Liao, H., Nakajima, S., Iwahara, M., Kobayashi, E., Sakuma, I., Yahagi, N., Dohi, T.: Intra-operative real-time 3-D information display system based on integral videography. In: Niessen, W., Viergever, M. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2001. Lecture Notes in Computer Science, vol. 2208, pp. 392–400. Springer Berlin, (2001)
Liao, H., Hata, N., Nakajima, S., Iwahara, M., Sakuma, I., Dohi, T.: Surgical navigation by autostereoscopic image overlay of integral videography. IEEE Trans. Inf. Technol. Biomed. 8(2), 114–121 (2004)
Liao, H., Inomata, T., Sakuma, I., Dohi, T.: Surgical navigation of integral videography image overlay for open MRI-guided glioma surgery. Med. Imaging Augmented Real. 4091, 187–194 (2006)
Liao, H.E., Inomata, T., Sakuma, I., Dohi, T.: 3-D augmented reality for MRI-guided surgery using integral videography autostereoscopic image overlay. IEEE Trans. Biomed. Eng. 57(6), 1476–1486 (2010). https://doi.org/10.1109/Tbme.2010.2040278
Lovo, E.E., Quintana, J.C., Puebla, M.C., Torrealba, G., Santos, J.L., Lira, I.H., Tagle, P.: A novel, inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery. 60(4 Suppl 2), 366–371.; discussion 371–362 (2007). https://doi.org/10.1227/01.NEU.0000255360.32689.FA
Low, D., Lee, C.K., Dip, L.L., Ng, W.H., Ang, B.T., Ng, I.: Augmented reality neurosurgical planning and navigation for surgical excision of parasagittal, falcine and convexity meningiomas. Br. J. Neurosurg. 24(1), 69–74 (2010). https://doi.org/10.3109/02688690903506093
Marmulla, R., Hoppe, H., Muhling, J., Eggers, G.: An augmented reality system for image-guided surgery. Int. J. Oral Maxillofac. Surg. 34(6), 594–596 (2005). https://doi.org/10.1016/j.ijom.2005.05.004
Meola, A., Cutolo, F., Carbone, M., Cagnazzo, F., Ferrari, M., Ferrari, V.: Augmented reality in neurosurgery: a systematic review. Neurosurg. Rev. (2016). https://doi.org/10.1007/s10143-016-0732-9
Mourgues, F., Coste-Maniere, P.: Flexible calibration of actuated stereoscopic endoscope for overlay in robot assisted surgery. Med. Image Comput. Comput-Assist. Interv. Miccai. 2488(Pt 1), 25–34 (2002.) 2002
Mukawa, H., Akutsu, K., Matsumura, I., Nakano, S., Yoshida, T., Kuwahara, M., Aiki, K., Ogawa, M.: Distinguished paper: a full color eyewear display using holographic planar waveguides. In: 2008 Sid International Symposium, Digest of Technical Papers, vol Xxxix, Books I–Iii 39, pp. 89–92 (2008)
Muller, M., Rassweiler, M.C., Klein, J., Seitel, A., Gondan, M., Baumhauer, M., Teber, D., Rassweiler, J.J., Meinzer, H.P., Maier-Hein, L.: Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int. J. Comput. Assist. Radiol. Surg. 8(4), 663–675 (2013). https://doi.org/10.1007/s11548-013-0828-4
Narita, Y., Tsukagoshi, S., Suzuki, M., Miyakita, Y., Ohno, M., Arita, H., Saito, Y., Kokojima, Y., Watanabe, N., Moriyama, N., Shibui, S.: Usefulness of a glass-free medical three-dimensional autostereoscopic display in neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 9(5), 905–911 (2014). https://doi.org/10.1007/s11548-014-0984-1
Navab, N., Traub, J., Sielhorst, T., Feuerstein, M., Bichlmeier, C.: Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput. Graph. Appl. 27(5), 10–14 (2007). https://doi.org/10.1109/Mcg.2007.117
Navab, N., Heining, S.M., Traub, J.: Camera Augmented Mobile C-Arm (CAMC): calibration, accuracy study, and clinical applications. IEEE Trans. Med. Imaging. 29(7), 1412–1423 (2010). https://doi.org/10.1109/Tmi.2009.2021947
Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopic surgical oncology. Surg. Oncol. Oxf. 20(3), 189–201 (2011). https://doi.org/10.1016/j.suronc.2011.07.002
Peters, T.M.: Image-guided surgery: from X-rays to virtual reality. Comput. Methods. Biomech. Biomed. Engin. 4(1), 27–57 (2000)
Peters, T.M.: Image-guidance for surgical procedures. Phys. Med. Biol. 51(14), R505–R540 (2006). https://doi.org/10.1088/0031-9155/51/14/R01
Plopski, A., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, G., Takemura, H.: Corneal-imaging calibration for optical see-through head-mounted displays. IEEE Trans. Vis. Comput. Graph. 21(4), 481–490 (2015). https://doi.org/10.1109/Tvcg.2015.2391857
Plopski, A., Orlosky, J., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, G.: Automated spatial calibration of HMD systems with unconstrained eye-cameras. In: Proceedings of the 2016 15th IEEE International Symposium on Mixed and Augmented Reality (Ismar), pp. 94–99 (2016). https://doi.org/10.1109/Ismar.2016.16
Rankin, T.M., Slepian, M.J., Armstrong, D.G.: Augmented reality in surgery. In: Latifi, R., Rhee, P., Gruessner, R.W.G. (eds.) Technological Advances in Surgery, Trauma and Critical Care, pp. 59–71. Springer, New York (2015)
Roberts, D.W., Strohbehn, J.W., Hatch, J.F., Murray, W., Kettenberger, H.: A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. J. Neurosurg. 65(4), 545–549 (1986). https://doi.org/10.3171/jns.1986.65.4.0545
Rolland, J.P., Cakmakci, O.: The past, present, and future of head-mounted display designs. In: Photonics Asia 2005, p. 10. SPIE
Rolland, J.P., Holloway, R.L., Fuchs, H.: A comparison of optical and video see-through head-mounted displays. Telemanipulator Telepresence Technol. 2351, 293–307 (1994)
Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical displays: a literature review of augmented reality. J. Disp. Technol. 4(4), 451–467 (2008). https://doi.org/10.1109/JDT.2008.2001575
Stetten, G., Chib, V., Hildebrand, D., Bursee, J.: Real time tomographic reflection: Phantoms for calibration and biopsy. In: IEEE and ACM International Symposium on Augmented Reality, Proceedings, Los Alamitos, IEEE Press, 11–19 (2001)
Su, L.M., Vagvoigyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.: Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology. 73(4), 896–900 (2009). https://doi.org/10.1016/j.urology.2008.11.040
Suenaga, H., Tran, H.H., Liao, H.G., Masamune, K., Dohi, T., Hoshi, K., Mori, Y., Takato, T.: Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study. Int. J. Oral Sci. 5(2), 98–102 (2013). https://doi.org/10.1038/ijos.2013.26
Suenaga, H., Tran, H.H., Liao, H., Masamune, K., Dohi, T., Hoshi, K., Takato, T.: Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study. BMC Med. Imaging. 15, 51 (2015). https://doi.org/10.1186/s12880-015-0089-5. ARTN 51
Tuceryan, M., Genc, Y., Navab, N.: Single-point active alignment method (SPAAM) for optical see-through HMD calibration for augmented reality. Presence-Teleop Virt Environ. 11(3), 259–276 (2002). https://doi.org/10.1162/105474602317473213
Wang, J., Suenaga, H., Yang, L., Kobayashi, E., Sakuma, I.: Video see-through augmented reality for oral and maxillofacial surgery. The international journal of medical robotics + computer assisted surgery: MRCAS 13(2) (2017). https://doi.org/10.1002/rcs.1754
Wesarg, S., Firle, E.A., Schwald, B., Seibert, H., Zogal, P., Roeddiger, S.: Accuracy of needle implantation in brachytherapy using a medical AR system – a phantom study. Med. Imaging 2004: Visual. Image-Guided Proced. Display. 5367, 341–352 (2004). https://doi.org/10.1117/12.535415
Wu, J.R., Wang, M.L., Liu, K.C., Hu, M.H., Lee, P.Y.: Real-time advanced spinal surgery via visible patient model and augmented reality system. Comput. Methods Prog. Biomed. 113(3), 869–881 (2014). https://doi.org/10.1016/j.cmpb.2013.12.071
Zeng, B., Meng, F., Ding, H., Wang, G.: A surgical robot with augmented reality visualization for stereoelectroencephalography electrode implantation. Int. J. Comput. Assist. Radiol. Surg. 12(8), 1355–1368 (2017). https://doi.org/10.1007/s11548-017-1634-1
Zhu, M., Liu, F., Chai, G., Pan, J.J., Jiang, T., Lin, L., Xin, Y., Zhang, Y., Li, Q.: A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery. Sci Rep. 7, 42365 (2017). https://doi.org/10.1038/srep42365
Zinser, M.J., Mischkowski, R.A., Dreiseidler, T., Thamm, O.C., Rothamel, D., Zoller, J.E.: Computer-assisted orthognathic surgery: waferless maxillary positioning, versatility, and accuracy of an image-guided visualisation display. Brit. J. Oral. Max. Surg. 51(8), 827–833 (2013a). https://doi.org/10.1016/j.bjoms.2013.06.014
Zinser, M.J., Sailer, H.F., Ritter, L., Braumann, B., Maegele, M., Zoller, J.E.: A paradigm shift in orthognathic surgery? A comparison of navigation, computer-aided designed/computer-aided manufactured splints, and “classic” intermaxillary splints to surgical transfer of virtual orthognathic planning. J. Oral. Maxillofac. Surg. 71(12.), 2151), e2151–e2121 (2013b). https://doi.org/10.1016/j.joms.2013.07.007
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this entry
Cite this entry
Cutolo, F. (2018). Augmented Reality in Image-Guided Surgery. In: Lee, N. (eds) Encyclopedia of Computer Graphics and Games. Springer, Cham. https://doi.org/10.1007/978-3-319-08234-9_78-2
Download citation
DOI: https://doi.org/10.1007/978-3-319-08234-9_78-2
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08234-9
Online ISBN: 978-3-319-08234-9
eBook Packages: Springer Reference Computer SciencesReference Module Computer Science and Engineering
Publish with us
Chapter history
-
Latest
Augmented Reality in Image-Guided Surgery- Published:
- 20 February 2018
DOI: https://doi.org/10.1007/978-3-319-08234-9_78-2
-
Original
Augmented Reality in Image-Guided Surgery- Published:
- 21 November 2017
DOI: https://doi.org/10.1007/978-3-319-08234-9_78-1