Abstract
In this paper we present an approach combining a finite element method and a deep neural network to learn complex elastic deformations with the objective of providing augmented reality during hepatic surgery. Derived from the U-Net architecture, our network is built entirely from physically-based simulations of a preoperative segmentation of the organ. These simulations are performed using an immersed-boundary method, which offers several numerical and practical benefits, such as not requiring boundary-conforming volume elements. We perform a quantitative assessment of the method using synthetic and ex vivo patient data. Results show that the network is capable of solving the deformed state of the organ using only a sparse partial surface displacement data and achieve similar accuracy as a FEM solution, while being about 100\(\times \) faster. When applied to an ex vivo liver example, we achieve the registration in only 3 ms with a mean target registration error (TRE) of 2.9 mm.
J.-N. Brunet and A. Mendizabal—Contributed equally to the paper.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
References
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Plantefeve, R., et al.: Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. Ann. Biomed. Eng. 44(1), 139–153 (2016)
Clements, L.W., Chapman, W.C., Dawant, B.M., Galloway, R.L., Miga, M.I.: Robust surface registration using salient anatomical features for image-guided liver surgery: algorithm and validation. Med. Phys. 35(6Part1), 2528–2540 (2008)
Haouchine, N., Dequidt, J., et al.: Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: ISMAR, pp. 199–208 (2013)
Suwelack, S., Röhl, S., Bodenstedt, S., Reichard, D., et al.: Physics-based shape matching for intraoperative image guidance. Med. Phys. 41(11), 111901 (2014)
Alvarez, P., et al.: Lung deformation between preoperative CT and intraoperative CBCT for thoracoscopic surgery: a case study. In: Medical Imaging, vol. 10576D (2018)
Modrzejewski, R., Collins, T., Bartoli, A., Hostettler, A., Marescaux, J.: Soft-body registration of pre-operative 3D models to intra-operative RGBD partial body scans. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 39–46. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_5
Peterlík, I., Duriez, C., Cotin, S.: Modeling and real-time simulation of a vascularized liver tissue. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7510, pp. 50–57. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33415-3_7
Petit, A., Cotin, S.: Environment-aware non-rigid registration in surgery using physics-based simulation. In: ACCV - 14th Asian Conference on Computer Vision (2018)
Meier, U., López, O., Monserrat, C., et al.: Real-time deformable models for surgery simulation: a survey. Comput. Methods Programs Biomed. 77(3), 183–197 (2005)
Marchesseau, S., Heimann, T., Chatelin, S., Willinger, R., Delingette, H.: Multiplicative Jacobian energy decomposition method for fast porous visco-hyperelastic soft tissue model. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6361, pp. 235–242. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15705-9_29
Miller, K., Joldes, G., Lance, D., Wittek, A.: Total Lagrangian explicit dynamics finite element algorithm for computing soft tissue deformation. Commun. Numer. Methods Eng. 23(2), 121–134 (2007)
Collins, T., Pizarro, D., Bartoli, A., Canis, M., Bourdel, N.: Real-time wide-baseline registration of the uterus in monocular laparoscopic videos. In: MICCAI (2013)
Heiselman, J.S., et al.: Characterization and correction of intraoperative soft tissue deformation in image-guided laparoscopic liver surgery. J. Med. Imag. 5(2), 021203 (2017)
Petit, A., Lippiello, V., Siciliano, B.: Real-time tracking of 3D elastic objects with an RGB-D sensor. In: IROS, pp. 3914–3921 (2015)
Düster, A., et al.: The finite cell method for three-dimensional problems of solid mechanics. Comput. Methods Appl. Mech. Eng. 197(45–48), 3768–3782 (2008)
Niroomandi, S., et al.: Real-time deformable models of non-linear tissues by model reduction techniques. Comput. Methods Programs Biomed. 91(3), 223–231 (2008)
Cifuentes, A., et al.: A performance study of tetrahedral and hexahedral elements in 3-D finite element structural analysis. Finite Elem. Anal. Des. 12, 313–318 (1992)
Benzley, S.E., et al.: A comparison of all hexagonal and all tetrahedral finite element meshes for elastic and elasto-plastic analysis. In: 4th IMR, vol. 17, pp. 179–191 (1995)
Wang, E., Nelson, T., Rauch, R.: Back to elements-tetrahedra vs. hexahedra. In: Proceedings of the 2004 International ANSYS Conference (2004)
Miller, K., Lu, J.: On the prospect of patient-specific biomechanics without patient-specific properties of tissues. J. Mech. Behav. Biomed. Mater. 27, 154–166 (2013)
Acknowledgements
This study was supported by H2020-MSCA-ITN Marie Skłodowska-Curie Actions, Innovative Training Networks (ITN) - H2020 MSCA ITN 2016 GA EU project number 722068 High Performance Soft Tissue Navigation (HiPerNav).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 3952 KB)
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Brunet, JN., Mendizabal, A., Petit, A., Golse, N., Vibert, E., Cotin, S. (2019). Physics-Based Deep Neural Network for Augmented Reality During Liver Surgery. In: Shen, D., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. MICCAI 2019. Lecture Notes in Computer Science(), vol 11768. Springer, Cham. https://doi.org/10.1007/978-3-030-32254-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-32254-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32253-3
Online ISBN: 978-3-030-32254-0
eBook Packages: Computer ScienceComputer Science (R0)
