A markerless automatic deformable registration framework for augmented reality navigation of laparoscopy partial nephrectomy
- 209 Downloads
Purpose Video see-through augmented reality (VST-AR) navigation for laparoscopic partial nephrectomy (LPN) can enhance intraoperative perception of surgeons by visualizing surgical targets and critical structures of the kidney tissue. Image registration is the main challenge in the procedure. Existing registration methods in laparoscopic navigation systems suffer from limitations such as manual alignment, invasive external marker fixation, relying on external tracking devices with bulky tracking sensors and lack of deformation compensation. To address these issues, we present a markerless automatic deformable registration framework for LPN VST-AR navigation.
Dense stereo matching and 3D reconstruction, automatic segmentation and surface stitching are combined to obtain a larger dense intraoperative point cloud of the renal surface. A coarse-to-fine deformable registration is performed to achieve a precise automatic registration between the intraoperative point cloud and the preoperative model using the iterative closest point algorithm followed by the coherent point drift algorithm. Kidney phantom experiments and in vivo experiments were performed to evaluate the accuracy and effectiveness of our approach.
The average segmentation accuracy rate of the automatic segmentation was 94.9%. The mean target registration error of the phantom experiments was found to be 1.28 ± 0.68 mm (root mean square error). In vivo experiments showed that tumor location was identified successfully by superimposing the tumor model on the laparoscopic view.
Experimental results have demonstrated that the proposed framework could accurately overlay comprehensive preoperative models on deformable soft organs automatically in a manner of VST-AR without using extra intraoperative imaging modalities and external tracking devices, as well as its potential clinical use.
KeywordsSurgical navigation Augmented reality Video see-through Dense 3D reconstruction Deformable registration
This work was supported by the National Natural Science Foundation of China (Grant No. 61701014).
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent was obtained from all individual participants included in the study.
- 7.Mountney P, Fallert J, Nicolau S, Soler L, Mewes PW (2014) An augmented reality framework for soft tissue surgery. Med Image Comput Comput Assist Interv 17(1):423–431Google Scholar
- 8.Baumhauer M, Simpfendörfer T, Müller-Stich BP, Teber D, Gutt CN, Rassweiler J, Meinzer H-P, Wolf I (2008) Soft tissue navigation for laparoscopic partial nephrectomy. IJCARS 3(3–4):307–314Google Scholar
- 9.Wild E, Teber D, Schmid D, Simpfendörfer T, Müller M, Baranski AC, Kenngott H, Kopka K, Maier-Hein L (2016) Robust augmented reality guidance with fluorescent markers in laparoscopic surgery. IJCARS 11(6):899–907Google Scholar
- 15.Thompson S, Totz J, Song Y, Johnsen S, Stoyanov D (2015) Accuracy validation of an image guided laparoscopy system for liver resection. Med Imaging 9415:941509-1–941509-12Google Scholar
- 17.Puerto Souza GA, Mariottini GL (2013) Toward long-term and accurate augmented-reality display for minimally-invasive surgery. In: ICRA, pp 5384–5389Google Scholar
- 18.Amir-Khalili A, Nosrati MS, Peyrat JM, Hamarneh G, Abugharbieh R (2013) Uncertainty-encoded augmented reality for robot assisted partial nephrectomy: a phantom study. In: AECAI@MICCAI, pp 182–191. https://doi.org/10.1007/978-3-642-40843-4_20
- 20.Hamarneh G, Amir-Khalili A, Nosrati M, Figueroa I, Kawahara J, Al-Alao O, Peyrat JM, Abi-Nahed J, Al-Ansari A, Abugharbieh R (2014) Towards multi-modal image-guided tumour identification in robot-assisted partial nephrectomy. In: MECBME, pp 159–162. https://doi.org/10.1109/mecbme.2014.6783230
- 21.Haouchine N, Dequidt J, Peterlik I, Kerrien E, Berger MO, Cotin S (2013) Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: ISMAR, pp 199–208. https://doi.org/10.1109/ismar.2013.6671780
- 24.Kong S, Haouchine N, Soares R, Klymchenko A, Andreiuk B, Marques B, Shabat G, Piechaud T, Diana M, Cotin S, Marescaux J (2017) Robust augmented reality registration method for localization of solid organs’ tumors using CT-derived virtual biomechanical model and fluorescent fiducials. Surg Endosc 31(7):2863–2871CrossRefGoogle Scholar
- 25.Wild E, Teber D, Schmid D, Simpfendörfer T, Müller M, Baranski A, Kenngott H, Kopka K, Maier-Hein L (2016) Robust augmented reality guidance with fluorescent markers in laparoscopic surgery. Int J Comput Ass Rad 11(6):899–907Google Scholar
- 26.Chang PL, Stoyanov D, Davison AJ, Edwards PE (2013) Real-time dense stereo reconstruction using convex optimisation with a cost-volume for image-guided robotic surgery. MICCAI 8149:42–49Google Scholar
- 27.Totz J, Thompson S, Stoyanov D, Gurusamy K, Davidson BR, Hawkes DJ, Clarkson MJ (2014) Fast semi-dense surface reconstruction from stereoscopic video in laparoscopic surgery. IPCAI 8498:206–215Google Scholar
- 28.Stoyanov D, Darzi A, Yang GZ (2004) Dense 3D depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery. MICCAI 3217:41–48Google Scholar
- 31.Hirschmuller H (2005) Accurate and efficient stereo processing by semi-global matching and mutual information. CVPR 2:807–814Google Scholar
- 34.He K, Gkioxari G, Dollár P, Girshick R (2017) Mask R-CNN. In: ICCV, pp 2980–2988. https://doi.org/10.1109/iccv.2017.322
- 37.Richard H, Andrew Z (2015) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, CambridgeGoogle Scholar
- 38.Besl PJ, Mckay ND (1992) A method for registration of 3-d shapes. Proc SPIE Int Soc Opt Eng 14(3):239–256Google Scholar
- 41.Myronenko A, Song X, Carreira-Perpinan MA (2007) Non-rigid point set registration: coherent point drift. In: Proceedings of advances in neural information processing systems, pp 1009–1016. https://doi.org/10.1109/tpami.20
- 43.Plantefeve R, Haouchine N, Radoux JP, Cotin S (2014) Automatic alignment of pre and intraoperative data using anatomical landmarks for augmented laparoscopic liver surgery. Lecture Notes in Computer Science, vol 8789. Springer, Berlin, pp 58–66Google Scholar