Robust surface tracking combining features, intensity and illumination compensation

  • Xiaofei DuEmail author
  • Neil Clancy
  • Shobhit Arya
  • George B. Hanna
  • John Kelly
  • Daniel S. Elson
  • Danail Stoyanov
Original Article



Recovering tissue deformation during robotic-assisted minimally invasive surgery procedures is important for providing intra-operative guidance, enabling in vivo imaging modalities and enhanced robotic control. The tissue motion can also be used to apply motion stabilization and to prescribe dynamic constraints for avoiding critical anatomical structures.


Image-based methods based independently on salient features or on image intensity have limitations when dealing with homogeneous soft tissues or complex reflectance. In this paper, we use a triangular geometric mesh model in order to combine the advantages of both feature and intensity information and track the tissue surface reliably and robustly.


Synthetic and in vivo experiments are performed to provide quantitative analysis of the tracking accuracy of our method, and we also show exemplar results for registering multispectral images where there is only a weak image signal.


Compared to traditional methods, our hybrid tracking method is more robust and has improved convergence in the presence of larger displacements, tissue dynamics and illumination changes.


Non-rigid surface tracking Multispectral imaging Minimally invasive surgery Illumination compensation 



Xiaofei Du is supported by the China Scholarship Council scholarship. Neil Clancy is supported by an Imperial College Junior Research Fellowship. Shobhit Arya is supported by an NIHR-HTD 240 Grant. The authors would like to thank the Northwick Park Institute for Medical Research (NPIMR) for their assistance with surgical arrangements.

Compliance with ethical standards

Conflicts of interest

The authors declare no conflict of interest.

Ethical standard

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all patients who were included in the study.

Supplementary material

Supplementary material 1 (mpg 5780 KB)

Supplementary material 2 (mpg 5789 KB)


  1. 1.
    Baker S, Matthews I (2004) Lucas–Kanade 20 years on: a unifying framework. Int J Comput Vis 56(3):221–255CrossRefGoogle Scholar
  2. 2.
    Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer Vision-ECCV 2006, Springer, Berlin, Heidelberg, pp 404–417Google Scholar
  3. 3.
    Bradski G (2000) The opencv library. Doctor Dobbs Journal 25(11):120–126Google Scholar
  4. 4.
    Braux-Zin J, Dupont R, Bartoli A (2013) Combining features and intensity for wide-baseline non-rigid surface registration. In: British machine vision conference (BMVC), BMVAGoogle Scholar
  5. 5.
    Clancy NT, Stoyanov D, James DR, Di Marco A, Sauvage V, Clark J, Yang GZ, Elson DS (2012) Multispectral image alignment using a three channel endoscope in vivo during minimally invasive surgery. Biomed Opt Exp 3(10):2567–2578CrossRefGoogle Scholar
  6. 6.
    Delabarre B, Marchand E (2012) Visual servoing using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 1689–1694Google Scholar
  7. 7.
    Fua P, Brechbühler C (1996) Imposing hard constraints on soft snakes. In: Computer vision-ECCV 1996, Springer, pp 495– 506Google Scholar
  8. 8.
    Giannarou S, Visentini-Scarzanella M, Yang GZ (2009) Affine-invariant anisotropic detector for soft tissue tracking in minimally invasive surgery. In: Biomedical imaging: from Nano to Macro, 2009. ISBI’09. IEEE international symposium on, IEEE, pp 1059–1062Google Scholar
  9. 9.
    Giannarou S, Visentini-Scarzanella M, Yang GZ (2013) Probabilistic tracking of affine-invariant anisotropic regions. Pattern Anal Mach Intell IEEE Trans 35(1):130–143CrossRefGoogle Scholar
  10. 10.
    Gröger M, Sepp W, Ortmaier T, Hirzinger G (2001) Reconstruction of image structure in presence of specular reflections. In: Pattern recognition, Springer, pp 53–60Google Scholar
  11. 11.
    Kalal Z, Mikolajczyk K, Matas J (2010) Forward–backward error: automatic detection of tracking failures. In: Pattern recognition (ICPR), 2010 20th international conference on, IEEE, pp 2756–2759Google Scholar
  12. 12.
    Keerthi SS, DeCoste D (2005) A modified finite newton method for fast solution of large scale linear svms. J Mach Learn Res 6:341–361Google Scholar
  13. 13.
    Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, Stoyanov D (2013) Optical techniques for 3d surface reconstruction in computer-assisted laparoscopic surgery. Med Image Anal 17(8):974–996CrossRefPubMedGoogle Scholar
  14. 14.
    Mangasarian OL (2002) A finite newton method for classification. Opt Method Softw 17(5):913–929CrossRefGoogle Scholar
  15. 15.
    Matthews I, Ishikawa T, Baker S (2004) The template update problem. Pattern Anal Mach Intell IEEE Trans 26(6):810–815CrossRefGoogle Scholar
  16. 16.
    Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: learning local deformation online. In: Medical image computing and computer-assisted intervention-MICCAI 2008, Springer, pp 364–372Google Scholar
  17. 17.
    Ortmaier TJ (2003) Motion compensation in minimally invasive robotic surgery. PhD thesis, Universität MünchenGoogle Scholar
  18. 18.
    Pickering MR, Muhit AA, Scarvell JM, Smith PN (2009) A new multi-modal similarity measure for fast gradient-based 2d–3d image registration. In: Engineering in medicine and biology society, 2009. EMBC 2009, annual international conference of the IEEE, IEEE, pp 5821–5824Google Scholar
  19. 19.
    Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122CrossRefGoogle Scholar
  20. 20.
    Puerto-Souza GA, Mariottini GL (2012) Hierarchical multi-affine (hma) algorithm for fast and accurate feature matching in minimally-invasive surgical images. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, IEEE, pp 2007–2012Google Scholar
  21. 21.
    Richa R, Sznitman R, Taylor R, Hager G (2011) Visual tracking using the sum of conditional variance. In: Intelligent robots and systems (IROS), 2011 IEEE/RSJ international conference on, IEEE, pp 2953–2958Google Scholar
  22. 22.
    Selka F, Nicolau SA, Agnus V, Bessaid A, Marescaux J, Soler L (2013) Evaluation of endoscopic image enhancement for feature tracking: a new validation framework. In: Augmented reality environments for medical imaging and computer-assisted interventions, Springer, pp 75–85Google Scholar
  23. 23.
    Stoyanov D (2012) Stereoscopic scene flow for robotic assisted minimally invasive surgery. In: Medical Image Computing and Computer-Assisted Intervention-MICCAI 2012, Springer, pp 479–486Google Scholar
  24. 24.
    Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345CrossRefPubMedGoogle Scholar
  25. 25.
    Stoyanov D, Yang GZ (2005) Removing specular reflection components for robotic assisted laparoscopic surgery. In: Image processing, 2005. ICIP 2005. IEEE international conference on, IEEE, vol 3, pp III-632Google Scholar
  26. 26.
    Stoyanov D, Darzi A, Yang GZ (2004) Dense 3d depth recovery for soft tissue deformation during robotically assisted laparoscopic surgery. In: Medical image computing and computer-assisted intervention-MICCAI 2004, Springer, pp 41–48Google Scholar
  27. 27.
    Stoyanov D, Darzi A, Yang GZ (2005) A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Comput Aided Surg 10(4):199–208Google Scholar
  28. 28.
    Stoyanov D, Rayshubskiy A, Hillman E (2012) Robust registration of multispectral images of the cortical surface in neurosurgery. In: Biomedical imaging (ISBI), 2012 9th IEEE international symposium on, IEEE, pp 1643–1646Google Scholar
  29. 29.
    Yip MC, Lowe DG, Salcudean SE, Rohling RN, Nguan CY (2012) Real-time methods for long-term tissue feature tracking in endoscopic scenes. In: Information processing in computer-assisted interventions, Springer, pp 33–43Google Scholar
  30. 30.
    Zhu J, Lyu MR, Huang TS (2009) A fast 2d shape recovery approach by fusing features and appearance. Pattern Anal Mach Intell IEEE Trans 31(7):1210–1224CrossRefGoogle Scholar

Copyright information

© CARS 2015

Authors and Affiliations

  • Xiaofei Du
    • 1
    Email author
  • Neil Clancy
    • 2
  • Shobhit Arya
    • 2
  • George B. Hanna
    • 2
  • John Kelly
    • 3
  • Daniel S. Elson
    • 2
  • Danail Stoyanov
    • 1
  1. 1.Department of Computer Science, Centre for Medical Image ComputingUniversity College LondonLondonUK
  2. 2.Department of Surgery and CancerImperial College LondonLondonUK
  3. 3.Division of Surgery and Interventional ScienceUniversity College LondonLondonUK

Personalised recommendations