EyeSAM: graph-based localization and mapping of retinal vasculature during intraocular microsurgery

  • Shohin Mukherjee
  • Michael Kaess
  • Joseph N. Martel
  • Cameron N. RiviereEmail author
Original Article



Robot-assisted intraocular microsurgery can improve performance by aiding the surgeon in operating on delicate micron-scale anatomical structures of the eye. In order to account for the eyeball motion that is typical in intraocular surgery, there is a need for fast and accurate algorithms that map the retinal vasculature and localize the retina with respect to the microscope.


This work extends our previous work by a graph-based SLAM formulation using a sparse incremental smoothing and mapping (iSAM) algorithm.


The resulting technique, “EyeSAM,” performs SLAM for intraoperative vitreoretinal surgical use while avoiding spurious duplication of structures as with the previous simpler technique. The technique also yields reduction in average pixel error in the camera motion estimation.


This work provides techniques to improve intraoperative tracking of retinal vasculature by handling loop closures and achieving increased robustness to quick shaky motions and drift due to uncertainties in the motion estimation.


Surgical robotics Simultaneous localization and mapping Retinal surgery Factor graphs 



This study was partially funded by the U.S. National Institutes of Health (Grant No. R01EB000526). This article does not contain any studies with human participants or animals performed by any of the authors.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Weiss JN, Bynoe LA (2001) Injection of tissue plasminogen activator into a branch retinal vein in eyes with central retinal vein occlusion. Ophthalmology 108(12):2249–2257CrossRefGoogle Scholar
  2. 2.
    Bynoe LA, Hutchins RK, Lazarus HS, Friedberg MA (2005) Retinal endovascular surgery for central retinal vein occlusion: initial experience of four surgeons. Retina 25:625–632CrossRefGoogle Scholar
  3. 3.
    Rogers S, McIntosh RL, Cheung N, Lim L, Wang JJ, Mitchell P, Kowalski JW, Nguyen H, Wong TY (2010) The prevalence of retinal vein occlusion: pooled data from population studies from the United States, Europe, Asia, and Australia. Ophthalmology 117(2):313–319CrossRefGoogle Scholar
  4. 4.
    Nambi M, Bernstein PS, Abbott JJ (2016) A compact telemanipulated retinal-surgery system that uses commercially available instruments with a quick-change adapter. J Med Robot Res 1(2):1630001CrossRefGoogle Scholar
  5. 5.
    Nasseri MA, Eder M, Eberts D, Nair S, Maier M, Zapp D, Lohmann CP, Knoll A (2013) Kinematics and dynamics analysis of a hybrid parallel-serial micromanipulator designed for biomedical applications. In: Proceedings of the IEEE/ASME international conference on advanced intelligent mechatronics, pp 293–299Google Scholar
  6. 6.
    Wilson JT, Gerber MJ, Prince SW, Chen CW, Schwartz SD, Hubschman JP, Tsao TC (2018) Intraocular robotic interventional surgical system (IRISS): mechanical design, evaluation, and master-slave manipulation. Int J Med Robot Comput Assist Surg 14(1):e1842CrossRefGoogle Scholar
  7. 7.
    Taylor R, Jensen P, Whitcomb L, Barnes A, Kumar R, Stoianovici D, Gupta P, Wang Z, Dejuan E, Kavoussi L (1999) A steady-hand robotic system for microsurgical augmentation. Int J Robot Res 18(12):1201–1210CrossRefGoogle Scholar
  8. 8.
    Üneri A, Balicki MA, Handa J, Gehlbach P, Taylor RH, Iordachita I (2010) New steady-hand eye robot with micro-force sensing for vitreoretinal surgery. In: Proceedings of the 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics, pp 814–9Google Scholar
  9. 9.
    Willekens K, Gijbels A, Schoevaerdts L, Esteveny L, Janssens T, Jonckx B, Feyen JH, Meers C, Reynaerts D, Vander Poorten E, Stalmans P (2017) Robot-assisted retinal vein cannulation in an in vivo porcine retinal vein occlusion model. Acta Ophthalmol 95(3):270–275CrossRefGoogle Scholar
  10. 10.
    Yang S, MacLachlan RA, Riviere CN (2015) Manipulator design and operation for a six-degree-of-freedom handheld tremor-canceling microsurgical instrument. IEEE ASME Trans Mechatron 20(2):761–772CrossRefGoogle Scholar
  11. 11.
    MacLachlan RA, Riviere CN (2009) High-speed microscale optical tracking using digital frequency-domain multiplexing. IEEE Trans Instrum Meas 58(6):1991–2001CrossRefGoogle Scholar
  12. 12.
    Yang S, Martel JN, Lobes LA Jr, Riviere CN (2018) Techniques for robot-aided intraocular surgery using monocular vision. Int J Robot Res 37(8):931–952CrossRefGoogle Scholar
  13. 13.
    Mukherjee S, Yang S, Maclachlan RA, Lobes LA, Martel JN, Riviere CN (2017) Toward monocular camera-guided retinal vein cannulation with an actively stabilized handheld robot. In: Proceedings of the IEEE international conference on robotics and automation, pp 2951–2956Google Scholar
  14. 14.
    Becker BC, Riviere CN (2013) Real-time retinal vessel mapping and localization for intraocular surgery. In: Proceedings of the IEEE international conference on robotics and automation, pp 5346–5351Google Scholar
  15. 15.
    Braun D, Yang S, Martel JN, Riviere CN, Becker BC (2018) EyeSLAM: real-time localization and mapping of retinal vessels during intraocular microsurgery. Int J Med Robot Comput Assist Surg 14:e1848CrossRefGoogle Scholar
  16. 16.
    Olson EB (2009) Real-time correlative scan matching. In: Proceedings of the IEEE international conference on robotics and automation, pp 4387–4393Google Scholar
  17. 17.
    Dellaert F, Kaess M (2017) Factor graphs for robot perception. Found Trends Robot 6(1–2):1–139Google Scholar
  18. 18.
    Kaess M, Johannsson H, Roberts R, Ila V, Leonard JJ, Dellaert F (2012) iSAM2: Incremental smoothing and mapping using the Bayes tree. Int J Robot Res 31(2):216–235CrossRefGoogle Scholar
  19. 19.
    Engel J, Sturm J, Cremers D (2013) Semi-dense visual odometry for a monocular camera. In: Proceedings of the IEEE international conference on computer vision, pp 1449–1456Google Scholar
  20. 20.
    Engel J, Schöps T, Cremers D (2014) LSD-SLAM: large-scale direct monocular SLAM. In: Proceedings of the European conference on computer vision, pp 834–849Google Scholar
  21. 21.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the international conference on computer vision, vol 2, pp 1150–1157Google Scholar
  22. 22.
    Bay H, Ess A, Tuytelaars T, van Gool L (2008) Speeded-up robust features (SURF). Comput Vis Image Underst 110(3):346–359CrossRefGoogle Scholar
  23. 23.
    Can A, Shen H, Turner JN, Tanenbaum HL, Roysam B (1999) Rapid automated tracing and feature extraction from live high-resolution retinal fundus images using direct exploratory algorithms. IEEE Trans Inf Technol Biomed 3(2):125–138CrossRefGoogle Scholar
  24. 24.
    Newman P, Ho K (2005) SLAM-loop closing with visually salient features. In: Proceedings of the IEEE international conference on robotics and automation, pp 635–642Google Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.The Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of OphthalmologyUniversity of PittsburghPittsburghUSA

Personalised recommendations