Advertisement

EasyLabels: weak labels for scene segmentation in laparoscopic videos

  • Félix Fuentes-HurtadoEmail author
  • Abdolrahim Kadkhodamohammadi
  • Evangello Flouty
  • Santiago Barbarisi
  • Imanol Luengo
  • Danail Stoyanov
Original Article
  • 19 Downloads

Abstract

Purpose

We present a different approach for annotating laparoscopic images for segmentation in a weak fashion and experimentally prove that its accuracy when trained with partial cross-entropy is close to that obtained with fully supervised approaches.

Methods

We propose an approach that relies on weak annotations provided as stripes over the different objects in the image and partial cross-entropy as the loss function of a fully convolutional neural network to obtain a dense pixel-level prediction map.

Results

We validate our method on three different datasets, providing qualitative results for all of them and quantitative results for two of them. The experiments show that our approach is able to obtain at least \(90\%\) of the accuracy obtained with fully supervised methods for all the tested datasets, while requiring \(\sim 13\)\(\times \) less time to create the annotations compared to full supervision.

Conclusions

With this work, we demonstrate that laparoscopic data can be segmented using very few annotated data while maintaining levels of accuracy comparable to those obtained with full supervision.

Keywords

Computer-assisted interventions Laparoscopy Instrument detection and segmentation 

Notes

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

For this type of study, formal consent is not required.

Supplementary material

11548_2019_2003_MOESM1_ESM.mp4 (3.3 mb)
Supplementary material 1 (mp4 3360 KB)
11548_2019_2003_MOESM2_ESM.mp4 (94.9 mb)
Supplementary material 2 (mp4 97132 KB)
11548_2019_2003_MOESM3_ESM.pdf (47 kb)
Supplementary material 3 (pdf 46 KB)

References

  1. 1.
    Bearman A, Russakovsky O, Ferrari V, Fei-Fei L (2016) What’s the point: semantic segmentation with point supervision. In: European conference on computer vision. Springer, pp 549–565Google Scholar
  2. 2.
    Bodenstedt S, Allan M, Agustinos A, Du X, Garcia-Peraza-Herrera L, Kenngott H, Kurmann T, Müller-Stich B, Ourselin S, Pakhomov D, Sznitman R, Teichmann M, Thoma M, Vercauteren T, Voros S, Wagner M, Wochner P, Maier-Hein L, Stoyanov D, Speidel S. (2018) Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery. arXiv preprint arXiv:1805.02475
  3. 3.
    Bodenstedt S, Ohnemus A, Katic D, Wekerle AL, Wagner M, Kenngott H, Müller-Stich B, Dillmann R, Speidel S. (2018) Real-time image-based instrument classification for laparoscopic surgery. arXiv preprint arXiv:1808.00178
  4. 4.
    Bouget D, Allan M, Stoyanov D, Jannin P (2017) Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35:633–654CrossRefGoogle Scholar
  5. 5.
    Chen LC, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818Google Scholar
  6. 6.
    Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  7. 7.
    Gao M, Xu Z, Lu L, Wu A, Nogues I, Summers RM, Mollura DJ (2016) Segmentation label propagation using deep convolutional neural networks and dense conditional random field. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, pp 1265–1268Google Scholar
  8. 8.
    García-Peraza-Herrera LC, Li W, Fidon L, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S (2017) Toolnet: holistically-nested real-time segmentation of robotic surgical tools. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 5717–5722Google Scholar
  9. 9.
    García-Peraza-Herrera LC, Li W, Gruijthuijsen C, Devreker A, Attilakos G, Deprest J, Vander Poorten E, Stoyanov D, Vercauteren T, Ourselin S (2016) Real-time segmentation of non-rigid surgical tools based on deep learning and tracking. In: International workshop on computer-assisted and robotic endoscopy. Springer, pp 84–95Google Scholar
  10. 10.
    Laina I, Rieke N, Rupprecht C, Vizcaíno JP, Eslami A, Tombari F, Navab N (2017) Concurrent segmentation and localization for tracking of surgical instruments. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 664–672Google Scholar
  11. 11.
    Lejeune L, Grossrieder J, Sznitman R (2018) Iterative multi-path tracking for video and volume segmentation with sparse point supervision. Med Image Anal 50:65–81CrossRefGoogle Scholar
  12. 12.
    Lin D, Dai J, Jia J, He K, Sun J (2016) Scribblesup: scribble-supervised convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3159–3167Google Scholar
  13. 13.
    Maier-Hein L, Ross T, Gröhl J, Glocker B, Bodenstedt S, Stock C, Heim E, Götz M, Wirkert S, Kenngott H, Speidel S, Maier-Hein K (2016) Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 616–623Google Scholar
  14. 14.
    MICCAI 2015: Endovis 2015 instrument segmentation and tracking. https://endovissub-instrument.grand-challenge.org (2015). [Online; Accessed 6-Nov-2018]
  15. 15.
    Pakhomov D, Premachandran V, Allan M, Azizian M, Navab N (2017) Deep residual learning for instrument segmentation in robotic surgery. arXiv preprint arXiv:1703.08580
  16. 16.
    Ross T, Zimmerer D, Vemuri A, Isensee F, Wiesenfarth M, Bodenstedt S, Both F, Kessler P, Wagner M, Müller B, Kengott H, Speidel S, Kop-Schneider A, Maier-Hein K, Maier-Hein L (2018) Exploiting the potential of unlabeled endoscopic video data with self-supervised learning. Int J Comput Assist Radiol Surg 13:1–9CrossRefGoogle Scholar
  17. 17.
    Schoeffmann K, Husslein H, Kletz S, Petscharnig S, Muenzer B, Beecks C (2017) Video retrieval in laparoscopic video recordings with dynamic content descriptors. Multimed Tools Appl 77:16813–16832.  https://doi.org/10.1007/s11042-017-5252-2 CrossRefGoogle Scholar
  18. 18.
    Shvets A, Rakhlin A, Kalinin AA, Iglovikov V (2018) Automatic instrument segmentation in robot-assisted surgery using deep learning. arXiv preprint arXiv:1803.01207
  19. 19.
    Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345CrossRefGoogle Scholar
  20. 20.
    Tang M, Djelouah A, Perazzi F, Boykov Y, Schroers C (2018) Normalized cut loss for weakly-supervised cnn segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake CityGoogle Scholar
  21. 21.
    Tang P, Wang X, Wang A, Yan Y, Liu W, Huang J, Yuille A (2018) Weakly supervised region proposal network and object detection. In: Proceedings of the European conference on computer vision (ECCV), pp 352–368Google Scholar
  22. 22.
    Vardazaryan A, Mutter D, Marescaux J, Padoy N (2018) Weakly-supervised learning for tool localization in laparoscopic videos. In: Stoyanov D et al (eds) Intravascular imaging and computer assisted stenting and large-scale annotation of biomedical data and expert label synthesis. LABELS 2018, CVII 2018, STENT 2018. Lecture Notes in Computer Science, vol 11043. Springer, Cham, pp 169–179Google Scholar
  23. 23.
    Wang X, You S, Li X, Ma H (2018) Weakly-supervised semantic segmentation by iteratively mining common object features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1354–1362Google Scholar
  24. 24.
    Zhao X, Liang S, Wei Y (2018) Pseudo mask augmented object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4061–4070Google Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.Digital SurgeryLondonUnited Kingdom
  2. 2.Wellcome/ESPRC Centre for Interventional and Surgical SciencesLondonUnited Kingdom

Personalised recommendations