Medical Image Segmentation Using Deep Neural Networks with Pre-trained Encoders

  • Alexandr A. KalininEmail author
  • Vladimir I. Iglovikov
  • Alexander Rakhlin
  • Alexey A. Shvets
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1098)


With the growth of popularity of deep neural networks for image analysis, segmentation is the most common subject of studies applying deep learning to medical imaging and establishing state-of-the-art performance results in many applications. However, it still remains a challenging problem, for which performance improvements can potentially benefit diagnosis and other clinical practice outcomes. In this chapter, we consider two applications of multiple deep convolutional neural networks to medical image segmentation. First, we describe angiodysplasia lesion segmentation from wireless capsule endoscopy videos. Angiodysplasia is the most common vascular lesion of the gastrointestinal tract in the general population and is important to detect as it may indicate the possibility of gastrointestinal bleeding and/or anemia. As a baseline, we consider the U-Net model and then we demonstrate further performance improvements by using different deep architectures with ImageNet pre-trained encoders. In the second example, we apply these models to semantic segmentation of robotic instruments in surgical videos. Segmentation of instruments in the vicinity of surgical scenes is a challenging problem that is important for intraoperative guidance that can help the decision-making process. We achieve highly competitive performance for binary as well as for multi-class instrument segmentation. In both applications, we demonstrate that networks that employ ImageNet pre-trained encoders consistently outperform the U-Net architecture trained from scratch.


  1. 1.
    H. Greenspan, B. Van Ginneken, R.M. Summers, Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35(5), 1153–1159 (2016)CrossRefGoogle Scholar
  2. 2.
    T. Ching, D.S. Himmelstein, B.K. Beaulieu-Jones, A.A. Kalinin, B.T. Do, G.P. Way, E. Ferrero, P.-M. Agapow, M. Zietz, M.M. Hoffman, W. Xie, G.L. Rosen, B.J. Lengerich, J. Israeli, J. Lanchantin, S. Woloszynek, A.E. Carpenter, A. Shrikumar, J. Xu, E.M. Cofer, C.A. Lavender, S.C. Turaga, A.M. Alexandari, Z. Lu, D.J. Harris, D. DeCaprio, Y. Qi, A. Kundaje, Y. Peng, L.K. Wiley, M.H.S. Segler, S.M. Boca, S.J. Swamidass, A. Huang, A. Gitter, C.S. Greene, Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15(141) (2018)Google Scholar
  3. 3.
    G. Litjens, T. Kooi, B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, J.A. Van Der Laak, B. Van Ginneken, C.I. Sánchez, A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  4. 4.
    A. Rakhlin, A. Shvets, V. Iglovikov, A.A. Kalinin, Deep convolutional neural networks for breast cancer histology image analysis, in International Conference Image Analysis and Recognition (Springer, 2018), pp. 737–744Google Scholar
  5. 5.
    A. Rakhlin, A.A. Shvets, A.A. Kalinin, A. Tiulpin, V.I. Iglovikov, S. Nikolenko, Breast tumor cellularity assessment using deep neural networks, in 2019 IEEE International Conference on Computer Vision Workshops (ICCVW) (IEEE, 2019)Google Scholar
  6. 6.
    A. Tiulpin, J. Thevenot, E. Rahtu, P. Lehenkari, S. Saarakkala, Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8, 1727 (2018)CrossRefGoogle Scholar
  7. 7.
    V.I. Iglovikov, A. Rakhlin, A.A. Kalinin, A.A. Shvets, Paediatric bone age assessment using deep convolutional neural networks, in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support (Springer, 2018), pp. 300–308Google Scholar
  8. 8.
    O. Ronneberger, P. Fischer, T. Brox, U-net: convolutional networks for biomedical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, 2015), pp. 234–241Google Scholar
  9. 9.
    J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3431–3440Google Scholar
  10. 10.
    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  11. 11.
    A. Rajkomar, S. Lingam, A.G. Taylor, M. Blum, J. Mongan, High-throughput classification of radiographs using deep convolutional neural networks. J. Digit. Imaging 30(1), 95–101 (2017)CrossRefGoogle Scholar
  12. 12.
    V. Iglovikov, A. Shvets, Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation (2018), arXiv:1801.05746
  13. 13.
    A.A. Shvets, A. Rakhlin, A.A. Kalinin, V.I. Iglovikov, Automatic instrument segmentation in robot-assisted surgery using deep learning, in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA) (IEEE, 2018), pp. 624–628Google Scholar
  14. 14.
    A. Chaurasia, E. Culurciello, Linknet: Exploiting encoder representations for efficient semantic segmentation (2017), arXiv:1707.03718
  15. 15.
    V. Iglovikov, S. Mushinskiy, V. Osin, Satellite imagery feature detection using deep convolutional neural network: a kaggle competition (2017), arXiv:1706.06169
  16. 16.
    K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition (2014), arXiv:1409.1556
  17. 17.
    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), pp. 770–778Google Scholar
  18. 18.
    D.P. Kingma, J. Ba, Adam: a method for stochastic optimization (2014), arXiv:1412.6980
  19. 19.
    P.G. Foutch, D.K. Rex, D.A. Lieberman, Prevalence and natural history of colonic angiodysplasia among healthy asymptomatic people. Am. J. Gastroenterol. 90(4) (1995)Google Scholar
  20. 20.
    J. Regula, E. Wronska, J. Pachlewski, Vascular lesions of the gastrointestinal tract. Best Pract. Res. Clin. Gastroenterol. 22(2), 313–328 (2008)CrossRefGoogle Scholar
  21. 21.
    S.L. Triester, J.A. Leighton, G.I. Leontiadis, D.E. Fleischer, A.K. Hara, R.I. Heigh, A.D. Shiff, V.K. Sharma, A meta-analysis of the yield of capsule endoscopy compared to other diagnostic modalities in patients with obscure gastrointestinal bleeding. Am. J. Gastroenterol. 100(11), 2407 (2005)CrossRefGoogle Scholar
  22. 22.
    R. Marmo, G. Rotondano, R. Piscopo, M. Bianco, L. Cipolletta, Meta-analysis: capsule enteroscopy vs. conventional modalities in diagnosis of small bowel diseases. Aliment. Pharmacol. Ther. 22(7), 595–604 (2005)CrossRefGoogle Scholar
  23. 23.
    MICCAI 2017 Endoscopic Vision Challenge: Angiodysplasia Detection and Localization,
  24. 24.
    D.K. Iakovidis, A. Koulaouzidis, Software for enhanced video capsule endoscopy: challenges for essential progress. Nat. Rev. Gastroenterol. Hepatol. 12(3), 172 (2015)CrossRefGoogle Scholar
  25. 25.
    M. Mackiewicz, J. Berens, M. Fisher, Wireless capsule endoscopy color video segmentation. IEEE Trans. Med. Imaging 27(12), 1769–1781 (2008)CrossRefGoogle Scholar
  26. 26.
    A. Karargyris, N. Bourbakis, Wireless capsule endoscopy and endoscopic imaging: a survey on various methodologies presented. IEEE Eng. Med. Biol. Mag. 29(1), 72–83 (2010)CrossRefGoogle Scholar
  27. 27.
    P. Szczypiński, A. Klepaczko, M. Pazurek, P. Daniel, Texture and color based image segmentation and pathology detection in capsule endoscopy videos. Comput. Methods Programs Biomed. 113(1), 396–411, (2014),
  28. 28.
    D.S. Mishkin, R. Chuttani, J. Croffie, J. DiSario, J. Liu, R. Shah, L. Somogyi, W. Tierney, L.M.W.K. Song, B.T. Petersen, Asge technology status evaluation report: wireless capsule endoscopy. Gastrointest. Endosc. 63(4), 539–545 (2006)CrossRefGoogle Scholar
  29. 29.
    G. Bradski, The OpenCV Library, in Dr. Dobb’s Journal of Software Tools (2000)Google Scholar
  30. 30.
    B. Münzer, K. Schoeffmann, L. Böszörmenyi, Content-based processing and analysis of endoscopic images and videos: a survey. Multimed. Tools Appl. 77(1), 1323–1362 (2018)CrossRefGoogle Scholar
  31. 31.
    S. Speidel, M. Delles, C. Gutt, R. Dillmann, Tracking of instruments in minimally invasive surgery for surgical skill analysis, in Medical Imaging and Augmented Reality (Springer, Berlin, 2006), pp. 148–155Google Scholar
  32. 32.
    C. Doignon, F. Nageotte, M. De Mathelin, Segmentation and guidance of multiple rigid objects for intra-operative endoscopic vision, in Dynamical Vision. (Springer, Berlin, 2007), pp. 314–327Google Scholar
  33. 33.
    Z. Pezzementi, S. Voros, G.D. Hager, Articulated object tracking by rendering consistent appearance parts, in IEEE International Conference on Robotics and Automation, 2009. ICRA’09. (IEEE, 2009), pp. 3940–3947Google Scholar
  34. 34.
    D. Bouget, R. Benenson, M. Omran, L. Riffaud, B. Schiele, P. Jannin, Detecting surgical tools by modelling local appearance and global shape. IEEE Trans. Med. Imaging 34(12), 2603–2617 (2015)CrossRefGoogle Scholar
  35. 35.
    L.C. García-Peraza-Herrera, W. Li, L. Fidon, C. Gruijthuijsen, A. Devreker, G. Attilakos, J. Deprest, E.B.V. Poorten, D. Stoyanov, T. Vercauteren, S. Ourselin, Toolnet: Holistically-nested real-time segmentation of robotic surgical tools, in Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (2017), pp. 5717–5722Google Scholar
  36. 36.
    M. Attia, M. Hossny, S. Nahavandi, H. Asadi, Surgical tool segmentation using a hybrid deep cnn-rnn auto encoder-decoder, in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (2017), pp. 3373–3378Google Scholar
  37. 37.
    D. Pakhomov, V. Premachandran, M. Allan, M. Azizian, N. Navab, Deep residual learning for instrument segmentation in robotic surgery (2017), arXiv:1703.08580
  38. 38.
    M. Allan, A. Shvets, T. Kurmann, Z. Zhang, R. Duggal, Y.-H. Su, N. Rieke, I. Laina, N. Kalavakonda, S. Bodenstedt, et al., 2017 robotic instrument segmentation challenge (2019), arXiv:1902.06426
  39. 39.
    A. Buslaev, A. Parinov, E. Khvedchenya, V.I. Iglovikov, A.A. Kalinin, Albumentations: fast and flexible image augmentations (2018), arXiv:1809.06839

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  • Alexandr A. Kalinin
    • 1
    Email author
  • Vladimir I. Iglovikov
    • 2
  • Alexander Rakhlin
    • 3
  • Alexey A. Shvets
    • 4
  1. 1.University of MichiganAnn ArborUSA
  2. 2.ODS.aiSan FranciscoUSA
  3. 3.Neuromation OUTallinnEstonia
  4. 4.Massachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations