Deep Learning with Convolutional Neural Networks for Histopathology Image Analysis

  • Dragan BošnačkiEmail author
  • Natal van Riel
  • Mitko Veta
Part of the Computational Biology book series (COBO, volume 30)


In the recent years, deep learning based methods and, in particular, convolutional neural networks, have been dominating the arena of medical image analysis. This has been made possible both with the advent of new parallel hardware and the development of efficient algorithms. It is expected that future advances in both of these directions will increase this domination. The application of deep learning methods to medical image analysis has been shown to significantly improve the accuracy and efficiency of the diagnoses. In this chapter, we focus on applications of deep learning in microscopy image analysis and digital pathology, in particular. We provide an overview of the state-of-the-art methods in this area and exemplify some of the main techniques. Finally, we discuss some open challenges and avenues for future work.



The authors would like to thank the anonymous reviewers as well as Stojan Trajanovski for their comments and suggestions that contributed to the final version of this paper.


  1. 1.
    15th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2015, Seoul, South Korea, 3–5 November 2015. IEEE (2015).
  2. 2.
    IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June 2015. IEEE Computer Society (2015).
  3. 3.
    Arbelle A, Raviv TR (2018) Microscopy cell segmentation via adversarial neural networks. In: 15th IEEE International symposium on biomedical imaging, ISBI 2018, Washington, DC, USA, 4–7 April 2018. IEEE, pp 645–648.
  4. 4.
    Arbelle A, Raviv TR (2018) Microscopy cell segmentation via convolutional LSTM networks. CoRR. arXiv:1895.11247
  5. 5.
    Ehteshami Bejnordi B, Veta M, Johannes van Diest P et al (2017) Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22):2199–2210. Scholar
  6. 6.
    Bejnordi BE, Litjens GJS, Timofeeva N, Otte-Holler I, Homeyer A, Karssemeijer N, van der Laak JAWM (2016) Stain specific standardization of whole-slide histopathological images. IEEE Trans Med Imag 35(2):404–415. Scholar
  7. 7.
    Bejnordi BE, Zuidhof G, Maschenka Balkenhol MH, Bult P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J (2017) Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J Med Imag 4:4–8. Scholar
  8. 8.
    Bekkers EJ, Lafarge MW, Veta M, Eppenhof KAJ, Pluim JPW, Duits R (2018) Roto-translation covariant convolutional networks for medical image analysis. CoRR. arXiv:1804.03393
  9. 9.
    Christ PF, Ettlinger F, Grün F, Elshaer MEA, Lipková J, Schlecht S, Ahmaddy F, Tatavarty S, Bickel M, Bilic P, Rempfler M, Hofmann F, D’Anastasi M, Ahmadi S, Kaissis G, Holch J, Sommer WH, Braren R, Heinemann V, Menze BH (2017) Automatic liver and tumor segmentation of CT and MRI volumes using cascaded fully convolutional neural networks. CoRR arXiv:1702.05970
  10. 10.
    Christiansen EM, Yang SJ, Ando DM, Javaherian A, Skibinski G, Lipnick S, Mount E, O’Neil A, Shah K, Lee AK, Goyal P, Fedus W, Poplin R, Esteva A, Berndl M, Rubin LL, Nelson P, Finkbeiner S (2018) In silico labeling: predicting fluorescent labels in unlabeled images. Cell 173(3):792–803.e19. Scholar
  11. 11.
    Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. CoRR. arXiv:1606.06650
  12. 12.
    Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J (2012) Deep neural networks segment neuronal membranes in electron microscopy images. In: Bartlett PL, Pereira FCN, Burges CJC, Bottou L, Weinberger KO (eds.) Advances in Neural information processing systems 25: 26th annual conference on neural information processing systems 2012. Proceedings of a meeting held 3–6 December 2012, Lake Tahoe, Nevada, United States, pp 2852–2860.
  13. 13.
    Ciresan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N (eds.) Medical image computing and computer-assisted intervention - MICCAI 2013 - 16th international conference, Nagoya, Japan, September 22-26, 2013, Proceedings, Part II, Lecture Notes in Computer Science, vol 8150. Springer, pp 411–418. Scholar
  14. 14.
    Codella NCF, Anderson D, Philips T, Porto A, Massey K, Snowdon J, Feris RS, Smith JR (2018) Segmentation of both diseased and healthy skin from clinical photographs in a primary care setting. CoRR. arXiv:1804.05944
  15. 15.
    Cruz-Roa A, Gilmore H, Basavanhally A, Feldman M, Ganesan S, Shih NNC, Tomaszewski J, González FA, Madabhushi A (2017) Accurate and reproducible invasive breast cancer detection in whole-slide images: a deep learning approach for quantifying tumor extent. Scientif Rep 7:46450 EP.
  16. 16.
    Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Darrell T, Saenko K (2015) Long-term recurrent convolutional networks for visual recognition and description. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, 7–12 June 2015, vol 2, pp 2625–2634.
  17. 17.
    Dozat T (2015) Incorporating nesterov momentum into adamGoogle Scholar
  18. 18.
    Drozdzal M, Vorontsov E, Chartrand G, Kadoury S, Pal C (2016) The importance of skip connections in biomedical image segmentation. CoRR. arXiv:1608.04117
  19. 19.
    Fukushima K (1980) Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol Cybernet 36:193–202CrossRefGoogle Scholar
  20. 20.
    Fukushima K, Miyake S (1982) Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recogn 15(6):455–469. Scholar
  21. 21.
    Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky VS (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17, 59:1–59:35.
  22. 22.
    Goldsborough P, Pawlowski N, Caicedo JC, Singh S, Carpenter A (2017) Cytogan: generative modeling of cell images. bioRxiv.,
  23. 23.
    Goodfellow IJ, Bengio Y, Courville AC (2016) Deep learning: adaptive computation and machine learning. MIT Press.
  24. 24.
    He K, Gkioxari G, Dollár P, Girshick RB (2017) Mask R-CNN. In: IEEE international conference on computer vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp 2980–2988. IEEE Computer Society.
  25. 25.
    He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR. arXiv:1512.03385
  26. 26.
    Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. Scholar
  27. 27.
    Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017. IEEE Computer Society, pp 2261–2269.
  28. 28.
    Johnson JW (2018) Adapting mask-rcnn for automatic nucleus segmentation. CoRR. arXiv:1805.00500
  29. 29.
    Katz G, Barrett C, Dill DL, Julian K, Kochenderfer MJ (2017) Towards proving the adversarial robustness of deep neural networks. In: Bulwahn L, Kamali M, Linker S (eds.) Proceedings first workshop on formal verification of autonomous vehicles, FVAV@iFM 2017, Turin, Italy, 19th September 2017. EPTCS, vol 257, pp 19–26. Scholar
  30. 30.
    Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR arXiv:1412.6980
  31. 31.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proceedings of the 25th international conference on neural information processing systems, vol 1, NIPS’12, pp 1097–1105. Curran Associates Inc., USA.
  32. 32.
    Lafarge MW, Pluim JPW, Eppenhof KAJ, Moeskops P, Veta M (2017) Domain-adversarial neural networks to address the appearance variability of histopathology images. CoRR. arXiv:1707.06183
  33. 33.
    Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition.,
  34. 34.
    Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T (eds) Computer Vision - ECCV 2014. Springer International Publishing, Cham, pp 740–755Google Scholar
  35. 35.
    Litjens GJS, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Imag Anal 42:60–88CrossRefGoogle Scholar
  36. 36.
    Lo SCB, Lou SLA, Lin JS, Freedman MT, Chien MV, Mun SK (1995) Artificial convolution neural network techniques and applications for lung nodule detection. IEEE Trans Med Imag 14(4):711–718. Scholar
  37. 37.
    Milletari F, Navab N, Ahmadi S (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. CoRR arXiv:1606.04797
  38. 38.
    Mobadersany P, Yousefi S, Amgad M, Gutman DA, Barnholtz-Sloan JS, Velázquez Vega JE, Brat DJ, Cooper LAD (2018) Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Nat Acad Sci 115(13):E2970–E2979., Scholar
  39. 39.
    Paeng K, Hwang S, Park S, Kim M (2017) A unified framework for tumor proliferation score prediction in breast histopathology. In: Cardoso MJ, Arbel T, Carneiro G, Syeda-Mahmood TF, Tavares JMRS, Moradi M, Bradley AP, Greenspan H, Papa JP, Madabhushi A, Nascimento JC, Cardoso JS, Belagiannis V, Lu Z (eds.) Deep learning in medical image analysis and multimodal learning for clinical decision support - Third international workshop, DLMIA 2017, and 7th international workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, 2017, Proceedings, Lecture Notes in Computer Science, vol 10553. Springer, pp 231–239. Scholar
  40. 40.
    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. CoRR arXiv:1505.04597
  41. 41.
    Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221–248. [pmid]CrossRefGoogle Scholar
  42. 42.
    Shi X, Chen Z, Wang H, Yeung D, Wong W, Woo W (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. CoRR. arXiv:1506.04214
  43. 43.
    Sundermann B, Feder S, Wersching H, Teuber A, Schwindt W, Kugel H, Heindel W, Arolt V, Berger K, Pfleiderer B (2017) Diagnostic classification of unipolar depression based on resting-state functional connectivity MRI: effects of generalization to a diverse sample. J Neural Trans 124(5):589–605. Scholar
  44. 44.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: IEEE conference on computer vision and pattern recognition, CVPR 2015, Boston, MA, USA, June 7–12 2015, pp. 1–9.
  45. 45.
    Trajanovski S, Mavroeidis D, Swisher CL, Gebre BG, Veeling B, Wiemker R, Klinder T, Tahmasebi A, Regis SM, Wald C, McKee BJ, MacMahon H, Pien H (2018) Towards radiologist-level cancer risk assessment in CT lung screening using deep learning. CoRR. arXiv:1804.01901
  46. 46.
    Vanschoren J, van Rijn JN, Bischl B (2015) Taking machine learning research online with openml. In: Proceedings of the 4th international workshop on big data, streams and heterogeneous source mining: algorithms, systems, programming models and applications, BigMine 2015, Sydney, Australia, August 10 2015. JMLR Workshop and Conference Proceedings, vol 41, pp 1–4.
  47. 47.
    Veličković P, Wang D, Lane ND, Liò P (2016) X-cnn: cross-modal convolutional neural networks for sparse datasets. In: 2016 IEEE symposium series on computational intelligence (SSCI), pp 1–8.
  48. 48.
    Wainberg M, Merico D, Delong A, Frey BJ (2018) Deep learning in biomedicine. Nat Biotechnol 36:829 EP. Scholar
  49. 49.
    Wang D, Khosla A, Gargeya R, Irshad H, Beck AH (2016) Deep learning for identifying metastatic breast cancer. CoRR. arXiv:1606.05718
  50. 50.
    Xu Y, Li Y, Liu M, Wang Y, Lai M, Chang EI (2016) Gland instance segmentation by deep multichannel side supervision. CoRR. arXiv:1607.03222

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Biomedical Engineering DepartmentEindhoven University of TechnologyEindhovenThe Netherlands

Personalised recommendations