Deep Learning in Medical Image Analysis

  • Heang-Ping ChanEmail author
  • Ravi K. Samala
  • Lubomir M. Hadjiiski
  • Chuan Zhou
Part of the Advances in Experimental Medicine and Biology book series (AEMB, volume 1213)


Deep learning is the state-of-the-art machine learning approach. The success of deep learning in many pattern recognition applications has brought excitement and high expectations that deep learning, or artificial intelligence (AI), can bring revolutionary changes in health care. Early studies of deep learning applied to lesion detection or classification have reported superior performance compared to those by conventional techniques or even better than radiologists in some tasks. The potential of applying deep-learning-based medical image analysis to computer-aided diagnosis (CAD), thus providing decision support to clinicians and improving the accuracy and efficiency of various diagnostic and treatment processes, has spurred new research and development efforts in CAD. Despite the optimism in this new era of machine learning, the development and implementation of CAD or AI tools in clinical practice face many challenges. In this chapter, we will discuss some of these issues and efforts needed to develop robust deep-learning-based CAD tools and integrate these tools into the clinical workflow, thereby advancing towards the goal of providing reliable intelligent aids for patient care.


Machine learning Deep learning Artificial intelligence Computer-aided diagnosis Medical imaging Big data Transfer learning Validation Quality assurance Interpretable AI 



This work is supported by National Institutes of Health award number R01 CA214981.


The authors have no conflicts to disclose.


  1. 1.
    Winsberg F, Elkin M, Macy J, Bordaz V, Weymouth W (1967) Detection of radiographic abnormalities in mammograms by means of optical scanning and computer analysis. Radiology 89:211–215CrossRefGoogle Scholar
  2. 2.
    Kimme C, O’Laughlin BJ, Sklansky J (1977) Automatic detection of suspicious abnormalities in breast radiographs. Data structures, computer graphics and pattern recognition. Academic Press, New YorkGoogle Scholar
  3. 3.
    Spiesberger W (1979) Mammogram inspection by computer. IEEE Trans Biomed Eng 26:213–219PubMedCrossRefPubMedCentralGoogle Scholar
  4. 4.
    Semmlow JL, Shadagopappan A, Ackerman LV, Hand W, Alcorn FS (1980) A fully automated system for screening mammograms. Comput Biomed Res 13:350–362PubMedCrossRefPubMedCentralGoogle Scholar
  5. 5.
    Doi K (2015) Chapter 1. Historical overview. In: Li Q, Nishikawa RM (eds) Computer-aided detection and diagnosis in medical imaging. Taylor & Francis Group, LLC, CRC Press, Boca Raton, FL, pp 1–17Google Scholar
  6. 6.
    Chan H-P, Doi K, Galhotra S, Vyborny CJ, MacMahon H, Jokich PM (1987) Image feature analysis and computer-aided diagnosis in digital radiography. 1. Automated detection of microcalcifications in mammography. Med Phys 14:538–548PubMedCrossRefPubMedCentralGoogle Scholar
  7. 7.
    Chan H-P, Doi K, Vyborny CJ, Schmidt RA, Metz CE, Lam KL et al (1990) Improvement in radiologists’ detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis. Investig Radiol 25:1102–1110CrossRefGoogle Scholar
  8. 8.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444PubMedPubMedCentralCrossRefGoogle Scholar
  9. 9.
    Fukushima K, Miyake S (1982) Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recogn 15:455–469CrossRefGoogle Scholar
  10. 10.
    LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W et al (1990) Handwritten digit recognition with a back-propagation network. Proc Adv Neural Inf Process Syst:396–404Google Scholar
  11. 11.
    Lo SCB, Lin JS, Freedman MT, Mun SK (1993) Computer-assisted diagnosis of lung nodule detection using artificial convolution neural network. Proc SPIE 1898:859–869CrossRefGoogle Scholar
  12. 12.
    Lo SCB, Chan H-P, Lin JS, Li H, Freedman M, Mun SK (1995) Artificial convolution neural network for medical image pattern recognition. Neural Netw 8:1201–1214CrossRefGoogle Scholar
  13. 13.
    Chan H-P, Lo SCB, Helvie MA, Goodsitt MM, Cheng SNC, Adler DD (1993) Recognition of mammographic microcalcifications with artificial neural network. Radiology 189(P):318Google Scholar
  14. 14.
    Chan H-P, Lo SCB, Sahiner B, Lam KL, Helvie MA (1995) Computer-aided detection of mammographic microcalcifications: pattern recognition with an artificial neural network. Med Phys 22:1555–1567PubMedCrossRefGoogle Scholar
  15. 15.
    Chan H-P, Sahiner B, Lo SCB, Helvie MA, Petrick N, Adler DD et al (1994) Computer-aided diagnosis in mammography: detection of masses by artificial neural network. Med Phys 21:875–876CrossRefGoogle Scholar
  16. 16.
    Sahiner B, Chan H-P, Petrick N, Wei D, Helvie MA, Adler DD et al (1995) Image classification using artificial neural networks. Proc SPIE 2434:838–845CrossRefGoogle Scholar
  17. 17.
    Wei D, Sahiner B, Chan H-P, Petrick N (1995) Detection of masses on mammograms using a convolution neural network. In: Proceedings of the International Conference on Acoustics, Speech and Signal Processing. 95CH2431-5, pp 3483–3486Google Scholar
  18. 18.
    Sahiner B, Chan H-P, Petrick N, Wei D, Helvie MA, Adler DD et al (1996) Classification of mass and normal breast tissue: a convolution neural network classifier with spatial domain and texture images. IEEE Trans Med Imaging 15:598–610PubMedCrossRefGoogle Scholar
  19. 19.
    Zhang W, Doi K, Giger ML, Wu Y, Nishikawa RM, Schmidt RA (1994) Computerized detection of clustered microcalcifications in digital mammograms using a shift-invariant artificial neural network. Med Phys 21:517–524PubMedCrossRefGoogle Scholar
  20. 20.
    Hinton GE, Osindero S, Teh Y-W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554CrossRefGoogle Scholar
  21. 21.
    Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. Proc Adv Neural Inf Process Syst 19:153–160Google Scholar
  22. 22.
    Erhan D, Bengio Y, Courville A, Manzagol P-A, Vincent P, Bengio S (2010) Why does unsupervised pre-training help deep learning? J Mach Learn Res 11:625–660Google Scholar
  23. 23.
    Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, pp 807–814Google Scholar
  24. 24.
    Glorot X, Bordes A, Yoshua B (2011) Deep sparse rectifier neural networks. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp 315–323Google Scholar
  25. 25.
    Ranzato MA, Huang FJ, Boureau Y-L, LeCun Y (eds) (2007) Unsupervised learning of invariant feature hierarchies with applications to object recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 17–22 June 2007. Minneapolis, MN, USAGoogle Scholar
  26. 26.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958Google Scholar
  27. 27.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning (ICML’15), vol 37, pp 448–456. arXiv:1502.03167Google Scholar
  28. 28.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst:1097–1105Google Scholar
  29. 29.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252CrossRefGoogle Scholar
  30. 30.
    He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770–778. arXiv:1512.03385Google Scholar
  31. 31.
    Sun C, Shrivastava A, Singh S, Gupta A (2017) Revisiting unreasonable effectiveness of data in deep learning era. arXiv:1707.02968Google Scholar
  32. 32.
    Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88CrossRefGoogle Scholar
  33. 33.
    Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH et al (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46(1):e1–e36PubMedPubMedCentralCrossRefGoogle Scholar
  34. 34.
    Mazurowski MA, Buda M, Saha A, Bashir MR (2019) Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI. J Magn Reson Imaging 49(4):939–954PubMedCrossRefGoogle Scholar
  35. 35.
    Fauw JD, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S et al (2018) Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 24(9):1342–1350PubMedCrossRefGoogle Scholar
  36. 36.
    Janowczyk A, Madahushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7:29PubMedPubMedCentralCrossRefGoogle Scholar
  37. 37.
    Chan H-P, Hadjiiski LM, Samala RK (2019) Computer-aided diagnosis in the era of deep learning. Med Phys (accepted)Google Scholar
  38. 38.
    Cole EB, Zhang Z, Marques HS, Hendrick RE, Yaffe MJ, Pisano ED (2014) Impact of computer-aided detection systems on radiologist accuracy with digital mammography. AJR Am J Roentgenol 203:909–916PubMedPubMedCentralCrossRefGoogle Scholar
  39. 39.
    Gilbert FJ, Astley SM, Gillan MGC, Agbaje OF, Wallis MG, James J et al (2008) Single reading with computer-aided detection for screening mammography. N Engl J Med 359(16):1675–1684PubMedCrossRefGoogle Scholar
  40. 40.
    Gromet M (2008) Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol 190(4):854–859. CrossRefGoogle Scholar
  41. 41.
    Taylor P, Potts HWW (2008) Computer aids and human second reading as interventions in screening mammography: two systematic reviews to compare effects on cancer detection and recall rate. Eur J Cancer 44:798–807PubMedCrossRefPubMedCentralGoogle Scholar
  42. 42.
    Fenton JJ, Abraham L, Taplin SH, Geller BM, Carney PA, D’Orsi C et al (2011) Effectiveness of computer-aided detection in community mammography practice. J Natl Cancer Inst 103(15):1152–1161. CrossRefPubMedPubMedCentralGoogle Scholar
  43. 43.
    Lehman CD, Wellman RD, Buist DSM, Kerlikowske K, Tosteson ANA, Miglioretti DL (2015) Diagnostic accuracy of digital screening mammography with and without computer-aided detection. JAMA Intern Med 175(11):1828–1837PubMedPubMedCentralCrossRefGoogle Scholar
  44. 44.
    Zech J, Pain M, Titano J, Badgeley M, Schefflein J, Su A et al (2018) Natural language-based machine learning models for the annotation of clinical radiology reports. Radiology 287(2):570–580. CrossRefPubMedPubMedCentralGoogle Scholar
  45. 45.
    Yan K, Wang X, Lu L, Summers RM (2018) DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. J Med Imaging 5(3):036501CrossRefGoogle Scholar
  46. 46.
    Oakden-Rayner L (2017) Exploring the ChestXray14 dataset: problems.https://lukeoakdenrayner.word
  47. 47.
    The Digital mammograph DREAM Challenge (2017).Google Scholar
  48. 48.
    Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Proceedings of the Advances in neural information processing systems (NIPS’14), pp 3320–3328Google Scholar
  49. 49.
    Samala RK, Chan H-P, Hadjiiski LM, Helvie MA, Richter CD, Cha K (2019) Breast cancer diagnosis in digital breast tomosynthesis: effects of training sample size on multi-stage transfer learning using deep neural nets. IEEE Trans Med Imaging 38(3):686–696PubMedCrossRefPubMedCentralGoogle Scholar
  50. 50.
    Taylor L, Nitschke G (2017) Improving deep learning using generic data augmentation. arXiv:1708.06020Google Scholar
  51. 51.
    Wang J, Perez L (2017) The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621Google Scholar
  52. 52.
    Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozairy S et al (2014) Generative Adversarial Nets. arXiv:1406.2661v1Google Scholar
  53. 53.
    Badano A, Graff CG, Badal A et al (2018) Evaluation of digital breast tomosynthesis as replacement of full-field digital mammography using an in silico imaging trial. JAMA Network Open 1(7):e185474. CrossRefPubMedPubMedCentralGoogle Scholar
  54. 54.
    Cha KH, Petrick N, Pezeshk A, Graff CG, Sharma D, Badal A et al (2019) Reducing overfitting of a deep learning breast mass detection algorithm in mammography using synthetic images. Proc SPIE 10950:1095004Google Scholar
  55. 55.
    Pezeshk A, Petrick N, Chen W, Sahiner B (2017) Seamless lesion insertion for data augmentation in CAD training. IEEE Trans Med Imaging 36(4):1005–1015PubMedCrossRefPubMedCentralGoogle Scholar
  56. 56.
    Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78:316–331CrossRefGoogle Scholar
  57. 57.
    Fukunaga K, Hayes RR (1989) Effects of sample size on classifier design. IEEE Trans Pattern Anal Mach Intell 11:873–885CrossRefGoogle Scholar
  58. 58.
    Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning. Springer-Verlag, New YorkCrossRefGoogle Scholar
  59. 59.
    Bishop CM (1995) Neural networks for pattern recognition. Clarendon Press, OxfordGoogle Scholar
  60. 60.
    Chan H-P, Sahiner B, Wagner RF, Petrick N (1999) Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiers. Med Phys 26:2654–2668PubMedCrossRefPubMedCentralGoogle Scholar
  61. 61.
    Sahiner B, Chan H-P, Petrick N, Wagner RF, Hadjiiski LM (2000) Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys 27:1509–1522PubMedPubMedCentralCrossRefGoogle Scholar
  62. 62.
    Petrick N, Sahiner B, Armato SG, Bert A, Correale L, Delsanto S et al (2013) Evaluation of computer-aided detection and diagnosis systems. Med Phys 40(8):087001. CrossRefPubMedPubMedCentralGoogle Scholar
  63. 63.
    Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK (2018) Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLOS Med 15(11):e1002683PubMedPubMedCentralCrossRefGoogle Scholar
  64. 64.
    Conant EF, Toledano AY, Periaswamy S, Fotin SV, Go J, Hoffmeister JW et al (2018) Improving accuracy and efficiency with concurrent use of artificial intelligence for digital breast tomosynthesis screening. In: Radiological Society of North America Scientific Assembly and Annual Meeting. RC215-14Google Scholar
  65. 65.
    Kyono T, Gilbert FJ, van der Schaar M (2019) Improving workflow efficiency for mammography using machine learning. J Am Coll Radiol. PubMedCrossRefPubMedCentralGoogle Scholar
  66. 66.
    Huo ZM, Summers RM, Paquerault S, Lo J, Hoffmeister J, Armato SG et al (2013) Quality assurance and training procedures for computer-aided detection and diagnosis systems in clinical use. Med Phys 40(7):077001. CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Lecture Notes in Computer Science Computer Vision – European Conference on Computer Vision (ECCV) 2014, vol 8689, pp 818–833CrossRefGoogle Scholar
  68. 68.
    Yosinski J, Clune J, Nguyen A, Fuchs T, Lipson H (2015) Understanding neural networks through deep visualization. arXiv:1506.06579v1Google Scholar
  69. 69.
    Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2921–2929Google Scholar
  70. 70.
    Samala RK, Chan H-P, Hadjiiski LM, Helvie MA, Cha KH, Richter CD (2017) Multi-task transfer learning deep convolutional neural network: application to computer-aided diagnosis of breast cancer on mammograms. Phys Med Biol 62:8894–8908PubMedPubMedCentralCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Heang-Ping Chan
    • 1
    Email author
  • Ravi K. Samala
    • 1
  • Lubomir M. Hadjiiski
    • 1
  • Chuan Zhou
    • 1
  1. 1.Department of RadiologyUniversity of MichiganAnn ArborUSA

Personalised recommendations