Skip to main content
SpringerLink
Log in

Deep learning with convolutional neural network in radiology

  • Review
  • Published:
Japanese Journal of Radiology Aims and scope Submit manuscript

Abstract

Deep learning with a convolutional neural network (CNN) is gaining attention recently for its high performance in image recognition. Images themselves can be utilized in a learning process with this technique, and feature extraction in advance of the learning process is not required. Important features can be automatically learned. Thanks to the development of hardware and software in addition to techniques regarding deep learning, application of this technique to radiological images for predicting clinically useful information, such as the detection and the evaluation of lesions, etc., are beginning to be investigated. This article illustrates basic technical knowledge regarding deep learning with CNNs along the actual course (collecting data, implementing CNNs, and training and testing phases). Pitfalls regarding this technique and how to manage them are also illustrated. We also described some advanced topics of deep learning, results of recent clinical studies, and the future directions of clinical application of deep learning techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44.

    Article  CAS  PubMed  Google Scholar 

  2. Fukushima K, Miyake S. Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position. Pattern Recognit. 1982;15:455–69.

    Article  Google Scholar 

  3. Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J Physiol. 1962;160:106–54.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Krizhevsky A, Sutskever I, Hinton G. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing System 25 (NIPS 2012). 2012. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks:Published. Accessed 14 Dec 2017.

  5. Kahn CE Jr. From images to actions: opportunities for artificial intelligence in radiology. Radiology. 2017;285:719–20.

    Article  PubMed  Google Scholar 

  6. Dreyer KJ, Geis JR. When machines think: radiology’s next frontier. Radiology. 2017;285:713–8.

    Article  PubMed  Google Scholar 

  7. Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology. 2016;278:563–77.

    Article  PubMed  Google Scholar 

  8. Skogen K, Ganeshan B, Good C, Critchley G, Miles K. Measurements of heterogeneity in gliomas on computed tomography relationship to tumour grade. J Neurooncol. 2013;111:213–9.

    Article  PubMed  Google Scholar 

  9. Lubner MG, Stabo N, Abel EJ, Del Rio AM, Pickhardt PJ. CT textural analysis of large primary renal cell carcinomas: pretreatment tumor heterogeneity correlates with histologic findings and clinical outcomes. AJR Am J Roentgenol. 2016;207:96–105.

    Article  PubMed  Google Scholar 

  10. Yasaka K, Akai H, Nojima H, Shinozaki-Ushiku A, Fukayama M, Nakajima J, et al. Quantitative computed tomography texture analysis for estimating histological subtypes of thymic epithelial tumors. Eur J Radiol. 2017;92:84–92.

    Article  PubMed  Google Scholar 

  11. Kickingereder P, Burth S, Wick A, Gotz M, Eidel O, Schlemmer HP, et al. Radiomic profiling of glioblastoma: identifying an imaging predictor of patient survival with improved performance over established clinical and radiologic risk models. Radiology. 2016;280:880–9.

    Article  PubMed  Google Scholar 

  12. Huang Y, Liu Z, He L, Chen X, Pan D, Ma Z, et al. Radiomics signature: a potential biomarker for the prediction of disease-free survival in early-stage (I or II) non-small cell lung cancer. Radiology. 2016;281:947–57.

    Article  PubMed  Google Scholar 

  13. Yip C, Landau D, Kozarski R, Ganeshan B, Thomas R, Michaelidou A, et al. Primary esophageal cancer: heterogeneity as potential prognostic biomarker in patients treated with definitive chemotherapy and radiation therapy. Radiology. 2014;270:141–8.

    Article  PubMed  Google Scholar 

  14. Ng F, Ganeshan B, Kozarski R, Miles KA, Goh V. Assessment of primary colorectal cancer heterogeneity by using whole-tumor texture analysis: contrast-enhanced CT texture as a biomarker of 5-year survival. Radiology. 2013;266:177–84.

    Article  PubMed  Google Scholar 

  15. Kiryu S, Akai H, Nojima M, Hasegawa K, Shinkawa H, Kokudo N, et al. Impact of hepatocellular carcinoma heterogeneity on computed tomography as a prognostic indicator. Sci Rep. 2017;7:12689.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Li H, Zhu Y, Burnside ES, Drukker K, Hoadley KA, Fan C, et al. MR imaging radiomics signatures for predicting the risk of breast cancer recurrence as given by research versions of MammaPrint, Oncotype DX, and PAM50 gene assays. Radiology. 2016;281:382–91.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Goh V, Ganeshan B, Nathan P, Juttla JK, Vinayan A, Miles KA. Assessment of response to tyrosine kinase inhibitors in metastatic renal cell cancer: CT texture as a predictive biomarker. Radiology. 2011;261:165–71.

    Article  PubMed  Google Scholar 

  18. Le QV, Ranzato M, Monga R, Devin M, Chen K, Corrado GS, et al. Building high-level features using large scale unsupervised learning. International Conference on Machine Learning. 2012. http://icml.cc/2012/papers. Accessed 14 Dec 2017.

  19. Nakao T, Hanaoka S, Nomura Y, Sato I, Nemoto M, Miki S, et al. Deep neural network-based computer-assisted detection of cerebral aneurysms in MR angiography. J Magn Reson Imaging. 2017. https://doi.org/10.1002/jmri.25842.

    PubMed  Google Scholar 

  20. Gonzalez G, Ash SY, Vegas Sanchez-Ferrero G, Onieva Onieva J, Rahaghi FN, Ross JC, et al. Disease staging and prognosis in smokers using deep learning in chest computed tomography. Am J Respir Crit Care Med. 2018;197:193–203.

    Article  PubMed  Google Scholar 

  21. Nair V, Hinton G. Rectified linear units improve restricted Boltzmann machines. International Conference on Machine Learning. 2010. Accessed 14 Dec 2017.

  22. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.

    Google Scholar 

  23. Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Cornell University Library. 2015. http://arxiv.org/abs/1502.03167. Accessed 30 April 2017.

  24. Banerjee I, Crawley A, Bhethanabotla M, Daldrup-Link HE, Rubin DL. Transfer learning on fused multiparametric MR images for classifying histopathological subtypes of rhabdomyosarcoma. Comput Med Imaging Graph. 2017. https://doi.org/10.1016/j.compmedimag.2017.05.002.

    Google Scholar 

  25. Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res. 2011;12:2121–59.

    Google Scholar 

  26. Kingma DP, Ba JL. Adam: a method for stochastic optimization. Cornell University Library. 2014. http://arxiv.org/abs/1412.6980. Accessed 30 April 2017.

  27. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. Cornell University Library. 2014. https://arxiv.org/abs/1409.4842. Accessed 14 Dec 2017.

  28. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Cornell University Library. 2015. https://arxiv.org/abs/1512.03385. Accessed 14 Dec 2017.

  29. Korfiatis P, Kline TL, Lachance DH, Parney IF, Buckner JC, Erickson BJ. Residual deep convolutional neural network predicts MGMT methylation status. J Digit Imaging. 2017;30:622–8.

    Article  PubMed  Google Scholar 

  30. Andrearczyk V, Whelan PF. Using filter banks in convolutional neural networks for texture classification. Cornell University Library. 2016. https://arxiv.org/abs/1601.02919. Accessed 30 April 2017.

  31. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. 2015. http://ieeexplore.ieee.org/document/7298965/?reload=true. Accessed 14 Dec 2017.

  32. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Cornell University Library. 2015. https://arxiv.org/abs/1505.04597. Accessed 14 Dec 2017.

  33. Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. Cornell University Library. 2015. https://arxiv.org/pdf/1511.00561. Accessed 14 Dec 2017.

  34. Mao X, Shen C, Yang Y. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Cornell University Library. 2016. https://arxiv.org/abs/1603.09056. Accessed 14 Dec 2017.

  35. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284:574–82.

    Article  PubMed  Google Scholar 

  36. Caruana R. Multitask learning. Mach Learn. 1997;28:41–75.

    Article  Google Scholar 

  37. Bengio Y. Deep learning of representations for unsupervised and transfer learning. In: JMLR: Workshop and Conference Proceedings. 2012;17–37.

  38. Mohamed AA, Berg WA, Peng H, Luo Y, Jankowitz RC, Wu S. A deep learning method for classifying mammographic breast density categories. Med Phys. 2018;45:314–21.

    Article  PubMed  Google Scholar 

  39. Larson DB, Chen MC, Lungren MP, Halabi SS, Stence NV, Langlotz CP. Performance of a deep-learning neural network model in assessing skeletal maturity on pediatric hand radiographs. Radiology. 2017. https://doi.org/10.1148/radiol.2017170236.

    Google Scholar 

  40. Cicero M, Bilbily A, Colak E, Dowdell T, Gray B, Perampaladas K, et al. Training and validating a deep convolutional neural network for computer-aided detection and classification of abnormalities on frontal chest radiographs. Invest Radiol. 2017;52:281–7.

    Article  PubMed  Google Scholar 

  41. Becker AS, Marcon M, Ghafoor S, Wurnig MC, Frauenfelder T, Boss A. Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer. Invest Radiol. 2017;52:434–40.

    Article  PubMed  Google Scholar 

  42. Prevedello LM, Erdal BS, Ryu JL, Little KJ, Demirer M, Qian S, et al. Automated critical test findings identification and online notification system using artificial intelligence in imaging. Radiology. 2017;285:923–31.

    Article  PubMed  Google Scholar 

  43. Wang X, Yang W, Weinreb J, Han J, Li Q, Kong X, et al. Searching for prostate cancer by fully automated magnetic resonance imaging classification: deep learning versus non-deep learning. Sci Rep. 2017;7:15415.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Ghafoorian M, Karssemeijer N, Heskes T, Bergkamp M, Wissink J, Obels J, et al. Deep multi-scale location-aware 3D convolutional neural networks for automated detection of lacunes of presumed vascular origin. Neuroimage Clin. 2017;14:391–9.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Jiang H, Ma H, Qian W, Gao M, Li Y. An automatic detection system of lung nodule based on multi-group patch-based deep learning network. IEEE J Biomed Health Inform. 2017. https://doi.org/10.1109/JBHI.2017.2725903.

    Google Scholar 

  46. Parsons DW, Jones S, Zhang X, Lin JC, Leary RJ, Angenendt P, et al. An integrated genomic analysis of human glioblastoma multiforme. Science. 2008;321:1807–12.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  47. Yan H, Parsons DW, Jin G, McLendon R, Rasheed BA, Yuan W, et al. IDH1 and IDH2 mutations in gliomas. N Engl J Med. 2009;360:765–73.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Hegi ME, Diserens AC, Gorlia T, Hamou MF, de Tribolet N, Weller M, et al. MGMT gene silencing and benefit from temozolomide in glioblastoma. N Engl J Med. 2005;352:997–1003.

    Article  CAS  PubMed  Google Scholar 

  49. Chang K, Bai HX, Zhou H, Su C, Bi WL, Agbodza E, et al. Residual convolutional neural network for determination of IDH status in low- and high-grade gliomas from MR imaging. Clin Cancer Res. 2017. https://doi.org/10.1158/1078-0432.CCR-17-2236.

    Google Scholar 

  50. Chi J, Walia E, Babyn P, Wang J, Groot G, Eramian M. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. J Digit Imaging. 2017;30:477–86.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Malempati S, Hawkins DS. Rhabdomyosarcoma: review of the Children’s Oncology Group (COG) Soft-Tissue Sarcoma Committee experience and rationale for current COG studies. Pediatr Blood Cancer. 2012;59:5–10.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging. 2016;35:1207–16.

    Article  PubMed  Google Scholar 

  53. Song Q, Zhao L, Luo X, Dou X. Using deep learning for classification of lung nodules on computed tomography images. J Healthc Eng. 2017;2017:8314740.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Nibali A, He Z, Wollersheim D. Pulmonary nodule classification with deep residual networks. Int J Comput Assist Radiol Surg. 2017;12:1799–808.

    Article  PubMed  Google Scholar 

  55. Yasaka K, Akai H, Kunimatsu A, Abe O, Kiryu S. Liver fibrosis: deep convolutional neural network for staging by using gadoxetic acid-enhanced hepatobiliary phase MR images. Radiology. 2017. https://doi.org/10.1148/radiol.2017171928.

    Google Scholar 

  56. Yasaka K, Akai H, Abe O, Kiryu S. Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology. 2018;286:887–96.

    Article  PubMed  Google Scholar 

  57. Ben-Cohen A, Klang E, Diamant I, Rozendorn N, Raskin SP, Konen E, et al. CT image-based decision support system for categorization of Liver metastases into primary cancer sites: initial results. Acad Radiol. 2017;24:1501–9.

    Article  PubMed  Google Scholar 

  58. Lao J, Chen Y, Li ZC, Li Q, Zhang J, Liu J, et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Sci Rep. 2017;7:10353.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Ibragimov B, Xing L. Segmentation of organs-at-risks in head and neck CT images using convolutional neural networks. Med Phys. 2017;44:547–57.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  60. Ibragimov B, Toesca D, Chang D, Koong A, Xing L. Combining deep learning with anatomy analysis for segmentation of portal vein for liver SBRT planning. Phys Med Biol. 2017;62:8943–58.

    PubMed  Google Scholar 

  61. Men K, Dai J, Li Y. Automatic segmentation of the clinical target volume and organs at risk in the planning CT for rectal cancer using deep dilated convolutional neural networks. Med Phys. 2017;44:6377–89.

    Article  CAS  PubMed  Google Scholar 

  62. Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep learning MR imaging-based attenuation correction for PET/MR imaging. Radiology. 2018;286:676–84.

    Article  PubMed  Google Scholar 

  63. Leynes AP, Yang J, Wiesinger F, Kaushik SS, Shanbhag DD, Seo Y, et al. Direct PseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). J Nucl Med. 2017. https://doi.org/10.2967/jnumed.117.198051.

    PubMed  Google Scholar 

  64. Yasaka K, Katsura M, Akahane M, Sato J, Matsuda I, Ohtomo K. Model-based iterative reconstruction for reduction of radiation dose in abdominopelvic CT: comparison to adaptive statistical iterative reconstruction. Springerplus. 2013;2:209.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Katsura M, Matsuda I, Akahane M, Sato J, Akai H, Yasaka K, et al. Model-based iterative reconstruction technique for radiation dose reduction in chest CT: comparison with the adaptive statistical iterative reconstruction technique. Eur Radiol. 2012;22:1613–23.

    Article  PubMed  Google Scholar 

  66. Pickhardt PJ, Lubner MG, Kim DH, Tang J, Ruma JA, del Rio AM, et al. Abdominal CT with model-based iterative reconstruction (MBIR): initial results of a prospective trial comparing ultralow-dose with standard-dose imaging. AJR Am J Roentgenol. 2012;199:1266–74.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Yamada Y, Jinzaki M, Tanami Y, Shiomi E, Sugiura H, Abe T, et al. Model-based iterative reconstruction technique for ultralow-dose computed tomography of the lung: a pilot study. Invest Radiol. 2012;47:482–9.

    Article  PubMed  Google Scholar 

  68. Deak Z, Grimm JM, Treitl M, Geyer LL, Linsenmaier U, Korner M, et al. Filtered back projection, adaptive statistical iterative reconstruction, and a model-based iterative reconstruction in abdominal CT: an experimental clinical study. Radiology. 2013;266:197–206.

    Article  PubMed  Google Scholar 

  69. Yasaka K, Katsura M, Hanaoka S, Sato J, Ohtomo K. High-resolution CT with new model-based iterative reconstruction with resolution preference algorithm in evaluations of lung nodules: comparison with conventional model-based iterative reconstruction and adaptive statistical iterative reconstruction. Eur J Radiol. 2016;85:599–606.

    Article  PubMed  Google Scholar 

  70. Yasaka K, Furuta T, Kubo T, Maeda E, Katsura M, Sato J, et al. Full and hybrid iterative reconstruction to reduce artifacts in abdominal CT for patients scanned without arm elevation. Acta Radiol. 2017;58:1085–93.

    Article  PubMed  Google Scholar 

  71. Chen H, Zhang Y, Zhang W, Liao P, Li K, Zhou J, et al. Low-dose CT via convolutional neural network. Biomed Opt Express. 2017;8:679–94.

    Article  PubMed  PubMed Central  Google Scholar 

  72. Chen H, Zhang Y, Kalra MK, Lin F, Chen Y, Liao P, et al. Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans Med Imaging. 2017;36:2524–35.

    Article  PubMed  Google Scholar 

  73. Mackin D, Fave X, Zhang L, Fried D, Yang J, Taylor B, et al. Measuring computed tomography scanner variability of radiomics features. Invest Radiol. 2015;50:757–65.

    Article  PubMed  PubMed Central  Google Scholar 

  74. Yasaka K, Akai H, Mackin D, Court L, Moros E, Ohtomo K, et al. Precision of quantitative computed tomography texture analysis using image filtering: a phantom study for scanner variability. Medicine (Baltimore). 2017;96:e6993.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

No specific funding was disclosed.

Author information

Authors and Affiliations

  1. Department of Radiology, The Institute of Medical Science, The University of Tokyo, 4-6-1 Shirokanedai, Minato-ku, Tokyo, 108-8639, Japan

    Koichiro Yasaka, Hiroyuki Akai & Akira Kunimatsu

  2. Department of Radiology, Graduate School of Medical Sciences, International University of Health and Welfare, 4-3 Kozunomori, Narita, Chiba, Japan

    Shigeru Kiryu

  3. Department of Radiology, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan

    Osamu Abe

Authors
  1. Koichiro Yasaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hiroyuki Akai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Akira Kunimatsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shigeru Kiryu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Osamu Abe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Koichiro Yasaka.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical statement

This article does not contain any studies with human participants or animals performed by any of the authors.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yasaka, K., Akai, H., Kunimatsu, A. et al. Deep learning with convolutional neural network in radiology. Jpn J Radiol 36, 257–272 (2018). https://doi.org/10.1007/s11604-018-0726-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11604-018-0726-3

Keywords

Search

Navigation