Nuclear Medicine and Molecular Imaging

, Volume 52, Issue 2, pp 109–118 | Cite as

Deep Learning in Nuclear Medicine and Molecular Imaging: Current Perspectives and Future Directions

  • Hongyoon Choi


Recent advances in deep learning have impacted various scientific and industrial fields. Due to the rapid application of deep learning in biomedical data, molecular imaging has also started to adopt this technique. In this regard, it is expected that deep learning will potentially affect the roles of molecular imaging experts as well as clinical decision making. This review firstly offers a basic overview of deep learning particularly for image data analysis to give knowledge to nuclear medicine physicians and researchers. Because of the unique characteristics and distinctive aims of various types of molecular imaging, deep learning applications can be different from other fields. In this context, the review deals with current perspectives of deep learning in molecular imaging particularly in terms of development of biomarkers. Finally, future challenges of deep learning application for molecular imaging and future roles of experts in molecular imaging will be discussed.


Deep learning Molecular imaging Machine learning Convolutional neural network Precision medicine 


Compliance with Ethical Standards

Conflict of Interest

Hongyoon Choi declares no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

For this study formal consent is not required.


  1. 1.
    Weber GM, Mandl KD, Kohane IS. Finding the missing link for big biomedical data. JAMA. 2014;311:2479–80.PubMedGoogle Scholar
  2. 2.
    Kuo M-H, Sahama T, Kushniruk AW, Borycki EM, Grunwell DK. Health big data analytics: current perspectives, challenges and potential solutions. Int J Big Data Intell. 2014;1:114–26.CrossRefGoogle Scholar
  3. 3.
    Bengio Y. Learning deep architectures for AI. Foundations and trends® in. Mach Learn. 2009;2:1–127.CrossRefGoogle Scholar
  4. 4.
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–44.CrossRefPubMedGoogle Scholar
  5. 5.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. Imagenet large scale visual recognition challenge. Int J Comput Vis. 2015;115:211–52.CrossRefGoogle Scholar
  6. 6.
    Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst. 2012;25:1090–8.Google Scholar
  7. 7.
    Alipanahi B, Delong A, Weirauch MT, Frey BJ. Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nat Biotechnol. 2015;33:831–8.CrossRefPubMedGoogle Scholar
  8. 8.
    Zhou J, Troyanskaya OG. Predicting effects of noncoding variants with deep learning–based sequence model. Nat Methods. 2015;12:931.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M et al. A survey on deep learning in medical image analysis. arXiv:170205747. 2017.Google Scholar
  10. 10.
    Xu Y, Mo T, Feng Q, Zhong P, Lai M, Eric I et al., editors. Deep learning of feature representation with multiple instance learning for medical image analysis. Acoustics, Speech and Signal Processing (ICASSP), 2014 I.E. International Conference; 2014.Google Scholar
  11. 11.
    Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, et al. Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans Med Imaging. 2016;35:1299–312.CrossRefPubMedGoogle Scholar
  12. 12.
    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–10.CrossRefPubMedGoogle Scholar
  13. 13.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–8.CrossRefPubMedGoogle Scholar
  14. 14.
    Mamoshina P, Vieira A, Putin E, Zhavoronkov A. Applications of deep learning in biomedicine. Mol Pharm. 2016;13:1445–54.CrossRefPubMedGoogle Scholar
  15. 15.
    Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. Cogn Model. 1988;5:1.Google Scholar
  16. 16.
    Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. Adv Neural Inf Proces Syst. 2006;19:153–160.Google Scholar
  17. 17.
    Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006;313:504–7.CrossRefPubMedGoogle Scholar
  18. 18.
    Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. 2014;15:1929–58.Google Scholar
  19. 19.
    Ioffe S, Szegedy C, editors. Batch normalization: accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning; 2015.Google Scholar
  20. 20.
    Nair V, Hinton GE, editors. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10); 2010.Google Scholar
  21. 21.
    Oquab M, Bottou L, Laptev I, Sivic J, editors. Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2014.Google Scholar
  22. 22.
    LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86:2278–324.CrossRefGoogle Scholar
  23. 23.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D et al., editors. Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition; 2015.Google Scholar
  24. 24.
    He K, Zhang X, Ren S, Sun J, editors. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016.Google Scholar
  25. 25.
    Vinyals O, Toshev A, Bengio S, Erhan D, editors. Show and tell: a neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015.Google Scholar
  26. 26.
    Girshick R, editor. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision; 2015.Google Scholar
  27. 27.
    Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. Adv Neural Inf Proces Syst. 2015;28:91–9.Google Scholar
  28. 28.
    Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv:160600915. 2016.Google Scholar
  29. 29.
    Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:151100561. 2015.Google Scholar
  30. 30.
    Elmore JG, Longton GM, Carney PA, Geller BM, Onega T, Tosteson AN, et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA. 2015;313:1122–32.CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Keyes JW Jr. SUV: standard uptake or silly useless value? J Nucl Med. 1995;36:1836–9.PubMedGoogle Scholar
  32. 32.
    Wang H, Zhou Z, Li Y, Chen Z, Lu P, Wang W, et al. Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18 F-FDG PET/CT images. EJNMMI Res. 2017;7:11.CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, et al. Brain tumor segmentation with deep neural networks. Med Image Anal. 2017;35:18–31.CrossRefPubMedGoogle Scholar
  34. 34.
    Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, et al. Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage. 2015;108:214–24.CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Roth HR, Lu L, Farag A, Shin H-C, Liu J, Turkbey EB et al., editors. Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin: Springer; 2015.Google Scholar
  36. 36.
    Choi H, Jin KH. Fast and robust segmentation of the striatum using deep convolutional neural networks. J Neurosci Methods. 2016;274:146–53.CrossRefPubMedGoogle Scholar
  37. 37.
    Shen W, Zhou M, Yang F, Yang C, Tian J, editors. Multi-scale convolutional neural networks for lung nodule classification. International Conference on Information Processing in Medical Imaging. Berlin: Springer; 2015.Google Scholar
  38. 38.
    Setio AAA, Ciompi F, Litjens G, Gerke P, Jacobs C, van Riel SJ, et al. Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks. IEEE Trans Med Imaging. 2016;35:1160–9.CrossRefPubMedGoogle Scholar
  39. 39.
    Kleesiek J, Urban G, Hubert A, Schwarz D, Maier-Hein K, Bendszus M, et al. Deep MRI brain extraction: a 3D convolutional neural network for skull stripping. NeuroImage. 2016;129:460–9.CrossRefPubMedGoogle Scholar
  40. 40.
    de Brebisson A, Montana G, editors. Deep neural networks for anatomical brain segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops; 2015.Google Scholar
  41. 41.
    Chen H, Dou Q, Yu L, Heng P-A. Voxresnet: deep voxelwise residual networks for volumetric brain segmentation. arXiv:160805895. 2016.Google Scholar
  42. 42.
    Avendi M, Kheradvar A, Jafarkhani H. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Med Image Anal. 2016;30:108–19.CrossRefPubMedGoogle Scholar
  43. 43.
    Charbonnier J-P, Van Rikxoort EM, Setio AA, Schaefer-Prokop CM, van Ginneken B, Ciompi F. Improving airway segmentation in computed tomography using leak detection with convolutional networks. Med Image Anal. 2017;36:52–60.CrossRefPubMedGoogle Scholar
  44. 44.
    Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging. 2016;35:1240–51.CrossRefPubMedGoogle Scholar
  45. 45.
    Ghafoorian M, Karssemeijer N, Heskes T, van Uden IW, Sanchez CI, Litjens G, et al. Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Sci Rep. 2017;7.
  46. 46.
    Trebeschi S, van Griethuysen JJ, Lambregts DM, Lahaye MJ, Parmer C, Bakers FC, et al. Deep learning for fully-automated localization and segmentation of rectal cancer on multiparametric MR. Sci Rep. 2017;7Google Scholar
  47. 47.
    Han D, Yu J, Yu Y, Zhang G, Zhong X, Lu J, et al. Comparison of 18 F-fluorothymidine and 18 F-fluorodeoxyglucose PET/CT in delineating gross tumor volume by optimal threshold in patients with squamous cell carcinoma of thoracic esophagus. Int J Radiat Oncol Biol Phys. 2010;76:1235–41.CrossRefPubMedGoogle Scholar
  48. 48.
    Tylski P, Stute S, Grotus N, Doyeux K, Hapdey S, Gardin I, et al. Comparative assessment of methods for estimating tumor volume and standardized uptake value in 18F-FDG PET. J Nucl Med. 2010;51:268–76.CrossRefPubMedGoogle Scholar
  49. 49.
    Dong C, Loy CC, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell. 2016;38:295–307.CrossRefPubMedGoogle Scholar
  50. 50.
    Kim J, Kwon Lee J, Mu Lee K, editors. Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016.Google Scholar
  51. 51.
    Xiang L, Qiao Y, Nie D, An L, Wang Q, Shen D. Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing. 2017.
  52. 52.
    Jiao J, Ourselin S. Fast PET reconstruction using Multi-scale Fully Convolutional Neural Networks. arXiv:170407244. 2017.Google Scholar
  53. 53.
    Jin KH, McCann MT, Froustey E, Unser M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans Image Process. 2017;26:4509–22.CrossRefGoogle Scholar
  54. 54.
    Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. arXiv:161107004. 2016.Google Scholar
  55. 55.
    Nie D, Trullo R, Petitjean C, Ruan S, Shen D. Medical Image Synthesis with Context-Aware Generative Adversarial Networks. arXiv:161205362. 2016.Google Scholar
  56. 56.
    Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med Phys. 2017;44:1408–19.CrossRefPubMedGoogle Scholar
  57. 57.
    Bezrukov I, Mantlik F, Schmidt H, Schölkopf B, Pichler BJ, editors. MR-based PET attenuation correction for PET/MR imaging. Seminars in nuclear medicine. Amsterdam: Elsevier; 2013.Google Scholar
  58. 58.
    Suk H-I, Lee S-W, Shen D. Initiative AsDN. Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Struct Funct. 2015;220:841–59.CrossRefPubMedGoogle Scholar
  59. 59.
    Choi H, Jin KH. Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. arXiv:170406033. 2017.Google Scholar
  60. 60.
    Choi H, Ha S, Im HJ, Paek SH, Lee DS. Refining diagnosis of Parkinson's disease with deep learning-based interpretation of dopamine transporter imaging. Neuroimage Clin. 2017.
  61. 61.
    Oakden-Rayner L, Carneiro G, Bessen T, Nascimento JC, Bradley AP, Palmer LJ. Precision radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework. Sci Rep. 2017;7:1648.CrossRefPubMedPubMedCentralGoogle Scholar
  62. 62.
    Li Z, Wang Y, Yu J, Guo Y, Cao W. Deep learning based Radiomics (DLR) and its usage in noninvasive IDH1 prediction for low grade glioma. Sci Rep. 2017;7Google Scholar
  63. 63.
    Aerts HJ, Velazquez ER, Leijenaar RT, Parmar C, Grossmann P, Cavalho S, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014;5Google Scholar
  64. 64.
    Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, et al. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer. 2012;48:441–6.CrossRefPubMedPubMedCentralGoogle Scholar
  65. 65.
    Choi H. Functional connectivity patterns of autism spectrum disorder identified by deep feature learning. arXiv:170707932. 2017.Google Scholar
  66. 66.
    Collins FS, Varmus H. A new initiative on precision medicine. N Engl J Med. 2015;372:793–5.CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Chaudhary K, Poirion OB, Lu L, Garmire L. Deep Learning based multi-omics integration robustly predicts survival in liver cancer. Clin Cancer Res. 2017.
  68. 68.
    Choi H, Na KJ. A risk stratification model for lung cancer based on gene coexpression network. bioRxiv. 2017.
  69. 69.
    Gal Y, Ghahramani Z, editors. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. International Conference on Machine Learning; 2016.Google Scholar
  70. 70.
    Van De Vijver MJ, He YD, Van't Veer LJ, Dai H, Hart AA, Voskuil DW, et al. A gene-expression signature as a predictor of survival in breast cancer. N Engl J Med. 2002;347:1999–2009.CrossRefPubMedGoogle Scholar
  71. 71.
    Carneiro G, Nascimento J, Bradley AP, editors. Unregistered multiview mammogram analysis with pre-trained deep learning models. International Conference on Medical Image Computing and Computer-Assisted Intervention. Berllin: Springer; 2015.Google Scholar
  72. 72.
    Warren E. Strengthening research through data sharing. N Engl J Med. 2016;375:401–3.CrossRefPubMedGoogle Scholar
  73. 73.
    Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. arXiv:170502315. 2017.Google Scholar

Copyright information

© Korean Society of Nuclear Medicine 2017

Authors and Affiliations

  1. 1.Cheonan Public Health CenterCheonanRepublic of Korea

Personalised recommendations