Medical Image Classification Using Deep Learning

  • Weibin Wang
  • Dong Liang
  • Qingqing Chen
  • Yutaro Iwamoto
  • Xian-Hua Han
  • Qiaowei Zhang
  • Hongjie Hu
  • Lanfen Lin
  • Yen-Wei ChenEmail author
Part of the Intelligent Systems Reference Library book series (ISRL, volume 171)


Image classification is to assign one or more labels to an image, which is one of the most fundamental tasks in computer vision and pattern recognition. In traditional image classification, low-level or mid-level features are extracted to represent the image and a trainable classifier is then used for label assignments. In recent years, the high-level feature representation of deep convolutional neural networks has proven to be superior to hand-crafted low-level and mid-level features. In the deep convolutional neural network, both feature extraction and classification networks are combined together and trained end-to-end. Deep learning techniques have also been applied to medical image classification and computer-aided diagnosis. In this chapter, we first introduce fundamentals of deep convolutional neural networks for image classification and then introduce an application of deep learning to classification of focal liver lesions on multi-phase CT images. The main challenge in deep-learning-based medical image classification is the lack of annotated training samples. We demonstrate that fine-tuning can significantly improve the accuracy of liver lesion classification, especially for small training samples.



We would like to thank Sir Run Run Shaw Hospital for providing medical data and helpful advice on this research. This work is supported in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant Nos. 18H03267, 18K18078, in part by Zhejiang Lab Program under the Grant No. 2018DG0ZX01, in part by the Key Science and Technology Innovation Support Program of Hangzhou under the Grant No. 20172011A038.


  1. 1.
    Huang, Y., et al.: Feature coding in image classification: a comprehensive study. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 493–506 (2014)CrossRefGoogle Scholar
  2. 2.
    Vailaya, A., et al.: Image classification for content-based indexing. IEEE Trans. Image Process. 10(1), 117–130 (2001)CrossRefGoogle Scholar
  3. 3.
    Collins, T.R., et al.: A system for video surveillance and monitoring. VSAM final report, pp. 1–68 (2000)Google Scholar
  4. 4.
    Kosala, R., Hendrik, B.: Web mining research: a survey. ACM SIGKDD Explor. Newsl. 2(1), 1–15 (2000)CrossRefGoogle Scholar
  5. 5.
    Pavlovic, I.V., Rajeev, S., et al.: Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Trans. Pattern Anal. Mach. Intell. 7, 677–695 (1997)CrossRefGoogle Scholar
  6. 6.
    Jain, A.K., Arun, R., Salil, P.: An introduction to biometric recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 4–20 (2004)CrossRefGoogle Scholar
  7. 7.
    Cheng, G., Guo, L., Zhao, T., et al.: Automatic landslide detection from remote-sensing im-agery using a scene classification method based on BoVW and pLSA. Int. J. Remote Sens. 34(1), 45–59 (2013)CrossRefGoogle Scholar
  8. 8.
    Csurka, G., et al.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, ECCV, vol. 1. no. 1–22 (2004)Google Scholar
  9. 9.
    Chang, C., Lin, C.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 27 (2011)Google Scholar
  10. 10.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  11. 11.
    Litjens, G., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)CrossRefGoogle Scholar
  12. 12.
    Deng, J., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009, CVPR 2009. IEEE (2009)Google Scholar
  13. 13.
    Alex, K., Sutskever, I., Hinton, E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  14. 14.
    Perronnin, F., Jorge, S., Thomas, M.: Improving the fisher kernel for large-scale image classification. In: European Conference on Computer Vision. Springer, Berlin, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Zeiler, D.M., Rob, F.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision. Springer, Cham (2014)CrossRefGoogle Scholar
  16. 16.
    Sermanet, P., et al.: Overfeat: integrated recognition, localization and detection using convolutional networks (2013). arXiv:1312.6229
  17. 17.
    Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  18. 18.
    He, K., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  19. 19.
    Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115 (2017)CrossRefGoogle Scholar
  20. 20.
    Simonyan, K., Andrew, Z.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556
  21. 21.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010)Google Scholar
  22. 22.
    Bi, L., Kim, J., Kumar, A., et al.: Automatic Liver Lesion Detection using Cascaded Deep Residual Networks (2017). arXiv:1704.02703
  23. 23.
    Liang, D., et al.: Combining convolutional and recurrent neural networks for classification of focal liver lesions in multi-phase CT images. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI2018) (2018)CrossRefGoogle Scholar
  24. 24.
    Liang, D., et al.: Residual convolutional neural networks with global and local path-ways for classification of focal liver lesions. In: Pacific Rim International Conference on Artificial Intelligence. Springer, Cham (2018)Google Scholar
  25. 25.
    Peng, L., et al.: Classification and quantification of emphysema using a multi-scale residual network. IEEE J. Biomed. Health Inform. (2019) (in press)Google Scholar
  26. 26.
    Tajbakhsh, N., et al.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35(5), 1299–1312 (2016)CrossRefGoogle Scholar
  27. 27.
    Wang, G., Li, W., Zuluaga, M.A., et al.: Interactive medical image segmentation using deep learning with image-specific fine-tuning. IEEE Trans. Med. Imaging (2018)Google Scholar
  28. 28.
    Xu, Y., et al.: Texture-specific bag of visual words model and spatial cone matching based method for the retrieval of focal liver lesions using multiphase contrast-enhanced CT images. Int. J. Comput. Assis. Radiol. Surg. 13, 151–164 (2018)CrossRefGoogle Scholar
  29. 29.
    Wang, J., et al.: Tensor-based sparse representations of multi-phase medical images for classification of focal liver lesions. Pattern Recognit. Lett. (2018)Google Scholar
  30. 30.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  31. 31.
    Wang, W., et al.: Classification of focal liver lesions using deep learning with fine-tuning. In: Proceedings of Digital Medicine and Image Processing (DMIP2018), pp. 56–60 (2018)Google Scholar
  32. 32.
    Frid-Adar, M., et al.: Modeling the intra-class variability for liver lesion detection using a multi-class patch-based CNN. In: International Workshop on Patch-Based Techniques in Medical Imaging, Springer, Cham (2017)CrossRefGoogle Scholar
  33. 33.
    Yasaka, K., et al.: Deep learning with convolutional neural network for differentiation of liver masses at dynamic contrast-enhanced CT: a preliminary study. Radiology 286(3), 170706 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Weibin Wang
    • 1
  • Dong Liang
    • 2
  • Qingqing Chen
    • 3
  • Yutaro Iwamoto
    • 1
  • Xian-Hua Han
    • 4
  • Qiaowei Zhang
    • 3
  • Hongjie Hu
    • 3
  • Lanfen Lin
    • 2
  • Yen-Wei Chen
    • 1
    • 5
    Email author
  1. 1.The Graduate School of Information Science and EngineeringRitsumeikan UniversityKusatsuJapan
  2. 2.College of Computer Science and Technology, Zhejiang UniversityHangzhouChina
  3. 3.Department of RadiologySir Run Run Shaw Hospital, Zhejiang UniversityHangzhouChina
  4. 4.Yamaguchi UniversityYamaguchiJapan
  5. 5.Zhejiang LabHangzhouChina

Personalised recommendations