Multi-task deep learning for fine-grained classification and grading in breast cancer histopathological images

  • Lingqiao Li
  • Xipeng Pan
  • Huihua YangEmail author
  • Zhenbing LiuEmail author
  • Yubei He
  • Zhongming Li
  • Yongxian Fan
  • Zhiwei Cao
  • Longhao Zhang


Fine-grained classification and grading of breast cancer (BC) histopathological images are of great value in clinical application. However, automatic classification and grading of BC histopathological images are complicated by (1) small inter-class variance and large intra-class variance exist in BC histopathological images, and (2) features extracted from similar histopathological images with different magnification are quite different. To address these issues, an improved deep convolution neural network model is proposed and the procedure can be divided into three main stages. Firstly, in the representation learning process, multi-class recognition task and verification task of image pair are combined. Secondly, in the feature extraction process, a prior knowledge is built, which is “the variances in feature outputs between different subclasses is relatively large while the variance between the same subclass is small.” Additionally, the prior information that histopathological images with different magnification belong to the same subclass are embedded in the feature extraction process, which contributes to less sensitive with image magnification. The experimental results based on three different histopathological image datasets show that the performance of the proposed method is better than state of the art, with better robustness and generalization ability.


Multi-task deep learning Histopathological image classification Fine-grained Convolutional neural network Breast cancer 



The authors would like to thank Spanhol et al. [33], Dimitropoulos et al. [8], and Dr. Andrew Janowczyk et al. [16] for publishing the datasets. We would like to express our gratitude to anonymous reviewers and editor for their helpful comments. This research was supported in part by the National Natural Science Foundation of China (Grant Nos. 21365008, 61462018, 61762026 and 61562013), and Natural Science Foundation of Guangxi Province (No. 2017GXNSFDA198025 and 2017GXNSFAA198278). The authors declare that they have no conflict of interest.


  1. 1.
    Akbar S, Peikari M, Salama S, Nofech-Mozes S, Martel A (2017) Transitioning between convolutional and fully connected layers in neural networks. In: Deep learning in medical image analysis and multimodal learning for clinical decision support (DLMIA)Google Scholar
  2. 2.
    Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: International conference on pattern recognition (ICPR), pp 2441–2446Google Scholar
  3. 3.
    Bell S, Bala K (2015) Learning visual similarity for product design with convolutional neural networks. ACM Trans Graph 34(4):98:1–98:10CrossRefGoogle Scholar
  4. 4.
    Chen H, Dou Q, Wang X et al (2016) Mitosis detection in breast cancer histology images via deep cascaded networks. In: Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI), pp 1160–1166Google Scholar
  5. 5.
    Chollet F (2016) Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357
  6. 6.
    Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 539–546Google Scholar
  7. 7.
    Codella N, Moradi M, Matasar M, Sveda-Mahmood T, Smith JR (2016) Lymphoma diagnosis in histopathology using a multi-stage visual learning approach. In: Proceedings of the SPIE, medical imaging 2016: digital pathologyGoogle Scholar
  8. 8.
    Dimitropoulos K, Barmpoutis P, Zioga C, Kamas A, Patsiaoura K, Grammalidis N (2017) Grading of invasive breast carcinoma through grassmannian VLAD encoding. Plos One 12(9):e0185110CrossRefGoogle Scholar
  9. 9.
    Dozat T (2015) Incorporating nesterov momentum into adam. Technical report,
  10. 10.
    Garud H, Karri SPK, Sheet D et al (2017) High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 828–833Google Scholar
  11. 11.
    Gupta V, Bhavsar A (2017) An integrated multi-scale model for breast cancer histopathological image classification with joint colour-texture features. In: 17th international conference on computer analysis of images and patterns (CAIP), pp 354–366Google Scholar
  12. 12.
    Gupta V, Bhavsar A (2017) Breast cancer histopathological image classification: is magnification important?. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 769–776Google Scholar
  13. 13.
    Gupta V, Singh A, Sharma K et al (2017) Automated classification for breast cancer histopathology images: is stain normalization important?. In: Computer assisted and robotic endoscopy and clinical image-based procedures (CLIP) 10550, pp 160–169Google Scholar
  14. 14.
    Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction by learning an invariant mapping. In: Computer vision and pattern recognition (CVPR), pp 1735–1742Google Scholar
  15. 15.
    Han Z, Wei B, Zheng Y, Yin Y, Li K, Li S (2017) Breast cancer multi-classification from histopathological images with structured deep learning model. Sci Rep 7(1):4172CrossRefGoogle Scholar
  16. 16.
    Janowczyk A, Madabhushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7(1):29CrossRefGoogle Scholar
  17. 17.
    Kowal M, Filipczuk P, Obuchowicz A et al (2013) Computer-aided diagnosis of breast cancer based on fine needle biopsy microscopic images. Comput Biol Med 43 (10):1563–1572CrossRefGoogle Scholar
  18. 18.
    Lan R, Zhou Y (2016) Quaternion-michelson descriptor for color image classification. IEEE Trans Image Process 25(11):5281–5292MathSciNetCrossRefGoogle Scholar
  19. 19.
    Lan R, Zhou Y, Tang Y (2016) Quaternionic local ranking binary pattern: a local descriptor of color images. IEEE Trans Image Process 25(2):566–579MathSciNetCrossRefGoogle Scholar
  20. 20.
    Lan R, Zhou Y, Tang Y (2017) Quaternionic weber local descriptor of color images. IEEE Trans Circuits Syst Video Techn 27(2):261–274CrossRefGoogle Scholar
  21. 21.
    Lan R, Zhong S, Liu Z et al (2018) A simple texture feature for retrieval of medical images. Multimed Tools Appl 77(9):10853–10866CrossRefGoogle Scholar
  22. 22.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444CrossRefGoogle Scholar
  23. 23.
    Li P, Wang D, Wang L, Lu H (2018) Deep visual tracking: review and experimental comparison. Pattern Recogn 76:323–338CrossRefGoogle Scholar
  24. 24.
    Li Y, Lu H, Li KC et al (2018) Non-uniform de-scattering and de-blurring of underwater images. Mob Netw Appl 23(4):352–362CrossRefGoogle Scholar
  25. 25.
    Lu H, Zhang L, Serikawa S (2012) Maximum local energy: an effective approach for multisensor image fusion in beyond wavelet transform domain. Comput Math Appl 64(5):996–1003CrossRefGoogle Scholar
  26. 26.
    Lu H, Li B, Zhu J et al (2017) Wound intensity correction and segmentation with convolutional neural networks. Concurr Comput Pract Experience 29(6):3927CrossRefGoogle Scholar
  27. 27.
    Lu H, Li Y, Chen M et al (2018) Brain intelligence: go beyond artificial intelligence. Mob Netw Appl 23(2):1–8Google Scholar
  28. 28.
    Pan X, Li L, Yang H et al (2017) Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing 229:88–99CrossRefGoogle Scholar
  29. 29.
    Pan X, Yang D, Li L et al (2018) Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks. World Wide Web-internet & Web Information Systems 21(6):1721–1743Google Scholar
  30. 30.
    Song Y, Zou J, Chang H et al (2017) Adapting fisher vectors for histopathology image classification. In: International symposium on biomedical imaging (ISBI), pp 600–603Google Scholar
  31. 31.
    Song Y, Chang H, Huang H, Cai W (2017) Supervised intra-embedding of fisher vectors for histopathology image classification. In: Medical image computing and computer-assisted intervention (MICCAI)Google Scholar
  32. 32.
    Spanhol FA, Cavalin P, Oliveira LS, Petitjean C, Heutte L (2017) Deep features for breast cancer histopathological image classification. In: IEEE international conference on systems, man, and cybernetics (IEEE SMC), pp 1868–1873Google Scholar
  33. 33.
    Spanhol FA, Oliveira LS, Petitjean C et al (2016) A dataset for breast cancer histopathological image classification. IEEE Trans Biomed Eng 63(7):1455–1462CrossRefGoogle Scholar
  34. 34.
    Spanhol FA, Oliveira LS, Petitjean C, Heutte L (2016) Breast cancer histopathological image classification using convolutional neural networks. In: 2016 international joint conference on neural networks, pp 2560–2567Google Scholar
  35. 35.
    Stewart B, Wild CP (2014) World cancer report 2014. International Agency for Research on Cancer, World Health OrganizationGoogle Scholar
  36. 36.
    Sun Y, Chen Y, Wang X et al (2014) Deep learning face representation by joint identification-verification. In: International conference on neural information processing systems (ICNIPS), pp 1988–1996Google Scholar
  37. 37.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S et al (2015) Going deeper with convolutions. In: Computer vision and pattern recognition (CVPR)Google Scholar
  38. 38.
    Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1701–1708Google Scholar
  39. 39.
    Wang D, Lu H, Li X (2011) Two dimensional principal components of natural images and its application. Neurocomputing 74(17):2745–2753CrossRefGoogle Scholar
  40. 40.
    Wang P, Hu X, Li Y, Liu Q, Zhu X (2016) Automatic cell nuclei segmentation and classification of breast cancer histopathology images. Signal Process 122:1–13CrossRefGoogle Scholar
  41. 41.
    Wang D, Khosla A, Gargeya R et al (2016) Deep learning for identifying metastatic breast cancer. arXiv:1606.05718
  42. 42.
    Wei B, Han Z, He X, Yin Y (2017) Deep learning model based breast cancer bistro pathological image classification. In: 2017 IEEE 2nd international conference on cloud computing and big data analysis (ICCCBDA)Google Scholar
  43. 43.
    Xiao T, Xu Y, Yang K et al (2015) The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 842–850Google Scholar
  44. 44.
    Zhang S, Metaxas D (2016) Large-scale medical image analytics: recent methodologies, applications and future directions. Med Image Anal 33:98–101CrossRefGoogle Scholar
  45. 45.
    Zhang Y, Zhang B, Coenen F et al (2014) One-class kernel subspace ensemble for medical image classification. EURASIP J Adv Signal Process 2014:17. CrossRefGoogle Scholar
  46. 46.
    Zhang X, Su H, Yang L, Zhang S (2015) Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  47. 47.
    Zhang X, Zhou F, Lin Y, Zhang S (2016) Embedding label structures for fine-grained feature representation. In: 29th IEEE conference on computer vision and pattern recognition (CVPR), pp 1114–1123Google Scholar
  48. 48.
    Zhi W, Yueng HWF, Chen Z et al (2017) Using transfer learning with convolutional neural networks to diagnose breast cancer from histopathological images. In: 24th international conference on neural information processing (ICNIP)Google Scholar
  49. 49.
    Zhu X, Li X, Zhang S (2016) Block-row sparse multiview multilabel learning for image classification. IEEE Trans Cybern 46(2):450–461CrossRefGoogle Scholar
  50. 50.
    Zhu X, Suk HI, Lee SW et al (2016) Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans Bio-Med Eng 63(3):607–618CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of AutomationBeijing University of Posts and TelecommunicationsBeijingChina
  2. 2.School of Computer Science and Information SecurityGuilin University of Electronic TechnologyGuilinChina
  3. 3.School of Computing and Information SystemsUniversity of MelbourneMelbourneAustralia

Personalised recommendations