Multi-planar Spatial-ConvNet for Segmentation and Survival Prediction in Brain Cancer

  • Subhashis BanerjeeEmail author
  • Sushmita Mitra
  • B. Uma Shankar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11384)


A new deep learning method is introduced for the automatic delineation/segmentation of brain tumors from multi-sequence MR images. A Radiomic model for predicting the Overall Survival (OS) is designed, based on the features extracted from the segmented Volume of Interest (VOI). An encoder-decoder type ConvNet model is designed for pixel-wise segmentation of the tumor along three anatomical planes (axial, sagittal and coronal) at the slice level. These are then combined, using a consensus fusion strategy, to produce the final volumetric segmentation of the tumor and its sub-regions. Novel concepts such as spatial-pooling and unpooling are introduced to preserve the spatial locations of the edge pixels for reducing segmentation error around the boundaries. We also incorporate shortcut connections to copy and concatenate the receptive fields from the encoder to the decoder part, for helping the decoder network localize and recover the object details more effectively. These connections allow the network to simultaneously incorporate high-level features along with pixel-level details. A new aggregated loss function helps in effectively handling data imbalance. The integrated segmentation and OS prediction system is trained and validated on the BraTS 2018 dataset.


Deep learning Convolutional neural network Spatial-pooling Brain tumor segmentation Survival prediction Radiomics Class imbalance handling 



We gratefully acknowledge the support of Intel Corporation for providing access to the Intel AI DevCloud platform used in this work.

S. Banerjee acknowledges the support provided to him by the Intel Corporation, through the Intel AI Student Ambassador Program.

This publication is an outcome of the R&D work undertaken project under the Visvesvaraya PhD Scheme of Ministry of Electronics & Information Technology, Government of India, being implemented by Digital India Corporation.

S. Mitra acknowledges the support provided to her by the Indian National Academy of Engineering, through the INAE Chair Professorship.


  1. 1.
    Bakas, S., et al.: Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci. Data 4, 170117 (2017)CrossRefGoogle Scholar
  2. 2.
    Bakas, S., et al.: Segmentation labels and radiomic features for the pre-operative scans of the TCGA-GBM collection. The Cancer Imaging Archive (2017).
  3. 3.
    Bakas, S., et al.: Segmentation labels and radiomic features for the pre-operative scans of the TCGA-LGG collection. The Cancer Imaging Archive (2017).
  4. 4.
    Bakas, S., Reyes, M., et al.: Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 (2018)
  5. 5.
    Banerjee, S., Mitra, S., Shankar, B.U.: Synergetic neuro-fuzzy feature selection and classification of brain tumors. In: Proceedings of IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–6 (2017)Google Scholar
  6. 6.
    Banerjee, S., Mitra, S., Uma Shankar, B.: Single seed delineation of brain tumor using multi-thresholding. Inf. Sci. 330, 88–103 (2016)CrossRefGoogle Scholar
  7. 7.
    Banerjee, S., Mitra, S., Uma Shankar, B.: Automated 3D segmentation of brain tumor using visual saliency. Inf. Sci. 424, 337–353 (2018)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Banerjee, S., Mitra, S., Uma Shankar, B., Hayashi, Y.: A novel GBM saliency detection model using multi-channel MRI. PLOS ONE 11(1), e0146388 (2016)CrossRefGoogle Scholar
  9. 9.
    Banerjee, S., Mitra, S., Sharma, A., Shankar, B.U.: A CADe system for gliomas in brain MRI using convolutional neural networks. arXiv preprint arXiv:1806.07589 (2018)
  10. 10.
    Coroller, T., et al.: Early grade classification in meningioma patients combining radiomics and semantics data. Med. Phys. 43, 3348–3349 (2016)CrossRefGoogle Scholar
  11. 11.
    Gillies, R.J., Kinahan, P.E., Hricak, H.: Radiomics: images are more than pictures, they are data. Radiology 278, 563–577 (2015)CrossRefGoogle Scholar
  12. 12.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  13. 13.
    Menze, B.H., et al.: The multimodal Brain Tumor image Segmentation benchmark (BraTS). IEEE Trans. Med. Imaging 34, 1993–2024 (2015)CrossRefGoogle Scholar
  14. 14.
    Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. IEEE (2016)Google Scholar
  15. 15.
    Mitra, S., Banerjee, S., Hayashi, Y.: Volumetric brain tumour detection from MRI using visual saliency. PLOS ONE 12, 1–14 (2017)Google Scholar
  16. 16.
    Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  17. 17.
    Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 240–248. Springer, Cham (2017). Scholar
  18. 18.
    Zhou, M., et al.: Radiomics in brain tumor: image assessment, quantitative feature descriptors, and machine-learning approaches. Am. J. Neuroradiol. 39, 208–216 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Subhashis Banerjee
    • 1
    • 2
    Email author
  • Sushmita Mitra
    • 1
  • B. Uma Shankar
    • 1
  1. 1.Machine Intelligence UnitIndian Statistical InstituteKolkataIndia
  2. 2.Department of CSEUniversity of CalcuttaKolkataIndia

Personalised recommendations