Fully Convolutional Deep Residual Neural Networks for Brain Tumor Segmentation

  • Peter D. ChangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10154)


In this paper, a fully convolutional residual neural network (FCR-NN) based on linear identity mappings is implemented for medical image segmentation, employed here in the setting of brain tumors. Inspired by deep residual networks which won the ImageNet ILSVRC 2015 classification challenge, the FCR-NN combines optimization gains from residual identity mappings with a fully convolutional architecture for image segmentation that efficiently accounts for both low- and high-level image features. After training two separate networks, one for the task of whole tumor segmentation and a second for tissue sub-region segmentation, the serial FCR-NN architecture exceeds state-of-the art with complete tumor, core tumor and enhancing tumor validation Dice scores of 0.87, 0.81 and 0.72 respectively. Despite each FCR-NN comprising a complex 22 layer architecture, the fully convolutional design allows for complete segmentation of a tumor volume within 2 s.


Convolutional Neural Network Residual Function Tumor Segmentation Forward Pass Recursive Topology 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author of this paper gratefully acknowledges the support of NVIDIA Corporation with the donation of GeForce GTX Titan X (12 GB) GPU used for this research.


  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks (2012)Google Scholar
  2. 2.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Networks, deep inside convolutional: visualising image classification models and saliency maps. In: ICLR, p. 1 (2014)Google Scholar
  3. 3.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 07–12 June, 1–9 Sep 2015 (2015)Google Scholar
  4. 4.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. 7(3), 171–180 (2015). Arxiv.Org
  5. 5.
    Ciresan, D., Giusti, A.: Deep neural networks segment neuronal membranes in electron microscopy images. In: Advances in Neural Information Processing Systems, pp. 1–9 (2012)Google Scholar
  6. 6.
    Yang, S., Ramanan, D.: Multi-scale recognition with DAG-CNNs (2015)Google Scholar
  7. 7.
    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation (2014)Google Scholar
  8. 8.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440 (2014)Google Scholar
  9. 9.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). doi: 10.1007/978-3-319-24574-4_28 CrossRefGoogle Scholar
  10. 10.
    Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., Larochelle, H.: Brain tumor segmentation with deep neural networks. In: Proceedings of MICCAI-BRATS (Multimodal Brain Tumor Segmentation Challenge), pp. 29–33 (2015)Google Scholar
  11. 11.
    Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., Maier-hein, K., Bendszus, M., Biller, A.: NeuroImage deep MRI brain extraction: a 3D convolutional neural network for skull stripping. NeuroImage 129, 460–469 (2016)CrossRefGoogle Scholar
  12. 12.
    Chen, H., Dou, Q., Yu, L., Heng, P.-A.: VoxResNet: deep voxelwise residual networks for volumetric brain segmentation, pp. 1–9 (2016). arXiv:1608.05895v1
  13. 13.
    Sachdeva, J., Kumar, V., Gupta, I., Khandelwal, N., Ahuja, C.K.: A novel content-based active contour model for brain tumor segmentation. Magn. Reson. Imaging 30(5), 694–715 (2012)CrossRefGoogle Scholar
  14. 14.
    Harati, V., Khayati, R., Farzan, A.: Fully automated tumor segmentation based on improved fuzzy connectedness algorithm in brain MR images. Comput. Biol. Med. 41(7), 483–492 (2011)CrossRefGoogle Scholar
  15. 15.
    Prastawa, M., Bullitt, E., Gerig, G.: Simulation of brain tumors in MR images for evaluation of segmentation efficacy. Med. Image Anal. 13(2), 297–311 (2009)CrossRefGoogle Scholar
  16. 16.
    Menze, B.H., Leemput, K., Lashkari, D., Weber, M.-A., Ayache, N., Golland, P.: A generative model for brain tumor segmentation in multi-modal images. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6362, pp. 151–159. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15745-5_19 CrossRefGoogle Scholar
  17. 17.
    Hamamci, A., Kucuk, N., Karaman, K., Engin, K., Unal, G.: Tumor - cut: segmentation of brain tumors on contrast enhanced MR images for radiosurgery applications. IEEE Trans. Med. Imaging 31(3), 790–804 (2012)Google Scholar
  18. 18.
    Zhu, Y., Young, G.S., Xue, Z., Huang, R.Y., You, H., Setayesh, K., Hatabu, H., Cao, F., Wong, S.T.: Semi-automatic segmentation software for quantitative clinical brain glioblastoma evaluation. Acad. Radiol. 19(8), 977–85 (2012)Google Scholar
  19. 19.
    Menze, B.H., Geremia, E., Ayache, N., Szekely, G.: Segmenting glioma in multi-modal images using a generative-discriminative model for brain lesion segmentation. In: Proceedings of MICCAI-BRATS (Multimodal Brain Tumor Segmentation Challenge), p. 7 (2012)Google Scholar
  20. 20.
    Meier, R., Reyes, M., Bauer, S., Slotboom, J., Wiest, R.: A hybrid model for multimodal brain tumor segmentation. In: Proceedings of NCI-MICCAI BRATS (Multimodal Brain Tumor Segmentation Challenge), pp. 31–37 (2013)Google Scholar
  21. 21.
    Bakas, S., Zeng, K., Sotiras, A., Rathore, S., Akbari, H., Gaonkar, B., Rozycki, M., Pati, S., Davatzikos, C.: Segmentation of gliomas in multimodal magnetic resonance imaging volumes based on a hybrid generative - discriminative framework. In: Proceedings of MICCAI-BRATS (Multimodal Brain Tumor Segmentation Challenge), pp. 5–12 (2015)Google Scholar
  22. 22.
    Menze, B.H., Van Leemput, K., Lashkari, D., Riklin-Raviv, T., Geremia, E., Alberts, E., Gruber, P., Wegener, S., Weber, M.-A., Szekely, G., Ayache, N., Golland, P.: A generative probabilistic model and discriminative extensions for brain lesion segmentation with application to tumor and stroke. IEEE Trans. Med. Imaging 35(4), 933–946 (2016)CrossRefGoogle Scholar
  23. 23.
    Bakas, S., et al.: GLISTRboost: combining multimodal MRI segmentation, registration, and biophysical tumor growth modeling with gradient boosting machines for glioma segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 144–155. Springer, Cham (2016). doi: 10.1007/978-3-319-30858-6_13 CrossRefGoogle Scholar
  24. 24.
    Zeng, J., See, A.P., Phallen, J., Jackson, C.M., Belcaid, Z., Ruzevick, J., Durham, N., Meyer, C., Harris, T.J., Albesiano, E., Pradilla, G., Ford, E., Wong, J., Hammers, H.-J., Mathios, D., Tyler, B., Brem, H., Tran, P.T., Pardoll, D., Drake, C.G., Lim, M.: Anti-PD-1 blockade and stereotactic radiation produce long-term survival in mice with intracranial gliomas. Int. J. Rad. Oncol. Biol. Phys. 86(2), 343–349 (2013)Google Scholar
  25. 25.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  26. 26.
    Pereira, S., Pinto, A., Alves, V., Silva, C.A.: Deep convolutional neural networks for the segmentation of gliomas in multi-sequence MRI. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 131–143. Springer, Cham (2016). doi: 10.1007/978-3-319-30858-6_12 CrossRefGoogle Scholar
  27. 27.
    Lecun, Y., Bengio, Y.: Convolutional networks for images, speech, and time-series (1995)Google Scholar
  28. 28.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines (2010)Google Scholar
  29. 29.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, pp. 1–11 (2015). arXiv:1502.03167
  30. 30.
    Bengio, Y., Boulanger-Lewandowski, N., Pascanu, R.: Advances in optimizing recurrent networks. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, pp. 8624–8628 (2013)Google Scholar
  31. 31.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification (2015)Google Scholar
  32. 32.
    Mandic, D.P.: A generalized normalized gradient descent algorithm. IEEE Signal Process. Lett. 11(2), 115–118 (2004)CrossRefGoogle Scholar
  33. 33.
    Vedaldi, A., Lenc, K.: MatConvNet. In: Proceedings of the 23rd ACM International Conference on Multimedia - MM 2015, pp. 689–692 (2015)Google Scholar
  34. 34.
    Agn, M., Puonti, O., Rosenschöld, P.M., Law, I., Leemput, K.: Brain tumor segmentation using a generative model with an RBM prior on tumor shape. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Handels, H. (eds.) BrainLes 2015. LNCS, vol. 9556, pp. 168–180. Springer, Cham (2016). doi: 10.1007/978-3-319-30858-6_15 CrossRefGoogle Scholar
  35. 35.
    Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.A., Arbel, T., Avants, B.B., Ayache, N., Buendia, P., Collins, D.L., Cordier, N., Corso, J.J., Criminisi, A., Das, T., Delingette, H., Demiralp, Ç., Durst, C.R., Dojat, M., Doyle, S., Festa, J., Forbes, F., Geremia, E., Glocker, B., Golland, P., Guo, X., Hamamci, A., Iftekharuddin, K.M., Jena, R., John, N.M., Konukoglu, E., Lashkari, D., Mariz, J.A., Meier, R., Pereira, S., Precup, D., Price, S.J., Raviv, T.R., Reza, S.M.S., Ryan, M., Sarikaya, D., Schwartz, L., Shin, H.C., Shotton, J., Silva, C.A., Sousa, N., Subbanna, N.K., Szekely, G., Taylor, T.J., Thomas, O.M., Tustison, N.J., Unal, G., Vasseur, F., Wintermark, M., Ye, D.H., Zhao, L., Zhao, B., Zikic, D., Prastawa, M., Reyes, M., Leemput, K.V.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging 34(10), 1993–2024 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Department of RadiologyColumbia University Medical CenterNew York CityUSA

Personalised recommendations