Saliency Based Deep Neural Network for Automatic Detection of Gadolinium-Enhancing Multiple Sclerosis Lesions in Brain MRI

  • Joshua Durso-FinleyEmail author
  • Douglas L. Arnold
  • Tal Arbel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11992)


The appearance of contrast-enhanced pathologies (e.g. lesion, cancer) is an important marker of disease activity, stage and treatment efficacy in clinical trials. The automatic detection and segmentation of these enhanced pathologies remains a difficult challenge, as they can be very small and visibly similar to other non-pathological enhancements (e.g. blood vessels). In this paper, we propose a deep neural network classifier for the detection and segmentation of Gadolinium enhancing lesions in brain MRI of patients with Multiple Sclerosis (MS). To avoid false positive and false negative assertions, the proposed end-to-end network uses an enhancement-based attention mechanism which assigns saliency based on the differences between the T1-weighted images before and after injection of Gadolinium, and works to first identify candidate lesions and then to remove the false positives. The effect of the saliency map is evaluated on 2293 patient multi-channel MRI scans acquired during two proprietary, multi-center clinical trials for MS treatments. Inclusion of the attention mechanism results in a decrease in false positive lesion voxels over a basic U-Net [2] and DeepMedic [6]. In terms of lesion-level detection, the framework achieves a sensitivity of 82% at a false discovery rate of 0.2, significantly outperforming the other two methods when detecting small lesions. Experiments aimed at predicting the presence of Gad lesion activity in patient scans (i.e. the presence of more than 1 lesion) result in high accuracy showing: (a) significantly improved accuracy over DeepMedic, and (b) a reduction in the errors in predicting the degree of lesion activity (in terms of per scan lesion counts) over a standard U-Net and DeepMedic.


Segmentation Gadolinium lesions Multiple Sclerosis Attention Deep learning 



This work was supported by an award from the International Progressive MS Alliance (PA-1603-08175).


  1. 1.
    Brosch, T., Tang, L.Y.W., Yoo, Y., Li, D.K.B., Traboulsee, A., Tam, R.: Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans. Med. Imaging 35(5), 1229–1239 (2016). Scholar
  2. 2.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. CoRR abs/1606.06650 (2016).
  3. 3.
    Datta, S., Sajja, B.R., He, R., Gupta, R.K., Wolinsky, J.S., Narayana, P.A.: Segmentation of gadolinium-enhanced lesions on MRI in multiple sclerosis. J. Magn. Reson. Imaging Off. J. Int. Soc. Magn. Reson. Med. 25(5), 932–937 (2007)Google Scholar
  4. 4.
    Fleishman, G.M., et al.: Joint intensity fusion image synthesis applied to multiple sclerosis lesion segmentation. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 43–54. Springer, Cham (2018). Scholar
  5. 5.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015).
  6. 6.
    Kamnitsas, K., et al.: DeepMedic for brain tumor segmentation. In: Crimi, A., Menze, B., Maier, O., Reyes, M., Winzeck, S., Handels, H. (eds.) BrainLes 2016. LNCS, vol. 10154, pp. 138–149. Springer, Cham (2016). Scholar
  7. 7.
    Kappos, L., et al.: Ocrelizumab in relapsing-remitting multiple sclerosis: a phase 2, randomised, placebo-controlled, multicentre trial. Lancet 378(9805), 1779–1787 (2011)CrossRefGoogle Scholar
  8. 8.
    Karimaghaloo, Z., Rivaz, H., Arnold, D.L., Collins, D.L., Arbel, T.: Temporal hierarchical adaptive texture crf for automatic detection of gadolinium-enhancing multiple sclerosis lesions in brain mri. IEEE Trans. Med. Imaging 34(6), 1227–1241 (2015). Scholar
  9. 9.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). arXiv:1412.6980. Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego (2015)
  10. 10.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015). Scholar
  11. 11.
    Linguraru, M.G., Pura, J.A., Chowdhury, A.S., Summers, R.M.: Multi-organ segmentation from multi-phase abdominal CT via 4D graphs using enhancement, shape and location optimization. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6363, pp. 89–96. Springer, Heidelberg (2010). Scholar
  12. 12.
    Miller, D., Barkhof, F., Nauta, J.: Gadolinium enhancement increases the sensitivity of MRI in detecting disease activity in multiple sclerosis. Brain 116(5), 1077–1094 (1993)CrossRefGoogle Scholar
  13. 13.
    Nair, T., Precup, D., Arnold, D.L., Arbel, T.: Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 655–663. Springer, Cham (2018). Scholar
  14. 14.
    Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 2010, pp. 807–814. Omnipress, USA (2010).
  15. 15.
    Nyul, L.G., Udupa, J.K., Zhang, X.: New variants of a method of MRI scale standardization. IEEE Trans. Med. Imaging 19(2), 143–150 (2000). Scholar
  16. 16.
    Sled, J.G., Zijdenbos, A.P., Evans, A.C.: A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Trans. Med. Imaging 17(1), 87–97 (1998). Scholar
  17. 17.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014). Scholar
  18. 18.
    Valverde, S., et al.: Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach. CoRR abs/1702.04869 (2017).
  19. 19.
    Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017).
  20. 20.
    Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. CoRR abs/1505.00853 (2015).
  21. 21.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. CoRR abs/1502.03044 (2015).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Joshua Durso-Finley
    • 1
    Email author
  • Douglas L. Arnold
    • 2
    • 3
  • Tal Arbel
    • 1
  1. 1.Centre for Intelligent MachinesMcGill UniversityMontrealCanada
  2. 2.Montreal Neurological InstituteMcGill UniversityMontrealCanada
  3. 3.NeuroRx ResearchMontrealCanada

Personalised recommendations