Advertisement

Hierarchical Multimodal Fusion of Deep-Learned Lesion and Tissue Integrity Features in Brain MRIs for Distinguishing Neuromyelitis Optica from Multiple Sclerosis

  • Youngjin YooEmail author
  • Lisa Y. W. Tang
  • Su-Hyun Kim
  • Ho Jin Kim
  • Lisa Eunyoung Lee
  • David K. B. Li
  • Shannon Kolind
  • Anthony Traboulsee
  • Roger Tam
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10435)

Abstract

Neuromyelitis optica spectrum disorder (NMOSD) is a disease of the central nervous system that is often misdiagnosed as multiple sclerosis (MS) because they share similar clinical and radiological characteristics. Two key pathological signs of NMOSD and MS that are detectable on magnetic resonance imaging (MRI) are white matter lesions and alterations in tissue integrity as measured by fractional anisotropy (FA) values on diffusion tensor images (DTIs). This paper proposes a multimodal deep learning model that discovers latent features in brain lesion masks and DTIs for distinguishing NMOSD from MS. The main technical challenge is to optimally extract and integrate features from two very heterogeneous image types (lesion masks and FA maps). Our solution is to first build two modality-specific pathways, each designed to accommodate the expected feature density and scale, then integrate them into a hierarchical multimodal fusion (HMF) model. The HMF model contains two multimodal fusion layers operating at two different scales, which in turn are joined by a multi-scale fusion layer. We hypothesize that the HMF approach would allow the automatic extraction of joint-features of heterogeneous image types to be optimized with greater efficiency and accuracy than the traditional multimodal approach of combining only the top-layer modality-specific features with a single fusion layer. The proposed model gives an average diagnostic accuracy of 81.3% (85.3% sensitivity and 75.0% specificity) on 82 NMOSD patients and 52 MS patients in a seven-fold cross-validation, which significantly outperforms the user-defined MRI features previously used in clinical studies, as well as deep-learned features using the conventional fusion approach.

Notes

Acknowledgements

This work was supported by the Natural Sciences and Engineering Research Council of Canada, the MS Society of Canada, the Milan and Maureen Ilich Foundation, and the National Research Foundation of Korea.

References

  1. 1.
    Kim, H.J., Paul, F., Lana-Peixoto, M.A., et al.: MRI characteristics of neuromyelitis optica spectrum disorder: an international update. Neurology 84(11), 1165–1173 (2015)CrossRefGoogle Scholar
  2. 2.
    Eshaghi, A., Riyahi-Alam, S., Saeedi, R., et al.: Classification algorithms with multi-modal data fusion could accurately distinguish neuromyelitis optica from multiple sclerosis. NeuroImage Clin. 7, 306–314 (2015)CrossRefGoogle Scholar
  3. 3.
    Eshaghi, A., Wottschel, V., et al.: Gray matter MRI differentiates neuromyelitis optica from multiple sclerosis using random forest. Neurology 87(23), 2463–2470 (2016)CrossRefGoogle Scholar
  4. 4.
    Yoo, Y., Tang, L.W., Brosch, T., Li, D.K.B., Metz, L., Traboulsee, A., Tam, R.: Deep learning of brain lesion patterns for predicting future disease activity in patients with early symptoms of multiple sclerosis. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 86–94. Springer, Cham (2016). doi: 10.1007/978-3-319-46976-8_10 CrossRefGoogle Scholar
  5. 5.
    Karpathy, A., Toderici, G., Shetty, S., et al.: Large-scale video classification with convolutional neural networks. In: Proceeding of IEEE CVPR (2014)Google Scholar
  6. 6.
    Ngiam, J., Khosla, A., et al.: Multimodal deep learning. In: Proceeding of ICML (2011)Google Scholar
  7. 7.
    Jeon, S., Yoon, U., Park, J.S., et al.: Fully automated pipeline for quantification and localization of white matter hyperintensity in brain magnetic resonance image. International Journal of Imaging Systems and Technology 21(2), 193–200 (2011)CrossRefGoogle Scholar
  8. 8.
    Jenkinson, M., Beckmann, C.F., et al.: FSL. NeuroImage 62(2), 782–790 (2012)CrossRefGoogle Scholar
  9. 9.
    Lee, H., Grosse, R., Ranganath, R., et al.: Unsupervised learning of hierarchical representations with convolutional deep belief networks. Communications of the ACM 54(10), 95–103 (2011)CrossRefGoogle Scholar
  10. 10.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML (2013)Google Scholar
  11. 11.
    Zeiler, M.: ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012)
  12. 12.
    Srivastava, N., Hinton, G.E., Krizhevsky, A., et al.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Prechelt, L.: Early stopping - but when? In: Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 1524, pp. 55–69. Springer, Heidelberg (1998). doi: 10.1007/3-540-49430-8_3 CrossRefGoogle Scholar
  14. 14.
    Montavon, G., Orr, G.B., Müller, K.-R. (eds.): Neural Networks: Tricks of the Trade. LNCS, vol. 7700. Springer, Heidelberg (2012)Google Scholar
  15. 15.
    Suk, H.I., Lee, S.W., et al.: Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage 101, 569–582 (2014)CrossRefGoogle Scholar
  16. 16.
    Neelakantan, A., Vilnis, L., Le, Q.V., et al.: Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807 (2015)
  17. 17.
    Xu, T., Zhang, H., Huang, X., Zhang, S., Metaxas, D.N.: Multimodal deep learning for cervical dysplasia diagnosis. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 115–123. Springer, Cham (2016). doi: 10.1007/978-3-319-46723-8_14 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Youngjin Yoo
    • 1
    • 2
    • 5
    Email author
  • Lisa Y. W. Tang
    • 3
    • 5
  • Su-Hyun Kim
    • 7
  • Ho Jin Kim
    • 7
  • Lisa Eunyoung Lee
    • 5
  • David K. B. Li
    • 3
    • 5
  • Shannon Kolind
    • 3
    • 4
    • 5
    • 6
  • Anthony Traboulsee
    • 4
    • 5
  • Roger Tam
    • 2
    • 3
    • 5
  1. 1.Department of Electrical and Computer EngineeringUniversity of British ColumbiaVancouverCanada
  2. 2.Biomedical Engineering ProgramUniversity of British ColumbiaVancouverCanada
  3. 3.Department of RadiologyUniversity of British ColumbiaVancouverCanada
  4. 4.Department of Medicine (Division of Neurology)University of British ColumbiaVancouverCanada
  5. 5.MS/MRI Research GroupUniversity of British ColumbiaVancouverCanada
  6. 6.Department of Physics and AstronomyUniversity of British ColumbiaVancouverCanada
  7. 7.Department of NeurologyResearch Institute and Hospital of National Cancer CenterGoyang-siRepublic of Korea

Personalised recommendations