, Volume 16, Issue 3–4, pp 295–308 | Cite as

Multi-Modality Cascaded Convolutional Neural Networks for Alzheimer’s Disease Diagnosis

  • Manhua Liu
  • Danni Cheng
  • Kundong Wang
  • Yaping Wang
  • the Alzheimer’s Disease Neuroimaging Initiative
Original Article


Accurate and early diagnosis of Alzheimer’s disease (AD) plays important role for patient care and development of future treatment. Structural and functional neuroimages, such as magnetic resonance images (MRI) and positron emission tomography (PET), are providing powerful imaging modalities to help understand the anatomical and functional neural changes related to AD. In recent years, machine learning methods have been widely studied on analysis of multi-modality neuroimages for quantitative evaluation and computer-aided-diagnosis (CAD) of AD. Most existing methods extract the hand-craft imaging features after image preprocessing such as registration and segmentation, and then train a classifier to distinguish AD subjects from other groups. This paper proposes to construct cascaded convolutional neural networks (CNNs) to learn the multi-level and multimodal features of MRI and PET brain images for AD classification. First, multiple deep 3D-CNNs are constructed on different local image patches to transform the local brain image into more compact high-level features. Then, an upper high-level 2D-CNN followed by softmax layer is cascaded to ensemble the high-level features learned from the multi-modality and generate the latent multimodal correlation features of the corresponding image patches for classification task. Finally, these learned features are combined by a fully connected layer followed by softmax layer for AD classification. The proposed method can automatically learn the generic multi-level and multimodal features from multiple imaging modalities for classification, which are robust to the scale and rotation variations to some extent. No image segmentation and rigid registration are required in pre-processing the brain images. Our method is evaluated on the baseline MRI and PET images of 397 subjects including 93 AD patients, 204 mild cognitive impairment (MCI, 76 pMCI +128 sMCI) and 100 normal controls (NC) from Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. Experimental results show that the proposed method achieves an accuracy of 93.26% for classification of AD vs. NC and 82.95% for classification pMCI vs. NC, demonstrating the promising classification performance.


Alzheimer’s disease diagnosis Multi-modality brain images Convolutional neural networks (CNNs) Cascaded CNNs Image classification 



This work was supported in part by National Natural Science Foundation of China (NSFC) under grants (No. 61375112, 61773263, U1504606), The National Key Research and Development Program of China (No.2016YFC0100903) and SMC Excellent Young Faculty program of SJTU. Data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (NIH Grant U01 AG024904). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: Abbott, AstraZeneca AB, Bayer Schering Pharma AG, Bristol-Myers Squibb, Eisai Global Clinical Development, Elan Corporation, Genentech, GE Healthcare, GlaxoSmithKline, Innogenetics, Johnson and Johnson, Eli Lilly and Co.,Medpace, Inc.,Merck and Co., Inc., Novartis AG, Pfizer Inc., F. Hoffman-La Roche, Schering-Plough, Synarc, Inc., as well as non-profit partners the Alzheimer’s Association and Alzheimer’s Drug Discovery Foundation, with participation from the U.S. Food and Drug Administration. Private sector contributions to ADNI are facilitated by the Foundation for the National Institutes of Health ( The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Disease Cooperative Study at the University of California, San Diego. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of California, Los Angeles.


  1. Adrien, P.A.G.M. (2015). Predicting Alzheimer’s disease: a neuroimaging study with 3D convolutional neural networks. arXiv:1502.02506 [cs.CV].Google Scholar
  2. Alberdi, A., Aztiria, A., & Basarab, A. (2016). On the early diagnosis of Alzheimer's disease from multimodal signals: A survey. Artificial Intelligence in Medicine, 71, 1–29.CrossRefPubMedGoogle Scholar
  3. Cabral, C., Silveira, M., Neuroimaging, A.S.D. (2013). Classification of Alzheimer’s disease from FDG-PET images using favourite class ensembles. 2013 35th Annual International Conference of the Ieee Engineering in Medicine and Biology Society (Embc), pp. 2477–2480.Google Scholar
  4. Cheng, B., Liu, M., Suk, H. I., Shen, D., & Zhang, D. (2015). Multimodal manifold-regularized transfer learning for MCI conversion prediction. Brain Imaging and Behavior, 9, 913–926.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Gerardin, E., Chetelat, G., Chupin, M., Cuingnet, R., Desgranges, B., Kim, H. S., Niethammer, M., Dubois, B., Lehericy, S., Garnero, L., Eustache, F., Colliot, O., & Initi, A.s. D. N. (2009). Multidimensional classification of hippocampal shape features discriminates Alzheimer’s disease and mild cognitive impairment from normal aging. NeuroImage, 47, 1476–1486.Google Scholar
  6. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. pp. 770–778.Google Scholar
  7. Hinrichs, C., Singh, V., Mukherjee, L., Xu, G., Chung, M. K., & Johnson, S. C. (2009). Spatially augmented LPboosting for AD classification with evaluations on the ADNI dataset. NeuroImage, 48, 138–149.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Hosseini-Asl, E., Keynton, R., & El-Baz, A. (2016). Alzheimer’s disease diagnostics by adaptation of 3D convolutional network. 2016 I.E. International Conference on Image Processing (ICIP), pp 126–130.Google Scholar
  9. Ishii, K., Kawachi, T., Sasaki, H., Kono, A. K., Fukuda, T., Kojima, Y., & Mori, E. (2005). Voxel-based morphometric comparison between early- and late-onset mild Alzheimer’s disease and assessment of diagnostic performance of z score images. AJNR American Journal of Neuroradiology, 26(2), 333–340.PubMedGoogle Scholar
  10. Jack Jr., C. R., Bernstein, M. A., Fox, N. C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P. J., L Whitwell, J., Ward, C., Dale, A. M., Felmlee, J. P., Gunter, J. L., Hill, D. L., Killiany, R., Schuff, N., Fox-Bosetti, S., Lin, C., Studholme, C., DeCarli, C. S., Krueger, G., Ward, H. A., Metzger, G. J., Scott, K. T., Mallozzi, R., Blezek, D., Levy, J., Debbins, J. P., Fleisher, A. S., Albert, M., Green, R., Bartzokis, G., Glover, G., Mugler, J., & Weiner, M. W. (2008). The Alzheimer's disease neuroimaging initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging: JMRI, 27, 685–691.CrossRefPubMedGoogle Scholar
  11. Kabani, N., MacDonald, D., Holmes, C. J., & Evans, A. (1998). A 3D atlas of the human brain. NeuroImage, 7, S717.Google Scholar
  12. Kloppel, S., Stonnington, C.M., Chu, C., Draganski, B., Scahill, R.I., Rohrer, J.D., Fox, N.C., Jack Jr, C.R., Ashburner, J., & Frackowiak, R.S.J. (2008). Automatic classification of MR scans in Alzheimer’s disease Brain 131(Pt 3):681–689.Google Scholar
  13. Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. International Conference on Neural Information Processing Systems, pp. 1097–1105.Google Scholar
  14. Lécun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proc IEEE, 86, 2278–2324.CrossRefGoogle Scholar
  15. Lerch, J. P., Pruessner, J., Zijdenbos, A. P., Collins, D. L., Teipel, S. J., Hampel, H., & Evans, A. C. (2008). Automated cortical thickness measurements from MRI can accurately separate Alzheimer's patients from normal elderly controls. Neurobiology of Aging, 29, 23–30.CrossRefPubMedGoogle Scholar
  16. Li, R., Zhang, W., Suk, H.I., Wang, L., Li, J., Shen, D., Ji, S., (2014). Deep learning based imaging data completion for improved brain disease diagnosis. Medical image computing and computer-assisted intervention: MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention 17, 305–312.Google Scholar
  17. Lin, T.Y., Roychowdhury, A., & Maji, S. (2015). Bilinear CNN models for fine-grained visual recognition. IEEE International Conference on Computer Vision, Santiago, Chile, pp 1449–1457.Google Scholar
  18. Liu, S., Cai, W., Che, H., Pujol, S., Kikinis, R., Feng, D., & Fulham, M. J. (2015). Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease. IEEE Transactions on Biomedical Engineering, 62, 1132–1140.CrossRefPubMedGoogle Scholar
  19. Lu, S., Xia, Y., Cai, T.W., & Feng, D.D. (2015). Semi-supervised manifold learning with affinity regularization for Alzheimer's disease identification using positron emission tomography imaging. 2015 37th Annual International Conference of the Ieee Engineering in Medicine and Biology Society (Embc), pp. 2251–2254.Google Scholar
  20. Minati, L., Edginton, T., Bruzzone, M. G., & Giaccone, G. (2009). Reviews: Current concepts in Alzheimer's disease: A multidisciplinary review. American Journal of Alzheimers Disease & Other Dementias, 24, 95–121.CrossRefGoogle Scholar
  21. Shen, D., Wu, G., & Suk, H. I. (2017). Deep learning in medical image analysis. Annual Review of Biomedical Engineering, 19, 221.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Silveira M, Marques, J. (2010). Boosting Alzheimer disease diagnosis using PET images. 20th IEEE international conference on pattern recognition (ICPR), pp. 2556–2559.Google Scholar
  23. Sled, J. G., Zijdenbos, A. P., & Evans, A. C. (1998). A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Trans Med Imaging, 17, 87–97.CrossRefPubMedGoogle Scholar
  24. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15, 1929–1958.Google Scholar
  25. Suk, H.I., Shen, D., 2013. Deep learning-based feature representation for AD/MCI classification. Medical image computing and computer-assisted intervention: MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention 16, 583–590.Google Scholar
  26. Suk, H. I., Lee, S. W., & Shen, D. (2014). Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage, 101, 569–582.CrossRefPubMedPubMedCentralGoogle Scholar
  27. Suk, H. I., Lee, S. W., & Shen, D. (2015). Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Structure and Function, 220, 841–859.CrossRefPubMedGoogle Scholar
  28. Wang, Y., Nie, J., Yap, P. T., Shi, F., Guo, L., & Shen, D. (2011). Robust deformable-surface-based skull-stripping for large-scale studies. Medical Image Computing and Computer-Assisted Intervention – MICCAI, 14(3), 635–642.PubMedGoogle Scholar
  29. Wang, Y., Zhang, P., An, L., Ma, G., Kang, J., Shi, F., Wu, X., Zhou, J., Lalush, D. S., & Lin, W. (2016). Predicting standard-dose PET image from low-dose PET and multimodal MR images using mapping-based sparse representation. Physics in Medicine and Biology, 61(2), 791–812.CrossRefPubMedGoogle Scholar
  30. Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). Learning to track for spatio-temporal action localization. pp. 3164–3172.Google Scholar
  31. Yan, W., Ma, G., Le, A., Feng, S., Pei, Z., Xi, W., Zhou, J., & Shen, D. (2017). Semi-supervised tripled dictionary learning for standard-dose PET image prediction using low-dose PET and multimodal MRI. IEEE Transactions on Biomedical Engineering, 64, 569–579.CrossRefGoogle Scholar
  32. Zeiler, M.D. (2012). ADADELTA: An adaptive learning rate method. Computer Science.Google Scholar
  33. Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. Basel: Springer International Publishing.CrossRefGoogle Scholar
  34. Zhang, D., Wang, Y., Zhou, L., Yuan, H., & Shen, D. (2011). Multimodal classification of Alzheimer's disease and mild cognitive impairment. NeuroImage, 55, 856–867.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Instrument Science and Engineering, School of EIEEShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Shanghai Engineering Research Center for Intelligent Diagnosis and Treatment InstrumentShanghai Jiao Tong UniversityShanghaiChina
  3. 3.School of Information EngineeringZhengzhou UniversityZhengzhouChina

Personalised recommendations