Multi-task Dictionary Learning Based on Convolutional Neural Networks for Longitudinal Clinical Score Predictions in Alzheimer’s Disease

  • Qunxi DongEmail author
  • Jie Zhang
  • Qingyang Li
  • Pau M. Thompson
  • Richard J. Caselli
  • Jieping Ye
  • Yalin Wang
  • for the Alzheimer’s Disease Neuroimaging Initiative
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1072)


Computer-aided diagnosis (CAD) systems for medical images are seen as effective tools to improve the efficiency of diagnosis and prognosis of Alzheimer’s disease (AD). The current state-of-the-art models for many images analyzing tasks are based on Convolutional Neural Networks (CNN). However, the lack of training data is a common challenge in applying CNN to the diagnosis of AD and its prodromal stages. Another challenge for CAD applications is the controversy between the requiring of longitudinal cortical structural information for higher diagnosis/prognosis accuracy and the computing ability for processing varied imaging features. To address these two challenges, we propose a novel computer-aided AD diagnosis system CNN-Stochastic Coordinate Coding (MSCC) which integrates CNN with transfer learning strategy, a novel MSCC algorithm and our effective AD-related biomarkers–multivariate morphometry statistics (MMS). We applied the novel CNN-MSCC system on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset to predict future cognitive clinical measures with baseline Hippocampal/Ventricle MMS features and cortical thickness. The experimental results showed that CNN-MSCC achieved superior results. The proposed system may aid in expediting the diagnosis of AD progress, facilitating earlier clinical intervention, and resulting in improved clinical outcomes.


Computer-aided diagnosis Multi-task dictionary learning Convolutional Neural Networks (CNN) Transfer learning Alzheimer’s Disease 



Algorithm development and image analysis for this study was funded, in part, by the National Institute on Aging (RF1AG051710 to QD, JZ, PMT, JY and YW, R01EB025-032 to YW, R01HL128818 to QD and YW, R01AG031581 and P30AG19610 to RJC, U54EB020403 to PMT and YW), the National Science Foundation (IIS-1421165 to JZ and YW), and Arizona Alzheimer’s Consortium (JZ, RJC and YW). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Tesla K40 GPU used for this research.

Data collection and sharing for this project was funded by the ADNI (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health ( The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California.


  1. 1.
    Brookmeyer, R., Johnson, E., Ziegler-Graham, K., Michael Arrighi, H.: Forecasting the global burden of Alzheimer’s disease. Alzheimer’s Dementia 3(3), 186–191 (2007)CrossRefGoogle Scholar
  2. 2.
    Folstein, M.E.: A practical method for grading the cognitive state of patients for the children. J. Psychiatr res 12, 189–198 (1975)CrossRefGoogle Scholar
  3. 3.
    Rosen, W.G., Mohs, R.C., Davis, K.L.: A new rating scale for Alzheimer’s disease. Am. J. Psychiatry (1984)Google Scholar
  4. 4.
    Buckner, R.L.: Memory and executive function in aging and AD: multiple factors that cause decline and reserve factors that compensate. Neuron 44(1), 195–208 (2004)CrossRefGoogle Scholar
  5. 5.
    Thompson, P.M., et al.: Mapping hippocampal and ventricular change in Alzheimer disease. Neuroimage 22(4), 1754–1766 (2004)CrossRefGoogle Scholar
  6. 6.
    Chung, M.K., Robbins, S., Evans, A.C.: Unified statistical approach to cortical thickness analysis. In: Christensen, G.E., Sonka, M. (eds.) IPMI 2005. LNCS, vol. 3565, pp. 627–638. Springer, Heidelberg (2005). Scholar
  7. 7.
    Frisoni, G.B., Fox, N.C., Jack Jr., C.R., Scheltens, P., Thompson, P.M.: The clinical use of structural MRI in Alzheimer disease. Nat. Rev. Neurol. 6(2), 67 (2010)CrossRefGoogle Scholar
  8. 8.
    Cacciaglia, R., et al.: Effects of APOE-\(\varepsilon \)4 allele load on brain morphology in a cohort of middle-aged healthy individuals with enriched genetic risk for Alzheimer’s disease. Alzheimer’s Dementia 14(7), 902–912 (2018)Google Scholar
  9. 9.
    Operto, G., et al.: White matter microstructure is altered in cognitively normal middle-aged APOE-\(\varepsilon \)4 homozygotes. Alzheimer’s Res. Ther. 10(1), 48 (2018)CrossRefGoogle Scholar
  10. 10.
    Dong, Q., et al.: Applying surface-based hippocampal morphometry to study APOE-E4 allele dose effects in cognitively unimpaired subjects. NeuroImage Clin. 22, 101744 (2019)CrossRefGoogle Scholar
  11. 11.
    Shi, J., et al.: Studying ventricular abnormalities in mild cognitive impairment with hyperbolic Ricci flow and tensor-based morphometry. NeuroImage 104, 1–20 (2015)CrossRefGoogle Scholar
  12. 12.
    Fan, Y., Wang, G., Lepore, N., Wang, Y.: A tetrahedron-based heat flux signature for cortical thickness morphometry analysis. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 420–428. Springer, Cham (2018). Scholar
  13. 13.
    Zhang, J., et al.: Multi-task sparse screening for predicting future clinical scores using longitudinal cortical thickness measures. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1406–1410. IEEE (2018)Google Scholar
  14. 14.
    Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 806–813 (2014)Google Scholar
  15. 15.
    Zhang, J.: Deep transfer learning via restricted Boltzmann machine for document classification. In: 2011 10th International Conference on Machine Learning and Applications and Workshops (ICMLA), vol. 1, pp. 323–326. IEEE (2011)Google Scholar
  16. 16.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  17. 17.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  18. 18.
    Turaga, S.C., et al.: Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Comput. 22(2), 511–538 (2010)CrossRefGoogle Scholar
  19. 19.
    Hazlett, H.C., et al.: Early brain development in infants at high risk for autism spectrum disorder. Nature 542(7641), 348 (2017)CrossRefGoogle Scholar
  20. 20.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  21. 21.
    Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via L1 minimization. Proc. Natl. Acad. Sci. 100(5), 2197–2202 (2003)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 689–696. ACM, New York (2009)Google Scholar
  23. 23.
    Lin, B., et al.: Stochastic coordinate coding and its application for drosophila gene expression pattern annotation. arXiv preprint arXiv:1407.8147 (2014)
  24. 24.
    Zhang, J., et al.: Hyperbolic space sparse coding with its application on prediction of Alzheimer’s disease in mild cognitive impairment. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9900, pp. 326–334. Springer, Cham (2016). Scholar
  25. 25.
    Zhang, J., et al.: Applying sparse coding to surface multivariate tensor-based morphometry to predict future cognitive decline. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 646–650. IEEE (2016)Google Scholar
  26. 26.
    Zhang, D., Shen, D., Initiative, A.D.N., et al.: Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. NeuroImage 59(2), 895–907 (2012)CrossRefGoogle Scholar
  27. 27.
    Zhang, W., et al.: Deep model based transfer and multi-task learning for biological image analysis. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1475–1484. ACM (2015)Google Scholar
  28. 28.
    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 248–255. IEEE (2009)Google Scholar
  29. 29.
    Kermany, D.S., et al.: Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5), 1122–1131 (2018)CrossRefGoogle Scholar
  30. 30.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodol.) 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Weiner, M.W., et al.: The Alzheimer’s disease neuroimaging initiative: a review of papers published since its inception. Alzheimer’s Dementia 9(5), e111–e194 (2013)CrossRefGoogle Scholar
  32. 32.
    Canutescu, A.A., Dunbrack, R.L.: Cyclic coordinate descent: a robotics algorithm for protein loop closure. Protein Sci. 12(5), 963–972 (2003)CrossRefGoogle Scholar
  33. 33.
    Zhang, T.: Solving large scale linear prediction problems using stochastic gradient descent algorithms. In: Proceedings of the Twenty-First International Conference on Machine Learning, p. 116. ACM (2004)Google Scholar
  34. 34.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Duchesne, S., Caroli, A., Geroldi, C., Louis Collins, D., Frisoni, G.B.: Relating one-year cognitive change in mild cognitive impairment to baseline MRI features. NeuroImage 47(4), 1363–1370 (2009)CrossRefGoogle Scholar
  36. 36.
    Jack Jr., C.R., et al.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magn. Reson. Imaging: Official J. Int. Soc. Magn. Reson. Med. 27(4), 685–691 (2008)CrossRefGoogle Scholar
  37. 37.
    Patenaude, B., Smith, S.M., Kennedy, D.N., Jenkinson, M.: A Bayesian model of shape and appearance for subcortical brain segmentation. Neuroimage 56(3), 907–922 (2011)CrossRefGoogle Scholar
  38. 38.
    Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. In: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1987, pp. 163–169. ACM, New York (1987)Google Scholar
  39. 39.
    Wang, Y., et al.: Surface-based TBM boosts power to detect disease effects on the brain: an n = 804 ADNI study. Neuroimage 56(4), 1993–2010 (2011)CrossRefGoogle Scholar
  40. 40.
    Han, X., Chenyang, X., Prince, J.L.: A topology preserving level set method for geometric deformable models. IEEE Trans. Pattern Anal. Mach. Intell. 25(6), 755–768 (2003)CrossRefGoogle Scholar
  41. 41.
    Wang, Y., Chan, T.F., Toga, A.W., Thompson, P.M.: Multivariate tensor-based brain anatomical surface morphometry via holomorphic one-forms. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009. LNCS, vol. 5761, pp. 337–344. Springer, Heidelberg (2009). Scholar
  42. 42.
    Fischl, B.: Freesurfer. Neuroimage 62(2), 774–781 (2012)CrossRefGoogle Scholar
  43. 43.
    Chung, M.K., Dalton, K.M., Davidson, R.J.: Tensor-based cortical surface morphometry via weighted spherical harmonic representation. IEEE Trans. Med. Imaging 27(8), 1143–1151 (2008)CrossRefGoogle Scholar
  44. 44.
    Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  45. 45.
    Zhou, J., Liu, J., Narayan, V.A., Ye, J.: Modeling disease progression via multi-task learning. Neuroimage 78, 233–248 (2013)CrossRefGoogle Scholar
  46. 46.
    Suzuki, K.: Overview of deep learning in medical imaging. Radiol. Phys. Technol. 10(3), 257–273 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • Qunxi Dong
    • 1
    Email author
  • Jie Zhang
    • 1
  • Qingyang Li
    • 1
  • Pau M. Thompson
    • 2
  • Richard J. Caselli
    • 3
  • Jieping Ye
    • 4
  • Yalin Wang
    • 1
  • for the Alzheimer’s Disease Neuroimaging Initiative
  1. 1.School of Computing, Informatics, and Decision Systems EngineeringArizona State UniversityTempeUSA
  2. 2.Imaging Genetics Center, Institute for Neuroimaging and InformaticsUniversity of Southern CaliforniaLos AngelesUSA
  3. 3.Department of NeurologyMayo Clinic ArizonaScottsdaleUSA
  4. 4.Department of Computational Medicine and BioinformaticsUniversity of MichiganAnn ArborUSA

Personalised recommendations