Skip to main content

Advertisement

SpringerLink
Log in

Multi-task neural networks for joint hippocampus segmentation and clinical score regression

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Feature representations extracted from hippocampus in magnetic resonance (MR) images are widely used in computer-aided Alzheimer’s disease (AD) diagnosis, and thus accurate segmentation for the hippocampus has been remaining an active research topic. Previous studies for hippocampus segmentation require either human annotation which is tedious and error-prone or pre-processing MR images via time-consuming non-linear registration. Although many automatic segmentation approaches have been proposed, their performance is often limited by the small size of hippocampus and complex confounding information around the hippocampus. In particular, human-engineered features extracted from segmented hippocampus regions (e.g., the volume of the hippocampus) are essential for brain disease diagnosis, while these features are independent of diagnosis models, leading to sub-optimal performance. To address these issues, we propose a multi-task deep learning (MDL) method for joint hippocampus segmentation and clinical score regression using MR images. The prominent advantages of our MDL method lie on that we don’t need any time-consuming non-linear registration for pre-processing MR images, and features generated by MDL are consistent with subsequent diagnosis models. Specifically, we first align all MR images onto a standard template, followed by a patch extraction process to approximately locate hippocampus regions in the template space. Using image patches as input data, we develop a multi-task convolutional neural network (CNN) for joint hippocampus segmentation and clinical score regression. The proposed CNN network contains two subnetworks, including 1) a U-Net with a Dice-like loss function for hippocampus segmentation, and 2) a convolutional neural network with a mean squared loss function for clinical regression. Note that these two subnetworks share a part of network parameters, to exploit the inherent association between these two tasks. We evaluate the proposed method on 407 subjects with MRI data from baseline Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. The experimental results suggest that our MDL method achieves promising results in both tasks of hippocampus segmentation and clinical score regression, compared with several state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://adni.loni.usc.edu/data-samples/mri/

  2. http://mipav.cit.nih.gov/index.php

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: A system for large-scale machine learning. In: Proceedings of the 12th USENIX symposium on operating systems design and implementation

  2. Ahmed O B, Mizotin M, Benois-Pineau J, Allard M, Catheline G, Amar C B, Initiative A D N et al (2015) Alzheimer’s disease diagnosis on structural MR images using circular harmonic functions descriptors on hippocampus and posterior cingulate cortex. Comput Med Imaging Graph 44:13–25

    Article  Google Scholar 

  3. Ashburner J (2007) A fast diffeomorphic image registration algorithm. NeuroImage 38(1):95–113

    Article  Google Scholar 

  4. Barnes J, Boyes R, Lewis E, Schott J, Frost C, Scahill R, Fox N (2007) Automatic calculation of hippocampal atrophy rates using a hippocampal template and the boundary shift integral. Neurobiol Aging 28(11):1657–1663

    Article  Google Scholar 

  5. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

    Book  Google Scholar 

  6. Cao X, Gao Y, Yang J, Wu G, Shen D (2016) Learning-based multimodal image registration for prostate cancer radiation therapy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 1–9

  7. Cao X, Yang J, Gao Y, Guo Y, Wu G, Shen D (2017) Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis. Medical Image Analysis

  8. Carmichael O T, Aizenstein H A, Davis S W, Becker J T, Thompson P M, Meltzer C C, Liu Y (2005) Atlas-based hippocampus segmentation in Alzheimer’s disease and mild cognitive impairment. NeuroImage 27(4):979–990

    Article  Google Scholar 

  9. Chang C C, Lin C J (2011) Libsvm: A library for support vector machines. ACM Trans Intell Syst Technol 2(3):27

    Article  Google Scholar 

  10. Chincarini A, Bosco P, Calvini P, Gemme G, Esposito M, Olivieri C, Rei L, Squarcia S, Rodriguez G, Bellotti R et al (2011) Local MRI analysis approach in the diagnosis of early and prodromal Alzheimer’s disease. NeuroImage 58 (2):469–480

    Article  Google Scholar 

  11. Clark K A, Woods R P, Rottenberg D A, Toga A W, Mazziotta J C (2006) Impact of acquisition protocols and processing streams on tissue segmentation of T1 weighted MR images. NeuroImage 29(1):185–202

    Article  Google Scholar 

  12. Coupé P, Manjón J V, Fonov V, Pruessner J, Robles M, Collins D L (2011) Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation. NeuroImage 54(2):940–954

    Article  Google Scholar 

  13. Coupé P, Eskildsen S F, Manjón J V, Fonov V S, Collins D L (2012) Simultaneous segmentation and grading of anatomical structures for patient’s classification: Application to Alzheimer’s disease. NeuroImage 59(4):3736–3747

    Article  Google Scholar 

  14. Dagher A, Owen A M, Boecker H, Brooks D J (2001) The role of the striatum and hippocampus in planning: A PET activation study in Parkinson’s disease. Brain 124(5):1020–1032

    Article  Google Scholar 

  15. Dill V, Franco A R, Pinho M S (2015) Automated methods for hippocampus segmentation: The evolution and a review of the state of the art. Neuroinformatics 13 (2):133

    Article  Google Scholar 

  16. Folstein M F, Folstein S E, McHugh P R (1975) “mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 12(3):189–198

    Article  Google Scholar 

  17. Hao Y, Wang T, Zhang X, Duan Y, Yu C, Jiang T, Fan Y (2014) Local label learning (LLL) for subcortical structure segmentation: Application to hippocampus segmentation. Hum Brain Mapp 35 (6):2674–2697

    Article  Google Scholar 

  18. Heckemann R A, Hajnal J V, Aljabar P, Rueckert D, Hammers A (2006) Automatic anatomical brain MRI segmentation combining label propagation and decision fusion. NeuroImage 33(1):115–126

    Article  Google Scholar 

  19. Hyman B T, Van Hoesen G W, Damasio A R, Barnes C L (1984) Alzheimer’s disease: Cell-specific pathology isolates the hippocampal formation. Science 225:1168–1171

    Article  Google Scholar 

  20. Iglesias J E, Sabuncu M R (2015) Multi-atlas segmentation of biomedical images: A survey. Med Image Anal 24(1):205–219

    Article  Google Scholar 

  21. Jack C R, Petersen R C, O’Brien P C, Tangalos E G (1992) MR-based hippocampal volumetry in the diagnosis of Alzheimer’s disease. Neurology 42(1):183–183

    Article  Google Scholar 

  22. Jack C R, Bernstein M A, Fox N C, Thompson P, Alexander G, Harvey D, Borowski B, Britson P J, L Whitwell J, Ward C (2008) The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J Magn Reson Imaging 27(4):685–691

    Article  Google Scholar 

  23. Jin K, Peel A L, Mao X O, Xie L, Cottrell B A, Henshall D C, Greenberg D A (2004) Increased hippocampal neurogenesis in Alzheimer’s disease. Proc Natl Acad Sci 101(1):343–347

    Article  Google Scholar 

  24. Kwak K, Yoon U, Lee D K, Kim G H, Seo S W, Na D L, Shim H J, Lee J M (2013) Fully-automated approach to hippocampus segmentation using a graph-cuts algorithm combined with atlas-based segmentation and morphological opening. Magn Reson Imaging 31(7):1190–1196

    Article  Google Scholar 

  25. Lian C, Ruan S, Denoeux T (2015) An evidential classifier based on feature selection and two-step classification strategy. Pattern Recogn 48(7):2318–2327

    Article  Google Scholar 

  26. Lian C, Ruan S, Denœux T, Jardin F, Vera P (2016) Selecting radiomic features from FDG-PET images for cancer treatment outcome prediction. Med Image Anal 32:257–268

    Article  Google Scholar 

  27. Lindner C, Thiagarajah S, Wilkinson J, Consortium T, Wallis G, Cootes T (2013) Fully automatic segmentation of the proximal femur using random forest regression voting. IEEE Trans Med Imaging 32(8):1462–1472

    Article  Google Scholar 

  28. Liu M, Zhang D, Shen D (2015) View-centralized multi-atlas classification for Alzheimer’s disease diagnosis. Hum Brain Mapp 36(5):1847–1865

    Article  Google Scholar 

  29. Liu M, Zhang D, Chen S, Xue H (2016) Joint binary classifier learning for ECOC-based multi-class classification. IEEE Trans Pattern Anal Mach Intell 38 (11):2335–2341

    Article  Google Scholar 

  30. Liu M, Zhang D, Shen D (2016) Relationship induced multi-template learning for diagnosis of Alzheimer’s disease and mild cognitive impairment. IEEE Trans Med Imaging 35(6):1463–1474

    Article  Google Scholar 

  31. Liu M, Zhang J, Yap P T, Shen D (2017) View-aligned hypergraph learning for alzheimer’s disease diagnosis with incomplete multi-modality data. Med Image Anal 36:123–134

    Article  Google Scholar 

  32. Liu M, Zhang J, Adeli E, Shen D (2018) Landmark-based deep multi-instance learning for brain disease diagnosis. Med Image Anal 43:157–168

    Article  Google Scholar 

  33. Moodley K, Minati L, Contarino V, Prioni S, Wood R, Cooper R, D’incerti L, Tagliavini F, Chan D (2015) Diagnostic differentiation of mild cognitive impairment due to Alzheimer’s disease using a hippocampus-dependent test of spatial memory. Hippocampus 25(8):939–951

    Article  Google Scholar 

  34. Pohl K M, Bouix S, Nakamura M, Rohlfing T, McCarley R W, Kikinis R, Grimson W E L, Shenton M E, Wells W M (2007) A hierarchical algorithm for MR brain image parcellation. IEEE Trans Med Imaging 26(9):1201–1212

    Article  Google Scholar 

  35. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. arXiv:150504597

  36. Sled J G, Zijdenbos A P, Evans A C (1998) A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Trans Med Imaging 17(1):87–97

    Article  Google Scholar 

  37. Tulving E, Markowitsch H J (1998) Episodic and declarative memory: Role of the hippocampus. Hippocampus 8(3):198–204

    Article  Google Scholar 

  38. Zandifar A, Fonov V, Coupé P, Pruessner J, Collins DL, Initiative ADN et al (2017) A comparison of accurate automatic hippocampal segmentation methods. NeuroImage

  39. Zarpalas D, Gkontra P, Daras P, Maglaveras N (2014) Accurate and fully automatic hippocampus segmentation using subject-specific 3D optimal local maps into a hybrid active contour model. IEEE J Trans Eng Health Med 2:1–16

    Article  Google Scholar 

  40. Zhang J, Liang J, Zhao H (2013) Local energy pattern for texture classification using self-adaptive quantization thresholds. IEEE Trans Image Process 22(1):31–42

    Article  MathSciNet  Google Scholar 

  41. Zhang J, Gao Y, Wang L, Tang Z, Xia J J, Shen D (2016) Automatic craniomaxillofacial landmark digitization via segmentation-guided partially-joint regression forest model and multiscale statistical features. IEEE Trans Biomed Eng 63(9):1820–1829

    Article  Google Scholar 

  42. Zhang J, Gao Y, Park SH, Zong X, Lin W, Shen D (2017) Structured learning for 3D perivascular spaces segmentation using vascular features. IEEE Transactions on Biomedical Engineering

  43. Zhang J, Liu M, An L, Gao Y, Shen D (2017) Alzheimer’s disease diagnosis using landmark-based features from longitudinal structural MR images. IEEE Journal of Biomedical and Health Informatics

  44. Zhang J, Liu M, Shen D (2017) Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks. IEEE Trans Image Process 26(10):4753–4764

    Article  MathSciNet  Google Scholar 

  45. Zhu X, Suk H I, Shen D (2014) A novel matrix-similarity based loss function for joint regression and classification in AD diagnosis. NeuroImage 100:91–105

    Article  Google Scholar 

  46. Zhu X, Suk H I, Wang L, Lee SW, Shen D (2015) A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Medical Image analysis

  47. Zhu Y, Zhu X, Kim M, Shen D, Wu G (2016) Early diagnosis of Alzheimer’s disease by joint feature selection and classification on temporally structured support vector machine. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 264–272

  48. Zhu Y, Zhu X, Zhang H, Gao W, Shen D, Wu G (2016) Reveal consistent spatial-temporal patterns from dynamic functional connectivity for autism spectrum disorder identification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 106–114

  49. Zhu Y, Zhu X, Kim M, Kaufer D, Wu G (2017) A novel dynamic hyper-graph inference framework for computer assisted diagnosis of neuro-diseases. In: International Conference on Information Processing in Medical Imaging, Springer, pp 158–169

  50. Zou K H, Warfield S K, Bharatha A, Tempany C M, Kaus M R, Haker S J, Wells W M, Jolesz F A, Kikinis R (2004) Statistical validation of image segmentation quality based on a spatial overlap index 1: Scientific reports. Acad Radiol 11(2):178–189

    Article  Google Scholar 

Download references

Acknowledgments

This study was supported by National Natural Science Foundation of China (Nos. 61703301, 61573023) one more support grant and University Science and Technology Project of Shandong Province (No. J17KA086).

Author information

Authors and Affiliations

  1. Taian Tumor Prevention and Treatment Hospital, Taian, 271000, China

    Liang Cao, Long Li, Feng Yin & Hui Shen

  2. Taian Maternity and Child Care Hospital, Taian, 271000, China

    Jifeng Zheng

  3. Taian Institute of Science and Technology Information, Taian, 271000, China

    Xin Fan

  4. Department of Radiology, Duke University, Durham, NC, 27705, USA

    Jun Zhang

Authors
  1. Liang Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Long Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jifeng Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Feng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hui Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, L., Li, L., Zheng, J. et al. Multi-task neural networks for joint hippocampus segmentation and clinical score regression. Multimed Tools Appl 77, 29669–29686 (2018). https://doi.org/10.1007/s11042-017-5581-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5581-1

Keywords

Search

Navigation