Multimedia Tools and Applications

, Volume 78, Issue 18, pp 25807–25828 | Cite as

Multi stream 3D hyper-densely connected network for multi modality isointense infant brain MRI segmentation

  • Saqib QamarEmail author
  • Hai Jin
  • Ran Zheng
  • Parvez Ahmad


Automatic accurate segmentation of medical images has significant role in computer-aided diagnosis and disease treatment. The segmentation of cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM) tissues plays an important role in infant brain structure for studying early brain development. However, this task is very challenging due to low contrast between GM and WM in isointense phase (approximately 6-8 months of age). In this study, we develop a hyper-densely connected convolutional neural network (CNN) for segmentation of volumetric infant brain. The proposed model provides dense connection between layers to improve the performance of flow information in the network. It also allows the multiscale contextual information by concatenating the feature maps of early, intermediate, and later layers. This architecture employs MR-T1 and T2 as input, which are processed in two separate independent paths, and then their low, intermediate, and high layer features are fused for final segmentation. An important change relative to earlier densely connected networks is the application of direct layer connections from the same and different paths. In this scenario, each modality is processed in an independent path, and dense connections occur not only between layers within the same path, but also between layers in different paths. Adopting such dense connectivity leads to benefits of deep supervision and improved gradient flow. Furthermore, by combining the feature maps of early, intermediate, and late convolutional layers, our architecture injects multiscale information into the final segmentation. This suggested approach is examined in the MICCAI Grand Challenge iSEG and obtains significant advantages over existing approaches in terms of parameter efficiency and segmentation accuracy on 6-month infant brain MRI segmentation.


Deep learning 3D CNN Infant brain segmentation Multi modality MRI 



This research is supported by National Key Research and Development Program of China under grant 2018YFB1003500.


  1. 1.
    Anbeek P, Vincken KL, Groenendaal F, Koeman A, Van Osch MJ (2008) Probabilistic brain tissue segmentation in neonatal magnetic resonance imaging. Pediatr Res 2(6):158–163CrossRefGoogle Scholar
  2. 2.
    Bui TD, Shin J, Moon T (2017) 3D densely convolutional networks for volumetric segmentation. arXiv:1709.03199
  3. 3.
    Cardoso M, Melbourne A, Kendall G, Modat M, Robertson N, Marlow N, Ourselin S (2013) Adapt: an adaptive preterm segmentation algorithm for neonatal brain MRI. NeuroImage 65:97–108CrossRefGoogle Scholar
  4. 4.
    Chen S-Q, Zhan R-H, Hu J-M, Zhang J (2017) Feature fusion based on convolutional neural network for SAR ATR. In: ITM Web of conferences, vol 12. EDP Sciences, p 05001Google Scholar
  5. 5.
    Chen H, Dou Q, Yu L, Qin J, Heng P-A (2018) Voxresnet: Deep voxelwise residual networks for brain segmentation from 3D MR images. NeuroImage 170:446–455. Segmenting the brainCrossRefGoogle Scholar
  6. 6.
    Chen H, Yu L, Dou Q, Shi L, Mok VCT, Heng PA (2015) Automatic detection of cerebral microbleeds via deep learning based 3D feature representation. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI), pp 764–767Google Scholar
  7. 7.
    Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: Learning dense volumetric segmentation from sparse annotation. In: Ourselin S, Joskowicz L, Sabuncu MR, Unal G, Wells W (eds) Medical image computing and computer-assisted intervention – MICCAI 2016. Springer International Publishing, Cham, pp 424–432Google Scholar
  8. 8.
    Ciresan D, Giusti A, Gambardella LM, Schmidhuber J (2012) Deep neural networks segment neuronal membranes in electron microscopy images. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates, Inc, pp 2843–2851Google Scholar
  9. 9.
    Criminisi A, Shotton J, Zikic D (2014) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med ImagingGoogle Scholar
  10. 10.
    Dolz J, Desrosiers C, Ayed IB (2018) 3D fully convolutional networks for subcortical segmentation in MRI: a large-scale study. NeuroImage 170:456–470. Segmenting the brainCrossRefGoogle Scholar
  11. 11.
    Dolz J, Massoptier L, Vermandel M (2015) Segmentation algorithms of subcortical brain structures on MRI for radiotherapy and radiosurgery: a survey. IRBM 36(4):200–212CrossRefGoogle Scholar
  12. 12.
    Dou Q, Chen H, Yu L, Zhao L, Qin J, Wang D, Mok VC, Shi L, Heng PA (2016) Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks. IEEE Trans Med Imaging 35(5):1182–1195CrossRefGoogle Scholar
  13. 13.
    Drozdzal M, Vorontsov E, Chartrand G, Kadoury S, Pal C (2016) The importance of skip connections in biomedical image segmentation. arXiv:1608.04117
  14. 14.
    Fechter T, Adebahr S, Baltas D, Ayed IB, Desrosiers C, Dolz J (2017) Esophagus segmentation in CT via 3D fully convolutional neural network and random walk. Med Phys 44(12):6341–6352CrossRefGoogle Scholar
  15. 15.
    Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: International conference on artificial intelligence and statistics, vol 15, pp 315–323Google Scholar
  16. 16.
    Gui L, Lisowski R, Faundez T, Hüppi PS, Lazeyras F, Kocher M (2012) Morphology-driven automatic segmentation of MR images of the neonatal brain. Med Image Anal 16(8):1565–1579CrossRefGoogle Scholar
  17. 17.
    Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin P-M, Larochelle H (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31CrossRefGoogle Scholar
  18. 18.
    Havaei M, Guizard N, Larochelle H, Jodoin P-M (2016) Deep learning trends for focal brain pathology segmentation in MRI. Springer International Publishing, Cham, pp 125–148Google Scholar
  19. 19.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 770–778Google Scholar
  20. 20.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: 2015 IEEE International conference on computer vision (ICCV), pp 1026–1034Google Scholar
  21. 21.
    Huang G, Liu Z, van der Maaten L, Weinberger K (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269Google Scholar
  22. 22.
    Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456Google Scholar
  23. 23.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia, MM ’14. ACM, New York, pp 675–678Google Scholar
  24. 24.
    Kamnitsas K, Ledig C, Newcombe VFJ, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal 36:61–78CrossRefGoogle Scholar
  25. 25.
    Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1646–1654Google Scholar
  26. 26.
    Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
  27. 27.
    Kontschieder P, Bulo SR, Bischof H, Pelillo M (2011) And structured class-labels in random forests for semantic image labelling. In: Proceedings of the 2011 international conference on computer vision, ICCV ’11. IEEE Computer Society, Washington, pp 2190–2197Google Scholar
  28. 28.
    Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun ACM 60(6):84–90CrossRefGoogle Scholar
  29. 29.
    Lan X, Ma AJ, Yuen PC (2014) Multi-cue visual tracking using robust feature-level fusion based on joint sparse representation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1194–1201Google Scholar
  30. 30.
    Lan X, Ma AJ, Yuen PC, Chellappa R (2015) Joint sparse representation and robust feature-level fusion for multi-cue visual tracking. IEEE Trans Image Process 24(12):5826–5841MathSciNetzbMATHCrossRefGoogle Scholar
  31. 31.
    Lan X, Zhang S, Yuen PC, Chellappa R (2018) Learning common and feature-specific patterns: a novel multiple-sparse-representation-based tracker. IEEE Trans Image Process 27(4):2022–2037MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRefGoogle Scholar
  33. 33.
    Li W, Shi F, Li G, Gao Y, Lin W, Gilmore JH, Shen D (2014) Segmentation of neonatal brain MR images using patch-driven level sets. NeuroImage 84:141–158, 1CrossRefGoogle Scholar
  34. 34.
    Li W, Shi F, Lin W, Gilmore JH, Shen D (2011) Automatic segmentation of neonatal images using convex optimization and coupled level sets. NeuroImage 58(3):805–817CrossRefGoogle Scholar
  35. 35.
    Liu C, Wechsler H (2001) Shape-and texture-based enhanced fisher classifier for face recognition. IEEE Trans on Image Process 10(4):598–608zbMATHCrossRefGoogle Scholar
  36. 36.
    Melbourne A, Cardoso MJ, Kendall GS, Robertson NJ, Marlow N, Ourselin S (2012) Neobrains12 challenge: adaptive neonatal MRI brain segmentation with myelinated white matter class and automated extraction of ventricles i-iv. MICCAI grand challenge: neonatal brain segmentation (NeoBrainSI2), pp 16–21Google Scholar
  37. 37.
    Milletari F, Navab N, Ahmadi SA (2016) V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3D vision (3DV), pp 565–571Google Scholar
  38. 38.
    Moeskops P, Benders Manon JNL, Chita SM, Kersbergen KJ, Groenendaal F, de Vries LS, Viergever MA, Isgum I (2015) Automatic segmentation of MR brain images of preterm infants using supervised classification. NeuroImage 118:628–641CrossRefGoogle Scholar
  39. 39.
    Moeskops P, Viergever MA, Mendrik AM, de Vries LS, Benders MJNL, Išgum I (2016) Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans Med Imaging 35(5):1252–1261CrossRefGoogle Scholar
  40. 40.
    Nie D, Wang L, Gao Y, Sken D (2016) Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: 2016 IEEE 13Th international symposium on biomedical imaging (ISBI), pp 1342–1345Google Scholar
  41. 41.
    Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N (eds) Medical image computing and computer-assisted intervention – MICCAI 2013. Springer, Berlin, pp 246–253Google Scholar
  42. 42.
    Prastawa M, Gilmore JH, Lin W, Gerig G (2005) Automatic segmentation of MR images of the developing newborn brain. Med Image Anal 9(5):457–66CrossRefGoogle Scholar
  43. 43.
    Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention – MICCAI 2015. Springer International Publishing, Cham, pp 234–241Google Scholar
  44. 44.
    Roth HR, Lu L, Seff A, Cherry KM, Hoffman J, Wang S, Liu J, Turkbey E, Summers RM (2014) A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention – MICCAI 2014. Springer International Publishing, Cham, pp 520–527Google Scholar
  45. 45.
    Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39(4):640–651CrossRefGoogle Scholar
  46. 46.
    Shi F, Fan Y, Tang S, Gilmore JH, Lin W, Shen D (2010) Neonatal brain image segmentation in longitudinal MRI studies. NeuroImage 49(1):391–400CrossRefGoogle Scholar
  47. 47.
    Shi F, Yap P-T, Fan Y, Gilmore JH, Lin W, Shen D (2010) Construction of multi-region-multi-reference atlases for neonatal brain MRI segmentation. NeuroImage 51(2):684–693CrossRefGoogle Scholar
  48. 48.
    Shi F, Yap P-T, Guorong W u, Jia H, Gilmore JH, Lin W, Shen D (2011) Infant brain atlases from neonates to 1- and 2-year-olds. PloS one 6(4):e18746CrossRefGoogle Scholar
  49. 49.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958MathSciNetzbMATHGoogle Scholar
  50. 50.
    Wang L, Nie D, Li G, Puybareau E, Dolz J, Zhang Q, Wang F, Xia J, Wu Z, Chen J, Thung K-H, Bui TD, Shin J, Zeng G, Zheng G, Fonov VS, Doyle A, Xu Y, Moeskops P, Pluim JPW, Desrosiers C, Ayed IB, Sanroma G, Benkarim OM, Casamitjana A, Vilaplana V, Lin W, Li G, Shen D (2019) Benchmark on automatic 6-month-old infant brain segmentation algorithms: the iseg-2017 challenge. IEEE Trans Med ImagingGoogle Scholar
  51. 51.
    Wang S, Kuklisova-Murgasova M, Schnabel JA (2012) An atlas-based method for neonatal MR brain tissue segmentation. In: MICCAI Grand challenge: Neonatal brain segmentation, pp 28–35Google Scholar
  52. 52.
    Warfield SK, Zou KH, Wells WM (2004) Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 23(7):903–921CrossRefGoogle Scholar
  53. 53.
    Weisenfeld NI, Mewes AUJ, Warfield SK (2006) Segmentation of newborn brain MRI. In: 3Rd IEEE international symposium on biomedical imaging: Nano to macro, 2006, pp 766–769Google Scholar
  54. 54.
    Weisenfeld NI, Warfield SK (2009) Automatic segmentation of newborn brain MRI. NeuroImage 47(2):564–572CrossRefGoogle Scholar
  55. 55.
    Wu J, Avants B (2012) Automatic registration-based segmentation for neonatal brains using ANTs and Atropos. In: MICCAI Grand challenge: Neonatal brain segmentation (neobrains12), pp 36–43Google Scholar
  56. 56.
    Xue H, Srinivasan L, Jiang S, Rutherford M, Edwards DA, Rueckert D, Hajnal JV (2007) Automatic segmentation and reconstruction of the cortex from neonatal MRI. NeuroImage 38(3):461–477CrossRefGoogle Scholar
  57. 57.
    Yang J, Yang J, Zhang D, Lu J (2003) Feature fusion: parallel strategy vs. serial strategy. Pattern Recognit 36(6):1369–1381zbMATHCrossRefGoogle Scholar
  58. 58.
    Yu L, Cheng J-Z, Dou Q, Yang X, Chen H, Qin J (2017) Automatic 3D cardiovascular MR segmentation with densely-connected volumetric convnets. In: Descoteaux M, Maier-Hein L, Franz A, Jannin P, Collins DL, Duchesne S (eds) International conference on medical image computing and computer assisted intervention - MICCAI. Springer International Publishing, Cham, pp 287–295Google Scholar
  59. 59.
    Zhang W, Li R, Deng H, Li W, Lin W, Ji S, Shen D (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage 108:214–224CrossRefGoogle Scholar
  60. 60.
    Zhuang S, Awate SP, Licht DJ, Gee JC (2007) Clinical neonatal brain MRI segmentation using adaptive nonparametric data models and intensity-based markov priors. In: Proceedings of the 10th international conference on medical image computing and computer-assisted intervention - volume part I, MICCAI’07. Springer, Berlin, pp 883–890Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Services Computing Technology and System Laboratory, School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina
  2. 2.Cluster and Grid Computing Laboratory, School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina
  3. 3.Big Data Technology and System Laboratory, School of Computer Science and TechnologyHuazhong University of Science and TechnologyWuhanChina

Personalised recommendations