Recurrent generative adversarial network for learning imbalanced medical image semantic segmentation

  • Mina RezaeiEmail author
  • Haojin Yang
  • Christoph Meinel


We propose a new recurrent generative adversarial architecture named RNN-GAN to mitigate imbalance data problem in medical image semantic segmentation where the number of pixels belongs to the desired object are significantly lower than those belonging to the background. A model trained with imbalanced data tends to bias towards healthy data which is not desired in clinical applications and predicted outputs by these networks have high precision and low recall. To mitigate imbalanced training data impact, we train RNN-GAN with proposed complementary segmentation mask, in addition, ordinary segmentation masks. The RNN-GAN consists of two components: a generator and a discriminator. The generator is trained on the sequence of medical images to learn corresponding segmentation label map plus proposed complementary label both at a pixel level, while the discriminator is trained to distinguish a segmentation image coming from the ground truth or from the generator network. Both generator and discriminator substituted with bidirectional LSTM units to enhance temporal consistency and get inter and intra-slice representation of the features. We show evidence that the proposed framework is applicable to different types of medical images of varied sizes. In our experiments on ACDC-2017, HVSMR-2016, and LiTS-2017 benchmarks we find consistently improved results, demonstrating the efficacy of our approach.


Imbalanced medical image semantic segmentation Recurrent generative adversarial network 



  1. 1.
    Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from
  2. 2.
    Afshin M, Ayed IB, Punithakumar K, Law M, Islam A, Goela A, Peters T, Li S (2014) Regional assessment of cardiac left ventricular myocardial function via mri statistical features. IEEE Trans Med Imaging 33(2):481–494CrossRefGoogle Scholar
  3. 3.
    Avola D, Cinque L (2008) Encephalic nmr image analysis by textural interpretation. In: Proceedings of the 2008 ACM symposium on applied computing, pp 1338–1342. ACMGoogle Scholar
  4. 4.
    Avola D, Cinque L, Di Girolamo M (2011) A novel t-cad framework to support medical image analysis and reconstruction. In: International conference on image analysis and processing, pp 414–423. SpringerGoogle Scholar
  5. 5.
    Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng PA, Cetin I, Lekadir K, Camara O, Ballester MAG et al (2018) Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical ImagingGoogle Scholar
  6. 6.
    Bi L, Kim J, Kumar A, Feng D (2017) Automatic liver lesion detection using cascaded deep residual networks. arXiv:1704.02703
  7. 7.
    Chollet F et al (2015) KerasGoogle Scholar
  8. 8.
    Christ PF, Ettlinger F, Grun F, Elshaer MEA, Lipkova J, Schlecht S, Ahmaddy F, Tatavarty S, Bickel M, Bilic P, Rempfler M, Hofmann F, D’Anastasi M, Ahmadi S, Kaissis G, Holch J, Sommer WH, Braren R, Heinemann V, Menze BH (2017) Automatic liver and tumor segmentation of CT and MRI volumes using cascaded fully convolutional neural networks. arXiv:1702.05970
  9. 9.
    Ciecholewski M (2011) Support vector machine approach to cardiac spect diagnosis. In: International workshop on combinatorial image analysis, pp 432–443. SpringerGoogle Scholar
  10. 10.
    Douzas G, Bacao F (2018) Effective data generation for imbalanced learning using conditional generative adversarial networks. Expert Syst Appl 91:464–471CrossRefGoogle Scholar
  11. 11.
    Drozdzal M, Chartrand G, Vorontsov E, Shakeri M, Di Jorio L, Tang A, Romero A, Bengio Y, Pal C, Kadoury S (2018) Learning normalized inputs for iterative estimation in medical image segmentation. Med Image Anal 44:1–13CrossRefGoogle Scholar
  12. 12.
    Eslami A, Karamalis A, Katouzian A, Navab N (2013) Segmentation by retrieval with guided random walks: application to left ventricle segmentation in mri. Med Image Anal 17(2):236– 253CrossRefGoogle Scholar
  13. 13.
    Fidon L, Li W, Garcia-Peraza-Herrera LC, Ekanayake J, Kitchen N, Ourselin S, Vercauteren T (2017) Generalised wasserstein dice score for imbalanced multi-class segmentation using holistic convolutional networks. In: International MICCAI Brainlesion workshop, pp 64–76. SpringerGoogle Scholar
  14. 14.
    Fischl B, Salat DH, Van Der Kouwe AJ, Makris N, Ségonne F, Quinn BT, Dale AM (2004) Sequence-independent segmentation of magnetic resonance images. Neuroimage 23:S69–S84CrossRefGoogle Scholar
  15. 15.
    Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative Adversarial Networks ArXiv e-printsGoogle Scholar
  16. 16.
    Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw 18(5-6):602–610CrossRefGoogle Scholar
  17. 17.
    Han X (2017) Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv:1704.07239
  18. 18.
    Hashemi SR, Salehi SSM, Erdogmus D, Prabhu SP, Warfield SK, Gholipour A (2018) Tversky as a loss function for highly unbalanced image segmentation using 3d fully convolutional deep networks. arXiv:1803.11078
  19. 19.
    Inda Maria-del-Mar RB, Seoane J (2014) Glioblastoma multiforme: A look inside its heterogeneous nature. In: Cancer archive 226-239Google Scholar
  20. 20.
    Isensee F, Jaeger PF, Full PM, Wolf I, Engelhardt S, Maier-Hein K H (2017) Automatic cardiac disease assessment on cine-mri via time-series segmentation and domain specific features. In: International workshop on statistical atlases and computational models of the heart, pp 120–129. SpringerGoogle Scholar
  21. 21.
    Ishida T, Niu G, Hu W, Sugiyama M (2017) Learning from complementary labels. In: Advances in neural information processing systems, pp 5639–5649Google Scholar
  22. 22.
    Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  23. 23.
    Jang J, Eo T, Kim M, Choi N, Han D, Kim D, Hwang D (2014) Medical image matching using variable randomized undersampling probability pattern in data acquisition. In: 2014 international conference on electronics, information and communications (ICEIC), pp 1–2.
  24. 24.
    Kaur R, Juneja M, Mandal A (2018) A comprehensive review of denoising techniques for abdominal ct images. Multimedia Tools and Applications pp 1–36Google Scholar
  25. 25.
    Kohl S, Bonekamp D, Schlemmer H, Yaqubi K, Hohenfellner M, Hadaschik B, Radtke J, Maier-Hein KH (2017) Adversarial networks for the detection of aggressive prostate cancer. arXiv:1702.08014
  26. 26.
    LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444CrossRefGoogle Scholar
  27. 27.
    Mahapatra D (2014) Automatic cardiac segmentation using semantic information from random forests. J Digit Imaging 27(6):794–804CrossRefGoogle Scholar
  28. 28.
    Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv:1411.1784
  29. 29.
    Moeskops P, Veta M, Lafarge MW, Eppenhof KAJ, Pluim JPW (2017) Adversarial training and dilated convolutions for brain MRI segmentation. arXiv:1707.03195
  30. 30.
    Nasr GE, Badr E, Joun C (2002) Cross entropy error function in neural networks: Forecasting gasoline demand. In: FLAIRS conference, pp 381–384Google Scholar
  31. 31.
    Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544Google Scholar
  32. 32.
    Peng P, Lekadir K, Gooya A, Shao L, Petersen SE, Frangi AF (2016) A review of heart chamber segmentation for structural and functional analysis using cardiac magnetic resonance imaging. Magn Reson Mater Phys, Biol Med 29(2):155–195CrossRefGoogle Scholar
  33. 33.
    Pohl KM, Fisher J, Grimson WEL, Kikinis R, Wells WM (2006) A bayesian model for joint segmentation and registration. Neuroimage 31(1):228–239CrossRefGoogle Scholar
  34. 34.
    Poudel RP, Lamata P, Montana G (2016) Recurrent fully convolutional neural networks for multi-slice mri cardiac segmentation. In: Reconstruction, segmentation, and analysis of medical images, pp 83–94. SpringerGoogle Scholar
  35. 35.
    Prabhu V, Kuppusamy P, Karthikeyan A, Varatharajan R (2018) Evaluation and analysis of data driven in expectation maximization segmentation through various initialization techniques in medical images. Multimed Tools Appl 77(8):10375–10390CrossRefGoogle Scholar
  36. 36.
    Qiu Q, Song Z (2018) A nonuniform weighted loss function for imbalanced image classification. In: Proceedings of the 2018 international conference on image and graphics processing, pp 78–82. ACMGoogle Scholar
  37. 37.
    Rohé MM, Sermesant M, Pennec X (2017) Automatic multi-atlas segmentation of myocardium with svf-net. In: Statistical atlases and computational modeling of the heart (STACOM) workshopGoogle Scholar
  38. 38.
    Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241. Springer International PublishingGoogle Scholar
  39. 39.
    Rota Bulo S, Neuhold G, Kontschieder P (2017) Loss max-pooling for semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2126– 2135Google Scholar
  40. 40.
    Shahzad R, Gao S, Tao Q, Dzyubachyk O, van der Geest R (2016) Automated cardiovascular segmentation in patients with congenital heart disease from 3d cmr scans: combining multi-atlases and level-sets. In: Reconstruction, segmentation, and analysis of medical images, pp 147–155Google Scholar
  41. 41.
    Sudre CH, Li W, Vercauteren T, Ourselin S, Cardoso M J (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp 240–248. SpringerGoogle Scholar
  42. 42.
    Tustison NJ, Avants BB, Cook PA, Zheng Y, Egan A, Yushkevich PA, Gee JC (2010) N4itk: improved n3 bias correction. IEEE Trans Med Imaging 29(6):1310–1320CrossRefGoogle Scholar
  43. 43.
    Vorontsov E, Tang A, Pal C, Kadoury S (2018) Liver lesion segmentation informed by joint liver segmentation. In: 15th IEEE international symposium on biomedical imaging (ISBI 2018), pp 1332– 1335Google Scholar
  44. 44.
    Vorontsov E, Tang A, Pal C, Kadoury S (2018) Liver lesion segmentation informed by joint liver segmentation. In: 15th IEEE international symposium on biomedical imaging (ISBI 2018), pp 1332–1335Google Scholar
  45. 45.
    Wolterink JM, Leiner T, Viergever MA, Išgum I (2016) Dilated convolutional neural networks for cardiovascular mr segmentation in congenital heart disease. In: Reconstruction, segmentation, and analysis of medical images, pp 95–102. SpringerGoogle Scholar
  46. 46.
    Wolterink JM, Leiner T, Viergever MA, Isgum I (2017) Automatic segmentation and disease classification using cardiac cine mr images. arXiv:1708.01141
  47. 47.
    Xu J, Schwing AG, Urtasun R (2014) Tell me what you see and i will show you where it is. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3190–3197Google Scholar
  48. 48.
    Xue Y, Xu T, Zhang H, Long LR, Huang X (2017) Segan: Adversarial network with multi-scalel1 loss for medical image segmentation. arXiv:1706.01805
  49. 49.
    Yu L, Yang X, Qin J, Heng PA (2016) 3d fractalnet: dense volumetric segmentation for cardiovascular mri volumes. In: Reconstruction, segmentation, and analysis of medical images, pp 103–110. SpringerGoogle Scholar
  50. 50.
    Yu X, Liu T, Gong M, Tao D (2018) Learning with biased complementary labels. In: The european conference on computer vision (ECCV)Google Scholar
  51. 51.
    Zhang YD, Muhammad K, Tang C (2018) Twelve-layer deep convolutional neural network with stochastic pooling for tea category classification on gpu platform. Multimedia Tools and Applications pp 1–19Google Scholar
  52. 52.
    Zhang YD, Zhao G, Sun J, Wu X, Wang ZH, Liu HM, Govindaraj VV, Zhan T, Li J (2017) Smart pathological brain detection by synthetic minority oversampling technique, extreme learning machine, and jaya algorithm. Multimedia Tools and Applications pp 1–20Google Scholar
  53. 53.
    Zhou Y, Berg TL (2016) Learning temporal transformations from time-lapse videos. In: European conference on computer vision, pp 262–277Google Scholar
  54. 54.
    Zhu J Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: The IEEE international conference on computer vision (ICCV)Google Scholar
  55. 55.
    Zhu W, Xie X (2016) Adversarial deep structural networks for mammographic mass segmentation. arXiv:1612.05970
  56. 56.
    Zotti C, Luo Z, Humbert O, Lalande A, Jodoin PM (2017) Gridnet with automatic shape prior registration for automatic mri cardiac segmentation. arXiv:1705.08943

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Hasso Plattner InstitutePotsdamGermany

Personalised recommendations