Cross-Modality Knowledge Transfer for Prostate Segmentation from CT Scans

  • Yucheng LiuEmail author
  • Naji Khosravan
  • Yulin Liu
  • Joseph Stember
  • Jonathan Shoag
  • Ulas Bagci
  • Sachin Jambawalikar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11795)


Creating large scale high-quality annotations is a known challenge in medical imaging. In this work, based on the CycleGAN algorithm, we propose leveraging annotations from one modality to be useful in other modalities. More specifically, the proposed algorithm creates highly realistic synthetic CT images (SynCT) from prostate MR images using unpaired data sets. By using SynCT images (without segmentation labels) and MR images (with segmentation labels available), we have trained a deep segmentation network for precise delineation of prostate from real CT scans. For the generator in our CycleGAN, the cycle consistency term is used to guarantee that SynCT shares the identical manually-drawn, high-quality masks originally delineated on MR images. Further, we introduce a cost function based on structural similarity index (SSIM) to improve the anatomical similarity between real and synthetic images. For segmentation followed by the SynCT generation from CycleGAN, automatic delineation is achieved through a 2.5D Residual U-Net. Quantitative evaluation demonstrates comparable segmentation results between our SynCT and radiologist drawn masks for real CT images, solving an important problem in medical image segmentation field when ground truth annotations are not available for the modality of interest.


Domain adaptation Deep learning CT synthesis Prostate segmentation 2.5D Generative Adversarial Networks 


  1. 1.
    Nordstrm, T., et al.: Prostate-specific antigen (PSA) density in the diagnostic algorithm of prostate cancer. Prostate Cancer Prostatic Dis. 21(1), 57–63 (2017)CrossRefGoogle Scholar
  2. 2.
    Smith, W.L., et al.: Prostate volume contouring: a 3D analysis of segmentation using 3DTRUS, CT, and MR. Int. J. Radiat. Oncol. Biol. Phys. 67(4), 1238–1247 (2007)CrossRefGoogle Scholar
  3. 3.
    Rasch, C., et al.: Definition of the prostate in CT and MRI: a multi-observer study. Int. J. Radiat. Oncol. Biol. Phys. 43(1), 57–66 (1999)CrossRefGoogle Scholar
  4. 4.
    Chowdhury, N., et al.: Concurrent segmentation of the prostate on MRI and CT via linked statistical shape models for radiotherapy planning. Med. Phys. 39(4), 2214–2228 (2012)CrossRefGoogle Scholar
  5. 5.
    Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv:1703.10593 (2017)
  6. 6.
    Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 14–23. Springer, Cham (2017). Scholar
  7. 7.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). arXiv:1505.04597CrossRefGoogle Scholar
  8. 8.
    Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 179–187. Springer, Cham (2016). Scholar
  9. 9.
    Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N., Huisman, H.: SPIE-AAPM PROSTATEx Challenge Data (2017).
  10. 10.
    Bloch, N., et al.: NCI-ISBI 2013 Challenge: Automated Segmentation of Prostate Structures. The Cancer Imaging Archive (2015).
  11. 11.
    Zhao, H., et al.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)CrossRefGoogle Scholar
  12. 12.
    Liu, C., et al.: Automatic segmentation of the prostate on CT images using deep neural networks (DNN). Int. J. Radiat. Oncol. Biol. Phys. 104(4), 924–932 (2019)CrossRefGoogle Scholar
  13. 13.
    Burgos, N., et al.: Iterative framework for the joint segmentation and CT synthesis of MR images: application to MRI-only radiotherapy treatment planning. Phys. Med. Biol. 62, 4237–4253 (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yucheng Liu
    • 1
    Email author
  • Naji Khosravan
    • 2
  • Yulin Liu
    • 1
    • 3
  • Joseph Stember
    • 1
  • Jonathan Shoag
    • 4
  • Ulas Bagci
    • 2
  • Sachin Jambawalikar
    • 1
  1. 1.Department of RadiologyColumbia University Irving Medical CenterNew YorkUSA
  2. 2.Center for Research in Computer VisionUniversity of Central FloridaOrlandoUSA
  3. 3.Department of Information and Computer EngineeringChung Yuan Christian UniversityTaoyuan CityTaiwan
  4. 4.Urologic OncologyWeill Cornell MedicineNew YorkUSA

Personalised recommendations