APNet: Semantic Segmentation for Pelvic MR Image

  • Ting-Ting Liang
  • Mengyan Sun
  • Liangcai GaoEmail author
  • Jing-Jing LuEmail author
  • Satoshi Tsutsui
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11257)


One of the time-consuming routine work for a radiologist is to discern anatomical structures from tomographic images. For assisting radiologists, this paper develops an automatic segmentation method for pelvic magnetic resonance (MR) images. The task has three major challenges (1) A pelvic organ can have various sizes and shapes depending on the axial image, which requires local contexts to segment correctly. (2) Different organs often have quite similar appearance in MR images, which requires global context to segment. (3) The number of available annotated images are very small to use the latest segmentation algorithms. To address the challenges, we propose a novel convolutional neural network called Attention-Pyramid network (APNet) that effectively exploits both local and global contexts, in addition to a data-augmentation technique that is particularly effective for MR images. In order to evaluate our method, we construct fine-grained (50 pelvic organs) MR image segmentation dataset, and experimentally confirm the superior performance of our techniques over the state-of-the-art image segmentation methods.


Medical image Semantic segmentation Convolutional neural networks Pyramid pooling Attention mechanism 


  1. 1.
    Abadi, M., Barham, P., Chen, J., Chen, Z., et al.: TensorFlow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI) (2016)Google Scholar
  2. 2.
    Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. arXiv:1511.00561 (2015)
  3. 3.
    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI (2016)Google Scholar
  4. 4.
    Chen, L.C., Yang, Y., Wang, J., Xu, W., Yuille, A.L.: Attention to scale: scale-aware semantic image segmentation. In: CVPR (2016)Google Scholar
  5. 5.
    Ciresan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. In: CVPR (2012)Google Scholar
  6. 6.
    Dowling, J.A., et al.: An atlas-based electron density mapping method for magnetic resonance imaging (MRI)-alone treatment planning and adaptive MRI-based prostate radiation therapy. Int. J. Radiat. Oncol. 83(1), e5–e11 (2011)CrossRefGoogle Scholar
  7. 7.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. TPAMI (2015)Google Scholar
  9. 9.
    Hiasa, Y., et al.: Cross-modality image synthesis from unpaired data using CycleGAN: effects of gradient consistency loss and training data size. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N. (eds.) SASHIMI 2018. LNCS, vol. 11037, pp. 31–41. Springer, Cham (2018). Scholar
  10. 10.
    Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. In: CVPR (2018)Google Scholar
  11. 11.
    Kazemifar, S., et al.: Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning. arXiv:1802.09587 (2018)CrossRefGoogle Scholar
  12. 12.
    Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: AISTATS (2015)Google Scholar
  13. 13.
    Liu, W., Rabinovich, A., Berg, A.C.: ParseNet: looking wider to see better. In: ICLR (2016)Google Scholar
  14. 14.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  15. 15.
    Ma, Z., Jorge, R.N.M., Mascarenhas, T., Tavares, J.M.R.S.: Segmentation of female pelvic cavity in axial T2-weighted MR images towards the 3D reconstruction. Int. J. Numer. Method Biomed. Eng. 28(6–7), 714–726 (2012). Scholar
  16. 16.
    Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. TPAMI (2010)Google Scholar
  17. 17.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  18. 18.
    Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. ACM Trans. Graph. (TOG) 25(3), 533–540 (2006). Proceedings of ACM SIGGRAPHCrossRefGoogle Scholar
  19. 19.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: ICML (2015)Google Scholar
  20. 20.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 (2015)
  21. 21.
    Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In: CVPR (2017)Google Scholar
  22. 22.
    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017)Google Scholar
  23. 23.
    Zhu, J.Y., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: CVPR (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.ICSTPeking UniversityBeijingChina
  2. 2.Peking Union Medical College HospitalBeijingChina
  3. 3.Indiana University BloomingtonBloomingtonUSA

Personalised recommendations