Interactive 3D Segmentation Editing and Refinement via Gated Graph Neural Networks

  • Xiaosong WangEmail author
  • Ling Zhang
  • Holger Roth
  • Daguang Xu
  • Ziyue Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11849)


The extraction of organ and lesion regions is an important yet challenging problem in medical image analysis. The accuracy of the segmentation is essential to the quantitative evaluation in many clinical applications. Nevertheless, automated segmentation approaches often suffer from a variety of errors, e.g., over-segmentation, under-detection, and dull edges, which often requires manual corrections on the algorithm-generated results. Therefore, an efficient segmentation editing and refinement tool is desired due to the need of (1) minimizing the repeated effort of human annotators on similar errors (e.g., under-segmentation cross several slices in 3D volumes); (2) an “intelligent” algorithm that can preserve the correct part of the segmentation while it can also align the erroneous part with the true boundary based on users’ limited input. This paper presents a novel solution that utilizes the gated graph neural networks to refine the 3D image volume segmentation from certain automated methods in an interactive mode. The pre-computed segmentation is converted to polygons in a slice-by-slice manner, and then we construct the graph by defining polygon vertices cross slices as nodes in a directed graph. The nodes are modeled with gated recurrent units to first propagate the features among neighboring nodes. Afterward, our framework outputs the movement prediction of each polygon vertex based on the converged states of nodes. We quantitatively demonstrate the refinement performance of our framework on the artificially degraded segmentation data. Up to \(10\%\) improvement in IOUs are achieved for the segmentation with a variety of error degrees and percentages.


Interactive Segmentation refinement Gated graph neural network 


  1. 1.
    Acuna, D., Ling, H., Kar, A., Fidler, S.: Efficient interactive annotation of segmentation datasets with polygon-RNN++. In: Proceedings of the IEEE Conference on CVPR, pp. 859–868 (2018)Google Scholar
  2. 2.
    Cho, K., Van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches (2014). arXiv:1409.1259
  3. 3.
    Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks (2015). arXiv preprint arXiv:1511.05493
  4. 4.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on CVPR, pp. 3431–3440 (2015)Google Scholar
  5. 5.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  6. 6.
    Simpson, A.L., et al.: A large annotated medical image dataset for the development and evaluation of segmentation algorithms (2019). arXiv preprint arXiv:1902.09063
  7. 7.
    Wang, G., et al.: Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans. Med. Imag. 37(7), 1562–1573 (2018)CrossRefGoogle Scholar
  8. 8.
    Zhao, F., Xie, X.: An overview of interactive medical image segmentation. Ann. BMVA 2013(7), 1–22 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Xiaosong Wang
    • 1
    Email author
  • Ling Zhang
    • 1
  • Holger Roth
    • 1
  • Daguang Xu
    • 1
  • Ziyue Xu
    • 1
  1. 1.Nvidia CorporationBethesdaUSA

Personalised recommendations