Adversarial Learning for Deformable Image Registration: Application to 3D Ultrasound Image Fusion

  • Zisheng LiEmail author
  • Masahiro Ogino
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11798)


We present an adversarial learning algorithm for deep-learning-based deformable image registration (DIR) and apply to 3D liver ultrasound image fusion. We consider DIR as a parametric optimization model that aims to find displacement field of deformation. We propose an adversarial learning framework inspired by generative adversarial network (GAN) to predict the displacement field without ground-truth spatial transformation. We use convolutional neural network (CNN) and a spatial transform layer as registration network to generate the registered image. Similarity metrics of image intensity and vessel masks are used as loss function for the training. We also optimize a discrimination network to measure the divergence between the registered image and the fixed image. Feedback from the discrimination network can guide the registration network for more accurate and realistic deformation. Moreover, we incorporate an autoencoder network to extract anatomical features from vessel masks as shape regularization. Our approach is end-to-end, only requires image pair as input in registration tasks. Experiments show that the proposed method outperforms state-of-the-art deep-learning-based methods.


GAN Deformable image registration Deep learning 


  1. 1.
    Roche, A., Pennec, X., Malandain, G., Ayache, N.: Rigid registration of 3-D ultrasound with MR images: a new approach combining intensity and gradient information. IEEE Trans. Med. Images 20(10), 1038–1049 (2001)CrossRefGoogle Scholar
  2. 2.
    Penney, G.P., Blackall, J.M., Hamady, M.S., Sabharwal, T.: Registration of freehand 3D ultrasound and magnetic resonance liver images. Med. Image Anal. 8, 81–91 (2004)CrossRefGoogle Scholar
  3. 3.
    Wein, W., Brunke, S., et al.: Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med. Image Anal. 12, 577–585 (2008)CrossRefGoogle Scholar
  4. 4.
    Wein, W., Ladikos, A., Fuerst, B., Shah, A., Sharma, K., Navab, N.: Global registration of ultrasound to MRI using the LC2metric for enabling neurosurgical guidance. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 34–41. Springer, Heidelberg (2013). Scholar
  5. 5.
    Lange, T., Papenberg, N., et al.: 3D ultrasound-CT registration of the liver using combined landmark-intensity information. Int. J. CARS 4, 79–88 (2009)CrossRefGoogle Scholar
  6. 6.
    Krebs, J., et al.: Robust non-rigid registration through agent-based action learning. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.Louis, Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 344–352. Springer, Cham (2017). Scholar
  7. 7.
    Rohé, M.-M., Datar, M., Heimann, T., Sermesant, M., Pennec, X.: SVF-Net: learning deformable image registration using shape matching. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 266–274. Springer, Cham (2017). Scholar
  8. 8.
    Sokooti, H., de Vos, B., Berendsen, F., Lelieveldt, B.P.F., Išgum, I., Staring, M.: Nonrigid image registration using multi-scale 3D convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 232–239. Springer, Cham (2017). Scholar
  9. 9.
    Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: fast predictive image registration–a deep learning approach. NeuroImage 158, 378–396 (2017)CrossRefGoogle Scholar
  10. 10.
    de Vos, B.D., Berendsen, F., Viergever, M.A.: End-to-end unsupervised deformable image registration with a convolutional neural network. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 204–212 (2017)Google Scholar
  11. 11.
    Li, H., Fan, Y.: Non-rigid image registration using fully convolutional networks with deep self-supervision. arXiv preprint arXiv:1709.00799 (2017)
  12. 12.
    Jaderberg, M., Simonyan, K., Zisserman, A.: Spatial transformer networks. In: NIPS 2015, pp. 2017–2025 (2015)Google Scholar
  13. 13.
    Goodfellow, I., et al.: Generative adversarial nets. In: NIPS 2014, pp. 2672–2680 (2014)Google Scholar
  14. 14.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV 2017, pp. 2223–2232 (2017)Google Scholar
  15. 15.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  16. 16.
    Oktay, O., Ferrante, E., Kamnitsas, K.: Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Trans. Med. Imaging 37(2), 384–395 (2018)CrossRefGoogle Scholar
  17. 17.
    Balakrishnan, G., Zhao, A., Sabuncu, M.R.: An unsupervised learning model for deformable medical image registration. In: CVPR 2018, pp. 9252–9260 (2018)Google Scholar
  18. 18.
    Hu, Y., Modat, M., Gibson, E., Ghavami, N.: Label-driven weakly-supervised learning for multimodal deformable image registration. In: ISBI 2018, pp. 1070–1074. IEEE (2018)Google Scholar
  19. 19.
    Fan, J., Cao, X., Xue, Z., Yap, P.-T., Shen, D.: Adversarial similarity network for evaluating image alignment in deep learning based registration. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 739–746. Springer, Cham (2018). Scholar
  20. 20.
    Hu, Y., et al.: Adversarial deformation regularization for training image registration neural networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 774–782. Springer, Cham (2018). Scholar
  21. 21.
    Mahapatra, D., Antony, B., Sedai, S.: Deformable medical image registration using generative adversarial networks. In: ISBI 2018, pp. 1449–1453. IEEE (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Research & Development GroupHitachi, Ltd.TokyoJapan

Personalised recommendations