Adversarial Image Registration with Application for MR and TRUS Image Fusion
Robust and accurate alignment of multimodal medical images is a very challenging task, which however is very useful for many clinical applications. For example, magnetic resonance (MR) and transrectal ultrasound (TRUS) image registration is a critical component in MR-TRUS fusion guided prostate interventions. However, due to the huge difference between the image appearances and the large variation in image correspondence, MR-TRUS image registration is a very challenging problem. In this paper, an adversarial image registration (AIR) framework is proposed. By training two deep neural networks simultaneously, one being a generator and the other being a discriminator, we can obtain not only a network for image registration, but also a metric network which can help evaluate the quality of image registration. The developed AIR-net is then evaluated using clinical datasets acquired through image-fusion guided prostate biopsy procedures and promising results are demonstrated.
The authors would like to thank NVIDIA Corporation for the donation of the Titan Xp GPU used for this research.
- 1.Siddiqui, M.M., et al.: Comparison of MR/ultrasound fusion-guided biopsy with ultrasound-guided biopsy for the diagnosis of prostate cancer. JAMA 313(4), 390–397 (2015)Google Scholar
- 2.Cao, X., Gao, Y., Yang, J., Wu, G., Shen, D.: Learning-based multimodal image registration for prostate cancer radiation therapy. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 1–9. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46726-9_1CrossRefGoogle Scholar
- 3.Cheng, X., Zhang, L., Zheng, Y.: Deep similarity learning for multimodal medical images. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 1–5 (2016)Google Scholar
- 4.Simonovsky, M., Gutiérrez-Becker, B., Mateus, D., Navab, N., Komodakis, N.: A deep metric for multimodal registration. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 10–18. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46726-9_2CrossRefGoogle Scholar
- 6.de Vos, B.D., Berendsen, F.F., Viergever, M.A., Staring, M., Igum, I.: End-to-end unsupervised deformable image registration with a convolutional neural network. arXiv:1704.06065 [cs] (2017)
- 7.Hu, Y., et al.: Label-driven weakly-supervised learning for multimodal deformable image registration. arXiv:1711.01666 [cs] (2017)
- 8.Cao, X., Yang, J., Wang, L., Xue, Z., Wang, Q., Shen, D.: Deep learning based inter-modality image registration supervised by intra-modality similarity. arXiv:1804.10735 [cs] (2018)
- 9.Goodfellow, I.J., et al.: Generative adversarial networks. arXiv:1406.2661 [cs, stat] (2014)
- 10.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778 (2016)Google Scholar
- 11.Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. CoRR abs/ arXiv:1504.03641 (2015)
- 12.Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv:1701.07875 [cs, stat] (2017)
- 13.Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS 2017 Workshop Autodiff (2017)Google Scholar
- 14.Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. arXiv:1506.02025 [cs] (2015)