Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks

  • Xueli Liu
  • Dongsheng Jiang
  • Manning WangEmail author
  • Zhijian SongEmail author
Original Article


Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients’ data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients’ data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration.

Graphical abstract

The workflow of proposed multi-modal image registration framework


Multi-modal registration Image synthesis Convolutional neural network 


Authors’ contributions

Xueli Liu and Dongsheng Jiang developed the algorithm, performed the experiments, analyzed the data, and drafted the manuscript. Manning Wang and Zhijian Song provided suggestions and helped to draft the manuscript. All authors have read and approved the final manuscript.


This study has been supported by the National Key Research and Development Program of China (2017YFC0110700) and the National Natural Science Foundation of China (grants 81471758 and 81701795). This research has also been partially supported by the Program of Shanghai Academic/Technology Research Leaders (16XD1424900).

Compliance with ethical standards

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.


  1. 1.
    Guo Y, Bennamoun M, Sohel F, Lu M, Wan J, Kwok NM (2016) A comprehensive performance evaluation of 3D local feature descriptors. Int J Comput Vis 116:66–89CrossRefGoogle Scholar
  2. 2.
    Woo J, Slomka PJ, Dey D, Cheng VY, Hong BW, Ramesh A, Berman DS, Karlsberg RP, Kuo J, Germano G (2009) Geometric feature-based multimodal image registration of contrast-enhanced cardiac CT with gated myocardial perfusion SPECT. Med Phys 36:5467–5479CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Hata N, Dohi T, Warfield SK, Kikinis R, Jolesz FA (1998) Multimodality deformable registration of pre- and intraoperative images for MRI-guided brain surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention pp 1067–74Google Scholar
  4. 4.
    Zitová B, Flusser J (2003) Image registration methods: a survey. Image Vis Comput 21:977–1000CrossRefGoogle Scholar
  5. 5.
    Rueckert D, Aljabar P (2010) Nonrigid registration of medical images: theory, methods, and applications [applications corner]. IEEE Signal Process Mag 27:113–119CrossRefGoogle Scholar
  6. 6.
    Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32:1153–1190CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Yang J, Li H, Jia Y (2013) Go-ICP: solving 3D registration efficiently and globally optimally. IEEE International Conference on Computer Vision pp 1457–64Google Scholar
  8. 8.
    Liu Y, Foteinos P, Chernikov A, Chrisochoides N (2012) Mesh deformation-based multi-tissue mesh generation for brain images. Eng Comput-Germany 28:305–318CrossRefGoogle Scholar
  9. 9.
    Shen D, Davatzikos C (2002) HAMMER: hierarchical attribute matching mechanism for elastic registration. IEEE Trans Med Imaging 21:1421–1439CrossRefPubMedGoogle Scholar
  10. 10.
    Wachinger C, Navab N (2012) Entropy and Laplacian images: structural representations for multi-modal registration. Med Image Anal 16:1–17CrossRefPubMedGoogle Scholar
  11. 11.
    Heinrich MP, Jenkinson M, Bhushan M, Matin T, Gleeson FV, Brady SM, Schnabel JA (2012) MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. Med Image Anal 16:1423–1435CrossRefPubMedGoogle Scholar
  12. 12.
    Heinrich MP, Jenkinson M, Papiez BW, Brady SM, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. International Conference on Medical Image Computing & Computer-assisted Intervention pp 187Google Scholar
  13. 13.
    Oktay O, Schuh A, Rajchl M, Keraudren K, Gómez A, Heinrich MP, Penney G, Rueckert D (2015) Structured decision forests for multi-modal ultrasound image registration. In: Medical Image Computing and Computer assisted Intervention – MICCAI 2015. Springer International Publishing, Berlin, pp 363–371Google Scholar
  14. 14.
    Jiang D, Shi Y, Yao D, Wang M, Song Z (2016) miLBP: a robust and fast modality-independent 3D LBP for multimodal deformable registration. Int J Comput Assist Radiol Surg 11:997–1005CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Jiang D, Shi Y, Chen X, Wang M, Song Z (2017) Fast and robust multimodal image registration using a local derivative pattern. Med Phys 44:497–509CrossRefGoogle Scholar
  16. 16.
    Klein S, Van-Der-Heide U, Lips I, Van-Vulpen M, Staring M, Pluim J (2008) Automatic segmentation of the prostate in 3D MR images by atlas matching using localized mutual information. Med Phys 35:1407–1417CrossRefGoogle Scholar
  17. 17.
    Staring M, Ua VDH, Klein S, Viergever MA, Pluim JP (2009) Registration of cervical MRI using multifeature mutual information. IEEE Trans Med Imaging 28:1412–1421CrossRefGoogle Scholar
  18. 18.
    Rivaz H, Collins DL (2014) Self-similarity weighted mutual information: a new nonrigid image registration metric. Med Image Anal 18(2):343–358CrossRefGoogle Scholar
  19. 19.
    Loeckx D, Slagmolen P, Maes F, Vandermeulen D, Suetens P (2009) Nonrigid image registration using conditional mutual information. Inf Process Med Imaging 29:19–29CrossRefGoogle Scholar
  20. 20.
    Woo J, Stone M, Prince JL (2015) Multimodal registration via mutual information incorporating geometric and spatial context. IEEE Trans Image Process 24:757CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Luan H, Qi F, Xue Z, Chen L, Shen D (2008) Multimodality image registration by maximization of quantitative-qualitative measure of mutual information. Pattern Recogn 41:285–298CrossRefGoogle Scholar
  22. 22.
    Rivaz H, Karimaghaloo Z, Fonov VS, Collins DL (2014) Nonrigid registration of ultrasound and MRI using contextual conditioned mutual information. IEEE Trans Med Imaging 33:708–725CrossRefPubMedGoogle Scholar
  23. 23.
    Wein W, Brunke S, Khamene A, Callstrom MR, Navab N (2008) Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med Image Anal 12:577–585CrossRefPubMedGoogle Scholar
  24. 24.
    Fuerst B, Wein W, Müller M, Navab N (2014) Automatic ultrasound-MRI registration for neurosurgery using the 2D and 3D LC(2) metric. Med Image Anal 18:1312–1319CrossRefPubMedGoogle Scholar
  25. 25.
    Roy S, Carass A, Jog A, Prince JL, Lee J (2014) MR to CT registration of brains using image synthesis. Proc SPIE Int Soc Opt Eng 9034:255–275Google Scholar
  26. 26.
    Huynh T, Gao Y, Kang J, Li W, Pei Z, Lian J, Shen D (2015) Estimating CT image from MRI data using structured random forest and auto-context model. IEEE Trans Med Imaging 35:174–183CrossRefPubMedPubMedCentralGoogle Scholar
  27. 27.
    Chen M, Jog A, Carass A, Prince JL (2015) Using image synthesis for multi-channel registration of different image modalities. Proc SPIE Int Soc Opt Eng 9413:1Google Scholar
  28. 28.
    Min C, Carass A, Jog A, Lee J, Roy S, Prince JL (2016) Cross contrast multi-channel image registration using image synthesis for MR brain images. Med Image Anal 36:2Google Scholar
  29. 29.
    Cao X, Gao Y, Yang J, Wu G, Shen D (2016) Learning-based multimodal image registration for prostate cancer radiation therapy. International Conference on Medical Image Computing 9902:1Google Scholar
  30. 30.
    Nguyen HV, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. International Conference on Medical Image Computing and Computer-Assisted Intervention pp 677–684Google Scholar
  31. 31.
    Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 39:640–651CrossRefPubMedGoogle Scholar
  32. 32.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. International Conference on Neural Information Processing Systems pp 1097–105Google Scholar
  33. 33.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Computer science arXiv:1409-1556v6Google Scholar
  34. 34.
    Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines International Conference on International Conference on Machine Learning pp 807–14Google Scholar
  35. 35.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Computer Science arXiv:1502-03167v3Google Scholar
  36. 36.
    Kybic J, Unser M (2003) Fast parametric elastic image registration. IEEE Trans Image Process 12:1427–1442CrossRefPubMedGoogle Scholar
  37. 37.
    Klein S, Staring M, Pluim JPW (2007) Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines. IEEE Trans Image Process 16:2879Google Scholar

Copyright information

© International Federation for Medical and Biological Engineering 2018

Authors and Affiliations

  1. 1.Digital Medical Research Center, School of Basic Medical SciencesFudan UniversityShanghaiChina
  2. 2.Shanghai Key Laboratory of Medical Imaging Computing and Computer Assisted InterventionShanghaiChina

Personalised recommendations