Image synthesis-based multi-modal image registration framework by using deep fully convolutional networks
- 60 Downloads
Multi-modal image registration has significant meanings in clinical diagnosis, treatment planning, and image-guided surgery. Since different modalities exhibit different characteristics, finding a fast and accurate correspondence between images of different modalities is still a challenge. In this paper, we propose an image synthesis-based multi-modal registration framework. Image synthesis is performed by a ten-layer fully convolutional network (FCN). The network is composed of 10 convolutional layers combined with batch normalization (BN) and rectified linear unit (ReLU), which can be trained to learn an end-to-end mapping from one modality to the other. After the cross-modality image synthesis, multi-modal registration can be transformed into mono-modal registration. The mono-modal registration can be solved by methods with lower computational complexity, such as sum of squared differences (SSD). We tested our method in T1-weighted vs T2-weighted, T1-weighted vs PD, and T2-weighted vs PD image registrations with BrainWeb phantom data and IXI real patients’ data. The result shows that our framework can achieve higher registration accuracy than the state-of-the-art multi-modal image registration methods, such as local mutual information (LMI) and α-mutual information (α-MI). The average registration errors of our method in experiment with IXI real patients’ data were 1.19, 2.23, and 1.57 compared to 1.53, 2.60, and 2.36 of LMI and 1.34, 2.39, and 1.76 of α-MI in T2-weighted vs PD, T1-weighted vs PD, and T1-weighted vs T2-weighted image registration, respectively. In this paper, we propose an image synthesis-based multi-modal image registration framework. A deep FCN model is developed to perform image synthesis for this framework, which can capture the complex nonlinear relationship between different modalities and discover complex structural representations automatically by a large number of trainable mapping and parameters and perform accurate image synthesis. The framework combined with the deep FCN model and mono-modal registration methods (SSD) can achieve fast and robust results in multi-modal medical image registration.
KeywordsMulti-modal registration Image synthesis Convolutional neural network
Xueli Liu and Dongsheng Jiang developed the algorithm, performed the experiments, analyzed the data, and drafted the manuscript. Manning Wang and Zhijian Song provided suggestions and helped to draft the manuscript. All authors have read and approved the final manuscript.
This study has been supported by the National Key Research and Development Program of China (2017YFC0110700) and the National Natural Science Foundation of China (grants 81471758 and 81701795). This research has also been partially supported by the Program of Shanghai Academic/Technology Research Leaders (16XD1424900).
Compliance with ethical standards
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
- 3.Hata N, Dohi T, Warfield SK, Kikinis R, Jolesz FA (1998) Multimodality deformable registration of pre- and intraoperative images for MRI-guided brain surgery. International Conference on Medical Image Computing and Computer-Assisted Intervention pp 1067–74Google Scholar
- 7.Yang J, Li H, Jia Y (2013) Go-ICP: solving 3D registration efficiently and globally optimally. IEEE International Conference on Computer Vision pp 1457–64Google Scholar
- 12.Heinrich MP, Jenkinson M, Papiez BW, Brady SM, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. International Conference on Medical Image Computing & Computer-assisted Intervention pp 187Google Scholar
- 13.Oktay O, Schuh A, Rajchl M, Keraudren K, Gómez A, Heinrich MP, Penney G, Rueckert D (2015) Structured decision forests for multi-modal ultrasound image registration. In: Medical Image Computing and Computer assisted Intervention – MICCAI 2015. Springer International Publishing, Berlin, pp 363–371Google Scholar
- 25.Roy S, Carass A, Jog A, Prince JL, Lee J (2014) MR to CT registration of brains using image synthesis. Proc SPIE Int Soc Opt Eng 9034:255–275Google Scholar
- 27.Chen M, Jog A, Carass A, Prince JL (2015) Using image synthesis for multi-channel registration of different image modalities. Proc SPIE Int Soc Opt Eng 9413:1Google Scholar
- 28.Min C, Carass A, Jog A, Lee J, Roy S, Prince JL (2016) Cross contrast multi-channel image registration using image synthesis for MR brain images. Med Image Anal 36:2Google Scholar
- 29.Cao X, Gao Y, Yang J, Wu G, Shen D (2016) Learning-based multimodal image registration for prostate cancer radiation therapy. International Conference on Medical Image Computing 9902:1Google Scholar
- 30.Nguyen HV, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. International Conference on Medical Image Computing and Computer-Assisted Intervention pp 677–684Google Scholar
- 32.Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. International Conference on Neural Information Processing Systems pp 1097–105Google Scholar
- 33.Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Computer science arXiv:1409-1556v6Google Scholar
- 34.Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines International Conference on International Conference on Machine Learning pp 807–14Google Scholar
- 35.Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Computer Science arXiv:1502-03167v3Google Scholar
- 37.Klein S, Staring M, Pluim JPW (2007) Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines. IEEE Trans Image Process 16:2879Google Scholar