Structural Similarity Based Anatomical and Functional Brain Imaging Fusion

  • Nishant KumarEmail author
  • Nico Hoffmann
  • Martin Oelschlägel
  • Edmund Koch
  • Matthias Kirsch
  • Stefan Gumhold
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11846)


Multimodal medical image fusion helps in combining contrasting features from two or more input imaging modalities to represent fused information in a single image. One of the pivotal clinical applications of medical image fusion is the merging of anatomical and functional modalities for fast diagnosis of malign tissues. In this paper, we present a novel end-to-end unsupervised learning based Convolutional neural network (CNN) for fusing the high and low frequency components of MRI-PET grayscale image pairs publicly available at ADNI by exploiting Structural Similarity Index (SSIM) as the loss function during training. We then apply color coding for the visualization of the fused image by quantifying the contribution of each input image in terms of the partial derivatives of the fused image. We find that our fusion and visualization approach results in better visual perception of the fused image, while also comparing favorably to previous methods when applying various quantitative assessment metrics.


Medical image fusion MRI-PET Convolutional neural networks (CNN) Structural similarity index (SSIM) 



This work was supported by the European Social Fund (project no. 100312752) and the Saxonian Ministry of Science and Art.


  1. 1.
    James, A.P., Dasarathy, B.V.: Medical image fusion: a survey of the state of the art. Inf. Fusion 19, 4–19 (2014)CrossRefGoogle Scholar
  2. 2.
    Nensa, F., Beiderwellen, K., Heusch, P., Wetter, A.: Clinical applications of PET/MRI: current status and future perspectives. Diagn. Interv. Radiol. 20(5), 438–447 (2014)CrossRefGoogle Scholar
  3. 3.
    Bhatnagar, G., Wu, Q.M.J., Liu, Z.: Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimedia 15(5), 1014–1024 (2013)CrossRefGoogle Scholar
  4. 4.
    Du, J., Li, W., Xiao, B., Nawaz, Q.: Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 194, 326–339 (2016)CrossRefGoogle Scholar
  5. 5.
    Du, J., Li, W., Xiao, B.: Anatomical-functional image fusion by information of interest in local Laplacian filtering domain. IEEE Trans. Image Process. 26(12), 5855–5866 (2017)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Yin, M., Liu, X., Liu, Y., Chen, X.: Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68(1), 49–64 (2019)CrossRefGoogle Scholar
  7. 7.
    Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)CrossRefGoogle Scholar
  8. 8.
    Li, H., He, X., Tao, D., Tang, Y., Wang, R.: Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recogn. 79, 130–146 (2018)CrossRefGoogle Scholar
  9. 9.
    Yang, B., Li, S.: Pixel-level image fusion with simultaneous orthogonal matching pursuit. Inf. Fusion 13(1), 10–19 (2012)CrossRefGoogle Scholar
  10. 10.
    Liu, Y., Liu, S., Wang, Z.: A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fusion 24, 147–164 (2015)CrossRefGoogle Scholar
  11. 11.
    Daneshvar, S., Ghassemian, H.: MRI and PET image fusion by combining IHS and retina-inspired models. Inf. Fusion 11(2), 114–123 (2010)CrossRefGoogle Scholar
  12. 12.
    Hoffmann, N., Weidner, F., Urban, P., et al.: Framework for 2D–3D image fusion of infrared thermography with preoperative MRI. Biomed. Eng./Biomedizinische Technik 62(6), 599–607 (2017)CrossRefGoogle Scholar
  13. 13.
    Liu, Y., Chen, X., Cheng, J., Peng, H.: A medical image fusion method based on convolutional neural networks. In: 20th International Conference on Information Fusion (Fusion), Xi’an, pp. 1–7 (2017).
  14. 14.
    Liu, Y., Chen, X., Wang, Z., Wang, Z.J., Ward, R.K., Wang, X.: Deep learning for pixel-level image fusion: recent advances and future prospects. Inf. Fusion 42, 158–173 (2018)CrossRefGoogle Scholar
  15. 15.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  16. 16.
    Clifford, R.J., et al.: The Alzheimer’s Disease neuroimaging initiative (ADNI). J. Magn. Reson. Imaging 27(4), 685–691 (2008)CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Wang, Q., Shen, Y.: Performances evaluation of image fusion techniques based on nonlinear correlation measurement. In: 21st IEEE Instrumentation and Measurement Technology Conference (IEEE Cat. No. 04CH37510), vol. 1, pp. 472–475 (2004)Google Scholar
  19. 19.
    Xydeas, C., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36, 308–309 (2000)CrossRefGoogle Scholar
  20. 20.
    Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: A non-reference image fusion metric based on mutual information of image features. Comput. Electr. Eng. 37(5), 744–756 (2011)CrossRefGoogle Scholar
  21. 21.
    Han, Y., Cai, Y., Cao, Y., Xu, X.: A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14(2), 127–135 (2013)CrossRefGoogle Scholar
  22. 22.
    Das, S., Kundu, M.K.: NSCT-based multimodal medical image fusion using pulse-coupled neural network and modified spatial frequency. Med. Biol. Engi. Comput. 50(10), 1105–1114 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nishant Kumar
    • 1
    Email author
  • Nico Hoffmann
    • 2
  • Martin Oelschlägel
    • 3
  • Edmund Koch
    • 3
  • Matthias Kirsch
    • 4
  • Stefan Gumhold
    • 1
  1. 1.Computer Graphics and VisualisationTechnische Universität DresdenDresdenGermany
  2. 2.Institute of Radiation PhysicsHelmholtz-Zentrum Dresden-RossendorfDresdenGermany
  3. 3.Clinical Sensoring and MonitoringTechnische Universität DresdenDresdenGermany
  4. 4.Department of NeurosurgeryUniversity Hospital Carl Gustav CarusDresdenGermany

Personalised recommendations