A Brief Survey on Multi Modalities Fusion

  • M. SumithraEmail author
  • S. Malathi
Conference paper
Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT, volume 35)


Medical images are taken by using different modalities like Magnetic resonance imaging (MRI), positron emission tomography (PET) image, computed tomography (CT), X-ray and Ultrasound. Each and every modalities has its own pros and cons. In now a days there are many modality images are fused and getting very good resultant image. This resultant image will give very good analysis about the disease. We can easily find out the disease portion with its exact circumference. Magnetic resonance imaging (MRI) and positron emission tomography (PET) image fusion is a late half breed methodology utilized in a few oncology applications. The MRI image demonstrates the brain tissue life structures and does not contain any useful data, while the PET image the mind work and has a low spatial goals. An ideal MRI–PET combination technique safeguards the practical data of the PET image and includes spatial attributes of the MRI image with the less conceivable spatial twisting. In this paper we discussed about different types of modalities fusing and which will get good result to analyse the disease perfectly and accurately.


MRI PET CT Ultrasound Fusion Modality 


  1. 1.
    Yin, M., Liu, X., Liu, Y., Chen, X.: Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68(1), 49–64 (2019)CrossRefGoogle Scholar
  2. 2.
    Lian, C., Ruan, S., Denœux, T., Li, H., Vera, P.: Joint tumor segmentation in PET-CT images using co-clustering and fusion based on belief functions. IEEE Trans. Image Process. 28(2), 755–766 (2019)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bernal, E.A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., Bala, R.: Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors. IEEE Trans. Multimed. 20(1), 107–118 (2018)CrossRefGoogle Scholar
  4. 4.
    Zhang, J., Xu, T., Chen, S., Wang, X.: Efficient colorful Fourier ptychographic microscopy reconstruction with wavelet fusion. In: IEEE Access, date of publication 29 May 2018, date of current version 29 June 2018Google Scholar
  5. 5.
    Shi, C., Luo, X., Guo, J., Najdovski, Z., Fukuda, T., Ren, H.: Three-dimensional intravascular reconstruction techniques based on intravascular ultrasound: a technical review. IEEE J. Biomed. Health Inform. 22(3), 806–817 (2018)CrossRefGoogle Scholar
  6. 6.
    El-Hariri, H., Pandey, P., Hodgson, A.J., Garbi, R.: Augmented reality visualisation for orthopaedic surgical guidance with pre- and intra-operative multimodal image data fusion. IEEE Trans. Healthc. Technol. Lett. 5(5), 189–193 (2018)CrossRefGoogle Scholar
  7. 7.
    Gibson, E., Giganti, F., Hu, Y., Bonmati, E., Bandula, S., Gurusamy, K., Davidson, B., Pereira, S.P., Clarkson, M.J., Barratt, D.C.: Automatic multi-organ segmentation on abdominal CT with dense V-Networks. IEEE Trans. Med. Imaging 37(8), 1822–1834 (2018)CrossRefGoogle Scholar
  8. 8.
    Huang, C.-C., Nguyen, M.-H.: X-Ray enhancement based on component attenuation, contrast adjustment and image fusion. IEEE Trans. Image Process. 28(1), 127–141 (2019)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Hazarika, A., Sarmah, A., Borah, R., Boro, M., Dutta, L., Kalita, P., kumar dev Choudhury, B.: Discriminant feature level fusion based learning for automatic staging of EEG signals. Healthcare Technol. Lett. 5(6), 226–230 (2018)Google Scholar
  10. 10.
    Hu, W., Lin, D., Cao, S., Liu, J., Chen, J., Calhoun, V.D., Wang, Y.P.: Adaptive sparse multiple canonical correlation analysis with application to imaging(epi)genomics study of schizophrenia. IEEE Trans. Biomed. Eng. 65(2), 390–399 (2018)Google Scholar
  11. 11.
    Huang, C., Xie, Y., Lan, Y., Hao, Y., Chen, F., Cheng, Y., Peng, Y.: A new framework for the integrative analytics of intravascular ultrasound and optical coherence tomography images. IEEE Transl. Content Min. 6, 2169–3536 (2018)Google Scholar
  12. 12.
    Chartsias, A., Joyce, T., Giuffrida, M.V., Tsaftaris, S.A.: Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans. Med. Imaging 37(3), 803–814 (2018)CrossRefGoogle Scholar
  13. 13.
    Queirós, S., Morais, P., Barbosa, D., Fonseca, J.C., Vilaça, J.L., D’hooge, J.: MITT: medical image tracking toolbox. IEEE Trans. Med. Imaging 37(11), 2547–2557 (2018)CrossRefGoogle Scholar
  14. 14.
    Fischer, P., Faranesh, A., Pohl, T., Maier, A., Rogers, T., Ratnayaka, K., Lederman, R., Hornegger, J.: An MR-based model for cardio-respiratory motion compensation of overlays in X-Ray fluoroscopy. IEEE Trans. Med. Imaging 37(1), 47–60 (2018)CrossRefGoogle Scholar
  15. 15.
    Xin, Y., Kong, L., Liu, Z., Wang, C., Zhu, H., Gao, M., Zhao, C., Xu, X.: Multimodal feature-level fusion for biometrics identification system on IoMT platform. IEEE Transl. Content Min. 6, 21418–21426 (2018)Google Scholar
  16. 16.
    Cao, X., Yang, J., Gao, Y., Wang, Q., Shen, D.: Region-adaptive deformable registration of CT/MRI pelvic images via learning-based image synthesis. IEEE Trans. Image Process. 27(7), 3500–3512 (2018)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Qi, S., Calhoun, V.D., van Erp, T.G., Bustillo, J., Damaraju, E., Turner, J.A., Du, Y., Yang, J., Chen, J., Yu, Q., Mathalon, D.H., Ford, J.M., Voyvodic, J., Mueller, B.A., Belger, A., McEwen, S., Potkin, S.G., Preda, A., Jiang, T., Sui, J.: Multimodal fusion with reference: searching for joint neuromarkers of working memory deficits in schizophrenia. IEEE Trans. Med. Imaging 37(1), 93–105 (2018)CrossRefGoogle Scholar
  18. 18.
    Kraguljac, N.V., Srivastava, A., Lahti, A.C.: Memory deficits in schizophrenia: a selective review of functional magnetic resonance imaging (FMRI) studies. Behav. Sci. 3(3), 330–347 (2013)CrossRefGoogle Scholar
  19. 19.
    Lett, T.A., Voineskos, A.N., Kennedy, J.L., Levine, B., Daskalakis, Z.J.: Treating working memory deficits in schizophrenia: A review of the neurobiology. Biol. Psychiatry 75(5), 361–370 (2014)CrossRefGoogle Scholar
  20. 20.
    Calhoun, V.D., Adali, T.: Feature-based fusion of medical imaging data. IEEE Trans. Inf. Technol. Biomed. 13(5), 711–720 (2009)CrossRefGoogle Scholar
  21. 21.
    Smith, S.M., et al.: Correspondence of the brain’s functional architecture during activation and rest. Proc. Nat. Acad. Sci. USA 106(31), 13040–13045 (2009)CrossRefGoogle Scholar
  22. 22.
    Yin, H.: Tensor sparse representation for 3-D medical image fusion using weighted average rule. IEEE Trans. Biomed. Eng. 65(11), 2622–2633 (2018)CrossRefGoogle Scholar
  23. 23.
    Liu, X., Mei, W., Du, H.: Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion. Neurocomputing 235, 131–139 (2017)CrossRefGoogle Scholar
  24. 24.
    Yang, B., Li, S.: Pixel-level image fusion with simultaneous orthogonal matching pursuit. Inf. Fusion 13(1), 10–19 (2012)CrossRefGoogle Scholar
  25. 25.
    Hait, E., Gilboa, G.: Spectral total-variation local scale signatures for image manipulation and fusion. IEEE Trans. Image Process. 28(2), 880–895 (2019)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Brox, T., Weickert, J.: A TV flow based local scale measure for texture discrimination. In: Proceedings of European Conference on Computer Vision, pp. 578–590. Springer, New York (2004)Google Scholar
  27. 27.
    Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-8(6), 679–698 (1986)CrossRefGoogle Scholar
  28. 28.
    Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)CrossRefGoogle Scholar
  29. 29.
    Arbeláez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)CrossRefGoogle Scholar
  30. 30.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5), 530–549 (2004)CrossRefGoogle Scholar
  31. 31.
    Dollár, P., Zitnick, C.L.: Fast edge detection using structured forests. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1558–1570 (2015)CrossRefGoogle Scholar
  32. 32.
    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)CrossRefGoogle Scholar
  33. 33.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision Pattern Recognition, vol. 1, no. 1, pp. 886–893, June 2005Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Sathyabama Institute of Science and TechnologyChennaiIndia
  2. 2.Panimalar Engineering CollegeChennaiIndia

Personalised recommendations