Multimodality image fusion, moving forward

  • Marina PiccinelliEmail author

Multimodality image fusion is an image processing procedure that allows the spatial alignment of multiple images acquired from one or more imaging devices and the visualization of the fused information within a single display or representation. The rationale behind the development and use of an image fusion strategy in a clinical environment resides in the fact that different images can synergistically contribute to a more comprehensive characterization of the disease and provide guidance for the most appropriate medical or interventional treatment.1-3 More than that, while traditionally physicians have been trained to mentally integrate different imaging tests to provide the best patient’s care, it has been widely demonstrated how a fused display can be far more informative and diagnostically powerful than a mental representation.4 Nowadays, applications for medical image fusion range from oncology and radiotherapy, to neurology, to image-guided surgical guidance, to cardiology. Over a decades-long period images from a large variety of imaging modalities have been used to develop image fusion frameworks—invasive and noninvasive, anatomical and functional—prompting the proliferation of hybrid devices, such as PET/CT and PET/MR, SPECT/CT, CT/MR.5-7

Specific to the cardiology field, image fusion has been primarily used to integrate anatomical datasets, such as CT, MR or fluoroscopy, to functional ones, such as SPECT or PET. For a number of cardiovascular diseases, anatomy and physiology represent two sides of the same coin: assessment of both can greatly improve physicians’ ability to diagnose and plan treatment. Coronary artery disease (CAD) assessment is greatly improved by the availability of information on the anatomy of coronary vasculature, the degrees of a stenosis or the presence of diffuse disease, the extent of myocardial ischemia.8-11 The initial attempts to fuse anatomical and functional information date back to the early 1990s12 and successively became more established techniques that proved the incremental diagnostic power of fused imagery compared to single modalities or side-by-side interpretations.13-15 In cardiac electrophysiology, the success rate of interventions such as cardiac ablation16 and cardiac resynchronization therapy (CRT) strongly relies on the ability to not only accurately select patients, but also to correctly plan the procedure and, in case of CRT, to deliver the pacemaker leads into optimal position.17

Nevertheless, despite extensive research and test applications, multimodality image fusion techniques have not been translated into clinical routine. The main reason is that image fusion remains a highly technical procedure at the conjunction of several scientific disciplines beyond medicine, such as image processing, computer vision and pattern recognition, and with a number of hurdles still to overcome. Primarily, image registration techniques18 can suffer from significant inaccuracies related to patient positioning and tissue non-rigid deformations. In situations of relative organ rigidity, such as with the brain, image registration can be both automated and accurate. But in cardiovascular applications cardiac motion and respiratory artifacts, not to mention patient motion during long acquisition times, can make rigidity assumptions inadequate and results suboptimal. On the other hand, techniques for deformable registration19 are not established yet, mostly manually driven and can be time inefficient.

The article by Massera et al20 in this issue of the Journal represents a constructive step forward towards enhancing current image registration techniques and provide access to physicians to user-friendly image fusion tools. The authors presented and tested their integrated software FusionQuant for 18-F Sodium Fluoride PET and CTA image fusion in the context of the study of aortic stenosis and assessment of leaflet calcifications. A major issue when fusing nuclear acquisitions to anatomical ones resides in the fact that PET/SPECT studies are acquired over a number of heart beats while a CT represents a single snapshot of the cardiac cycle. Routinely, an averaged nuclear image or only a small portion of the nuclear gated data is used for fusion purposes, likely reducing accuracy. Instead, in their work the authors integrated into the FusionQuant software a deformable image registration algorithm that uses the entire gated acquisition and compared their results to another software package with only rigid image registration capabilities. The more sophisticated image registration approach resulted in decreased image noise, increased SNR and improved scan-rescan reproducibility of the measured indexes. Another important feature of image fusion, which is also highlighted in the work of Massera et al,20 is in fact its ability to investigate over-time disease progression or treatment outcomes by fusing follow-up images to previous acquisitions in an objective and consistent manner. Importantly, these enhancements came without impacting usability, as only simple manual interactions were required, and a decrease in total image processing time.

The key to enter a clinical environment remains undoubtedly automation. Clinicians do not have the time for prolonged image-processing tasks and possess little expertise in assessing the efficacy of complicated registration algorithms. In this field developers of multimodality image fusion frameworks and applications have still some work to do. The nuclear cardiology community has witnessed the tremendous impact that software packages for automated and robust image quantification have had in the clinical world. Image quantification, and particularly image segmentation, is less established for anatomical acquisitions. The normal variability in anatomical structures in addition to variations associated with pathological conditions make the design of general algorithms particularly challenging. And yet they are the more necessary, as the number of images per dataset have moved from a few hundreds to thousands. The recent introduction of machine learning (ML) techniques has transformed many areas of medical image processing with important developments also for the assessment of cardiovascular diseases.21 Despite some reasonable concerns on the use of ML techniques without experts’ supervision, there are fields of image processing that could greatly benefit from the availability of automated and reliable algorithms for image quantification.

Multimodality image fusion is also by definition cross-specialty. In addition to the further technical developments necessary in image processing methods, an environment of collaboration and communication between the different departments and radiology specialties will be fundamental. Image fusion applications will need to seamlessly access raw data, complete their processing tasks and enabling visualization possibly from a single workstation. Concurrently, physicians may encounter a learning curve in approaching multimodality image fusion frameworks as they will be asked to interpret images from nuclear and anatomical imaging and to use new visualization techniques and hybrid displays.

It is also worth addressing two main concerns that are commonly raised against the use of multimodality image fusion for diagnostic purposes: the appropriateness of performing multiple imaging tests and the radiation burden associated with them. Promoting the development of more sophisticated tools for image fusion and their utilization in clinical environments does not equate to support an unchecked use of imaging resources and possibilities. In fact, while in many instances the first test allows to properly diagnose the patient with no need for additional imaging, there are controversial cases for which frequently more than one exam is administered before a final diagnosis can be made. Multimodality image fusion—if available—represents the best way to take full advantage of the information already obtained, potentially reducing the need for further assessments or interventions down the line. On the issue of patient radiation exposure, pressure from the public and the medical communities has contributed to a renewed effort from physicians and imaging companies towards decreasing radiation doses for patients. The nuclear cardiology community is promoting guidelines22 that encourage departments to decrease radiation burden and develop new technologies to minimize injected doses and acquisition times. In cardiac CT, specific protocols have been designed to perform clinical-quality acquisitions with a minimal radiation burden. A compelling case for multimodality image fusion could then be made particularly for certain populations. For instance, in CAD assessment, a significant percentage of patients scheduled for elective coronary angiography ultimately do not exhibit obstructive disease.23 A moderate increase in radiation dose during initial diagnosis could prevent unnecessary procedures. In complicated surgical interventions, a treatment plan could be outlined beforehand reducing procedure time while improving outcomes.

Although it is generally accepted that images from complementary modalities provide incremental prognostic and diagnostic value, in the PACIFIC study Danad et al24 investigated the accuracy of major noninvasive techniques (functional, anatomical and hybrid) for ischemic heart disease assessment and reported no additional diagnostic value for hybrid imaging, either by means of SPECT/CTA or PET/CTA. While differences in methods for image quantification may be one of the reasons for these findings, the study underlines the existence of a dynamic and not completely understood relationship between anatomy and function, in this case between the degree of a stenosis and the indication of ischemia. Robust multimodality image fusion techniques by investigating different aspects of a disease can certainly help us unravel some of these mechanisms.



There are no conflicts of interest for this publication.


  1. 1.
    Blankstein R, Di Carli M. Integration of coronary anatomy and myocardial perfusion imaging. Nat Rev Cardiol 2010;7:226–36.CrossRefGoogle Scholar
  2. 2.
    Prior JO, Fahrad H, Muller O. Multimodality imaging in ischemic cardiomyopathy. Curr Cardiovasc Imaging Rep 2014;7:9285.CrossRefGoogle Scholar
  3. 3.
    Abi-Jaoudeh N, Kruecker J, Kadoury S, et al. Multimodality image fusion-guided procedures: Techniques, accuracy, and applications. Cardiovasc Interv Radiol 2012;35:986–98.CrossRefGoogle Scholar
  4. 4.
    Santana C, Garcia EV, Faber TL, et al. Diagnostic performance of fusion myocardial imaging (MPI) and computed tomography angiography. J Nucl Cardiol 2009;16:201–11.CrossRefGoogle Scholar
  5. 5.
    Giannopoulos AA, Gaemperli O. Hybrid imaging in ischemic heart disease. Rev Esp Cardiol (Engl Ed) 2018;71:382–90.CrossRefGoogle Scholar
  6. 6.
    Nensa F, Poeppel TD, Beiderwellen K, et al. Hybrid PET/MR imaging of the heart: Feasibility and initial results. Radiology 2013;268:366–73.CrossRefGoogle Scholar
  7. 7.
    Flotats A, Knuuti J, Gutberlet M, et al. Hybrid cardiac imaging, SPECT/CT and PET/CT. A joint position statement by the European Association of Nuclear Medicine (EANM), the European Society of Cardiac Radiology (ESCR), and the European Council of Nuclear Cardiology (ECNC). Eur J Nucl Med Mol Imaging 2011;38:201–12.CrossRefGoogle Scholar
  8. 8.
    Liga R, Vontobel J, Rovai D, et al. Multicenter, multi-device hybrid imaging study of coronary artery disease: Results from the Evaluation of Integrated Cardiac Imaging for the detection and characterization of ischaemic heart disease (EVINCI) hybrid imaging population. Eur Heart J 2016;17:951–60.Google Scholar
  9. 9.
    Schaap J, Kauling RM, Boekholdt SM, et al. Incremental diagnostic accuracy of hybrid SPET/CT coronary angiography in a population with an intermediate to high pre-test likelihood of coronary artery disease. Eur Heart J Cardiovasc Imaging 2013;14:642–9.CrossRefGoogle Scholar
  10. 10.
    Kajander S, Joutsiniemi E, Saraste M, et al. Cardiac positron emission tomography/computed tomography imaging accurately detects anatomically and functionally significant coronary artery disease. Circulation 2010;122:603–13.CrossRefGoogle Scholar
  11. 11.
    Danad I, Raijmakers PG, Appelman YE, et al. Hybrid imaging using H215O PET and CT-based coronary angiography for the detection of coronary artery disease. J Nucl Med 2013;54:55–63.CrossRefGoogle Scholar
  12. 12.
    Peifer JW, Ezquerra NF, Cooke CD, Mullick R, Klein L, Hyche ME, Garcia EV. Visualization of multimodality cardiac imagery. IEEE Trans Biomed Eng 1990;37:744–56.CrossRefGoogle Scholar
  13. 13.
    Rispler S, Keidar Z, Ghersin E, et al. Integrated Single-photon emission computed tomography and computed tomography coronary angiography for the assessment of the hemodynamically significant coronary artery disease. J Am Coll Cardiol 2007;49:1059–67.CrossRefGoogle Scholar
  14. 14.
    Slomka PJ, Cheng VY, Dey D, et al. Quantitative analysis of myocardial perfusion SPECT anatomically guided by coregistered 64-slice coronary CT angiography. J Nucl Med 2009;50:1621–30.CrossRefGoogle Scholar
  15. 15.
    Piccinelli M, Santana C, Sirineni GKR, et al. Diagnostic performance of the quantification of myocardium at risk from MPI SPECT/CTA 2G fusion for detecting obstructive coronary disease: A multicenter trial. J Nucl Cardiol 2018;25:1376–86.CrossRefGoogle Scholar
  16. 16.
    Govil A, Calkins H, Spragg DD. Fusion of imaging technologies: How, when, and for whom? J Interv Card Electrophysiol 2011;32:195–203.CrossRefGoogle Scholar
  17. 17.
    Zhou W, Hou X, Tang X, et al. 3D fusion of venous anatomy on fluoroscopy venograms with epical surface on SPECT myocardial perfusion images for guiding CRT LV lead placement. JACC Cardiovasc Imaging 2014;7:1239–48.CrossRefGoogle Scholar
  18. 18.
    Hill DLG, Batchelor PG, Holden M, et al. Medical image registration. Phys Med Biol 2001;46:R1–45.CrossRefGoogle Scholar
  19. 19.
    Rueckert D, Aljabar P. Nonrigid registration of medical images: Theory, methods, and applications. IEEE Signal Process Mag 2010;27:113–9.CrossRefGoogle Scholar
  20. 20.
    Massera D, Mhairi D, Kwiecinski J, et al. Analytical quantification of aortic uptake 18F-sodium fluoride PET uptake. Current Issue.Google Scholar
  21. 21.
    Al’Aref SJ, Anchouche K, Singh G, et al. Clinical applications of machine learning in cardiovascular disease and its relevance to cardiac imaging. Eur Heart J 2018 (in press).Google Scholar
  22. 22.
    Cerqueira MD, Allman KC, Ficaro PE, et al. Recommendations for reducing radiation exposure in myocardial perfusion imaging. J Nucl Cardiol. 2010;17:709–18.CrossRefGoogle Scholar
  23. 23.
    Patel MR, Dai D, Hernandez AF, et al. Prevalence and predictors of nonobstructive coronary artery disease identified with coronary angiography in contemporary clinical practice. Am Heart J. 2014;167:846–52.CrossRefGoogle Scholar
  24. 24.
    Danad I, Raijmakers PG, Driessen RS, et al. Comparison of coronary CT angiography, SPECT, PET, and hybrid imaging for diagnosis of ischemic heart disease determined by fractional flow reserve. JAMA Cardiol. 2017;2:1100–7.CrossRefGoogle Scholar

Copyright information

© American Society of Nuclear Cardiology 2019

Authors and Affiliations

  1. 1.Department of Radiology and Imaging SciencesEmory University School of MedicineAtlantaUSA

Personalised recommendations