Fusion of 3D real-time echocardiography and cine MRI using a saliency analysis

  • Angélica AtehortúaEmail author
  • Mireille Garreau
  • Antoine Simon
  • Erwan Donal
  • Mathieu Lederlin
  • Eduardo Romero
Original Article



This paper presents a novel 3D multimodal registration strategy to fuse 3D real-time echocardiography images with cardiac cine MRI images. This alignment is performed in a saliency space, which is designed to maximize similarity between the two imaging modalities. This fusion improves the quality of the available information.


The method performs in two steps: temporal and spatial registrations. A temporal alignment is firstly achieved by nonlinearly matching pairs of correspondences between the two modalities using a dynamic time warping. A temporal registration is then carried out by applying nonrigid transformations in a common saliency space where normalized cross correlation between temporal pairs of salient volumes is maximized.


The alignment performance was evaluated with a set of 18 subjects, 3 with cardiomyopathies and 15 healthy, by computing the Dice score and Hausdorff distance with respect to manual delineations of the left ventricle cavity in both modalities. A Dice score and Hausdorff distance of \(0.86\pm 0.04\) and \(13.61\pm 3.86\,\hbox {mm}\), respectively, were obtained. In addition, the deformation field was estimated by quantifying its foldings, obtaining a 98% of regularity in the deformation field.


The 3D multimodal registration strategy presented is performed in a saliency space. Unlike state-of-the-art methods, the presented one takes advantage of the temporal information of the heart to construct this common space, ending up with two well-aligned modalities and regular deformation fields. This preliminary study was evaluated on heterogeneous data composed of two different datasets, healthy and pathological cases, showing similar performances in both cases. Future work will focus on testing the presented strategy in a larger dataset with a balanced number of classes.


Image registration Spatio-temporal saliency Cardiac cine MRI images Echocardiography 



This work was supported by Colciencias-Colombia, Grant No. 647 (2015 call for National Ph.D. studies), and Région Bretagne in the framework of the Investissement d’Avenir Program through Labex CAMI (ANR-11-LABX-0004).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Susstrunk S (2012) Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282CrossRefGoogle Scholar
  2. 2.
    Atehortúa A, Garreau M, Romero E (2017) Fusion of 4D echocardiography and cine cardiac magnetic resonance volumes using a salient spatio-temporal analysis. In 13th International conference on medical information processing and analysis (Vol. 10572, p. 105721A). International Society for Optics and PhotonicsGoogle Scholar
  3. 3.
    Badesch DB, Champion HC, Sanchez MAG, Hoeper MM, Loyd JE, Manes A, McGoon M, Naeije R, Olschewski H, Oudiz RJ (2009) Diagnosis and assessment of pulmonary arterial hypertension. J Am Coll Cardiol 54(1 Supplement):S55–S66CrossRefGoogle Scholar
  4. 4.
    Berndt DJ, Clifford J (1994) Using dynamic time warping to find patterns in time series. In: KDD workshop. vol. 10, pp. 359–370. Seattle, WAGoogle Scholar
  5. 5.
    Betancur J, Simon A, Langella B, Leclercq C, Hernández A, Garreau M (2016) Synchronization and registration of cine magnetic resonance and dynamic computed tomography images of the heart. IEEE J Biomed Health Inform 20(5):1369–1376CrossRefGoogle Scholar
  6. 6.
    Burt P, Adelson E (1983) The Laplacian pyramid as a compact image code. IEEE Trans Commun 31(4):532–540CrossRefGoogle Scholar
  7. 7.
    Dahlem K, Michels G, Kobe C, Bunck AC, Ten Freyhaus H, Pfister R (2017) Diagnosis of cardiac transthyretin amyloidosis based on multimodality imaging. Clin Res Cardiol 106(6):471–473CrossRefGoogle Scholar
  8. 8.
    Freeman WT, Adelson EH (1991) The design and use of steerable filters. IEEE Trans Pattern Anal Mach Intell 9:891–906CrossRefGoogle Scholar
  9. 9.
    Grewenig S, Zimmer S, Weickert J (2011) Rotationally invariant similarity measures for nonlocal image denoising. J Vis Commun Image Represent 22(2):117–130CrossRefGoogle Scholar
  10. 10.
    Haugaa KH, Basso C, Badano LP, Bucciarelli-Ducci C, Cardim N, Gaemperli O, Galderisi M, Habib G, Knuuti J, Lancellotti P (2017) Comprehensive multi-modality imaging approach in arrhythmogenic cardiomyopathyan expert consensus document of the european association of cardiovascular imaging. Eur Heart J Cardiovasc Imaging 18(3):237–253CrossRefGoogle Scholar
  11. 11.
    Huang X, Hill NA, Ren J, Guiraudon G, Boughner D, Peters TM (2005) Dynamic 3d ultrasound and mr image registration of the beating heart. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 171–178. SpringerGoogle Scholar
  12. 12.
    Itti L (2005) Quantifying the contribution of low-level saliency to human eye movements in dynamic scenes. Vis Cognit 12(6):1093–1123CrossRefGoogle Scholar
  13. 13.
    James AP, Dasarathy BV (2014) Medical image fusion: a survey of the state of the art. Inf Fus 19:4–19CrossRefGoogle Scholar
  14. 14.
    Kanai T, Kadoya N, Ito K, Onozato Y, Cho SY, Kishi K, Jingu K (2014) Evaluation of accuracy of B-spline transformation-based deformable image registration with different parameter settings for thoracic images. J Radiat Res 55(6):1163–1170CrossRefGoogle Scholar
  15. 15.
    Kiss G, Thorstensen A, Amundsen B, Claus P, D’hooge J, Torp H (2012) Fusion of 3d echocardiographic and cardiac magnetic resonance volumes. In: 2012 IEEE International Ultrasonics Symposium. pp. 126–129. IEEEGoogle Scholar
  16. 16.
    Klein S, Pluim JP, Staring M, Viergever MA (2009) Adaptive stochastic gradient descent optimisation for image registration. Int J Comput Vis 81(3):227CrossRefGoogle Scholar
  17. 17.
    Klein S, Staring M, Murphy K, Viergever MA, Pluim JPW (2010) Elastix: a toolbox for intensity-based medical image registration. IEEE Trans Med Imaging 29(1):196–205CrossRefGoogle Scholar
  18. 18.
    Klein S, Staring M, Pluim JP (2007) Evaluation of optimization methods for nonrigid medical image registration using mutual information and B-splines. IEEE Trans Image Process 16(12):2879–2890CrossRefGoogle Scholar
  19. 19.
    Ma YL, Penney GP, Rinaldi CA, Cooklin M, Razavi R, Rhode KS (2009) Echocardiography to magnetic resonance image registration for use in image-guided cardiac catheterization procedures. Phys Med Biol 54(16):5039CrossRefGoogle Scholar
  20. 20.
    Mizuguchi Y, Oishi Y, Miyoshi H, Iuchi A, Nagase N, Oki T (2008) The functional role of longitudinal, circumferential, and radial myocardial deformation for regulating the early impairment of left ventricular contraction and relaxation in patients with cardiovascular risk factors: a study with two-dimensional strain imaging. J Am Soc Echocardiogr 21(10):1138–1144CrossRefGoogle Scholar
  21. 21.
    Nagueh SF, Chang SM, Nabi F, Shah DJ, Estep JD (2017) Imaging to diagnose and manage patients in heart failure with reduced ejection fraction. Circ Cardiovasc Imaging 10(4):e005615CrossRefGoogle Scholar
  22. 22.
    Olsen FJ, Bertelsen L, de Knegt MC, Christensen TE, Vejlstrup N, Svendsen JH, Jensen JS, Biering-Sorensen T (2016) Multimodality cardiac imaging for the assessment of left atrial function and the association with atrial arrhythmias. Circ Cardiovasc Imaging 9(10):e004947CrossRefGoogle Scholar
  23. 23.
    Ou Y, Akbari H, Bilello M, Da X, Davatzikos C (2014) Comparative evaluation of registration algorithms in different brain databases with varying difficulty: results and insights. IEEE Trans Med Imaging 33(10):2039–2065CrossRefGoogle Scholar
  24. 24.
    Perperidis D, Mohiaddin RH, Rueckert D (2005) Spatio-temporal free-form registration of cardiac MR image sequences. Med Image Anal 9(5):441–456CrossRefGoogle Scholar
  25. 25.
    Puyol-Anton E, Sinclair M, Gerber B, Amzulescu MS, Langet H, De Craene M, Aljabar P, Piro P, King AP (2017) A multimodal spatiotemporal cardiac motion atlas from MR and ultrasound data. Med Image Anal 40:96–110CrossRefGoogle Scholar
  26. 26.
    Rueckert D, Sonoda LI, Hayes C, Hill DL, Leach MO, Hawkes DJ (1999) Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging 18(8):712–721CrossRefGoogle Scholar
  27. 27.
    Szasz O (1952) On products of summability methods. Proc Am Math Soc 3(2):257–263CrossRefGoogle Scholar
  28. 28.
    Thorpe S, Fize D, Marlot C (1996) Speed of processing in the human visual system. Nature 381(6582):520CrossRefGoogle Scholar
  29. 29.
    Tobon-Gomez C, De Craene M, Mcleod K, Tautz L, Shi W, Hennemuth A, Prakosa A, Wang H, Carr-White G, Kapetanakis S (2013) Benchmarking framework for myocardial tracking and deformation algorithms: an open access database. Med Image Anal 17(6):632–648CrossRefGoogle Scholar
  30. 30.
    Valsangiacomo Buechel ER, Mertens LL (2012) Imaging the right heart: the use of integrated multimodality imaging. Eur Heart J 33(8):949–960CrossRefGoogle Scholar
  31. 31.
    Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC, Gerig G (2006) User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31(3):1116–1128CrossRefGoogle Scholar
  32. 32.
    Zhai Y, Shah M (2006) Visual attention detection in video sequences using spatiotemporal cues. In: Proceedings of the 14th ACM international conference on Multimedia. pp. 815–824. ACMGoogle Scholar
  33. 33.
    Zhang W, Noble JA, Brady JM (2007) Spatio-temporal registration of real time 3d ultrasound to cardiovascular MR sequences. In: International conference on medical image computing and computer-assisted intervention. pp. 343–350. SpringerGoogle Scholar
  34. 34.
    Zhang W, Noble JA, Brady JM (2007) Adaptive non-rigid registration of real time 3d ultrasound to cardiovascular MR images. In: Biennial international conference on information processing in medical imaging. pp. 50–61. SpringerGoogle Scholar
  35. 35.
    Zhao N, Basarab A, Kouamé D, Tourneret JY (2016) Joint segmentation and deconvolution of ultrasound images using a hierarchical Bayesian model based on generalized Gaussian priors. IEEE Trans Image Process 25(8):3736–3750CrossRefGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.Univ Rennes, CHU Rennes, Inserm, LTSI - UMR 1099RennesFrance
  2. 2.Universidad Nacional de ColombiaBogotáColombia

Personalised recommendations