Skip to main content
SpringerLink
Log in

Multimodal Cardiac Segmentation Using Disentangled Representation Learning

  • Conference paper
  • First Online:
Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges (STACOM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12009))

Abstract

Magnetic Resonance (MR) protocols use several sequences to evaluate pathology and organ status. Yet, despite recent advances, the analysis of each sequence’s images (modality hereafter) is treated in isolation. We propose a method suitable for multimodal and multi-input learning and analysis, that disentangles anatomical and imaging factors, and combines anatomical content across the modalities to extract more accurate segmentation masks. Mis-registrations between the inputs are handled with a Spatial Transformer Network, which non-linearly aligns the (now intensity-invariant) anatomical factors. We demonstrate applications in Late Gadolinium Enhanced (LGE) and cine MRI segmentation. We show that multi-input outperforms single-input models, and that we can train a (semi-supervised) model with few (or no) annotations for one of the modalities. Code is available at https://github.com/agis85/multimodal_segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bookstein, F.L.: Principal warps: thin-plate splines and the decomposition of deformations. IEEE PAMI 11(6), 567–585 (1989)

    Article  Google Scholar 

  2. Chartsias, A., et al.: Disentangled representation learning in cardiac image analysis. Med. Image Anal. 58, 101535 (2019)

    Article  Google Scholar 

  3. Chollet, F.: Keras (2015). https://keras.io

  4. Dolz, J., Gopinath, K., Yuan, J., Lombaert, H., Desrosiers, C., Ayed, I.B.: HyperDense-Net: a hyper-densely connected CNN for multi-modal image segmentation. In: IEEE TMI (2018)

    Google Scholar 

  5. Fidon, L., et al.: Scalable multimodal convolutional networks for brain tumour segmentation. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 285–293. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_33

    Chapter  Google Scholar 

  6. Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)

    Article  Google Scholar 

  7. Huang, X., Liu, M.-Y., Belongie, S., Kautz, J.: Multimodal unsupervised image-to-image translation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 179–196. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_11

    Chapter  Google Scholar 

  8. Huo, Y., et al.: SynSeg-Net: synthetic segmentation without target modality ground truth. In: IEEE TMI (2018)

    Google Scholar 

  9. Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: NIPS, pp. 2017–2025 (2015)

    Google Scholar 

  10. Joyce, T., Chartsias, A., Tsaftaris, S.A.: Robust multi-modal MR image synthesis. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 347–355. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_40

    Chapter  Google Scholar 

  11. Kim, H.W., Farzaneh-Far, A., Kim, R.J.: Cardiovascular magnetic resonance in patients with myocardial infarction: current and emerging applications. JACC 55(1), 1–16 (2009)

    Article  Google Scholar 

  12. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)

    Google Scholar 

  13. Lee, H.-Y., Tseng, H.-Y., Huang, J.-B., Singh, M., Yang, M.-H.: Diverse image-to-image translation via disentangled representations. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 36–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_3

    Chapter  Google Scholar 

  14. Liu, J., Xie, H., Zhang, S., Gu, L.: Multi-sequence myocardium segmentation with cross-constrained shape and neural network-based initialization. CMIG 71, 49–57 (2019)

    Google Scholar 

  15. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: On the effectiveness of least squares generative adversarial networks. In: IEEE PAMI (2019)

    Google Scholar 

  16. Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: visual reasoning with a general conditioning layer. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  17. Qin, C., Shi, B., Liao, R., Mansi, T., Rueckert, D., Kamen, A.: Unsupervised deformable registration for multi-modal images via disentangled representations. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 249–261. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20351-1_19

    Chapter  Google Scholar 

  18. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  19. Stirrat, C.G., et al.: Ferumoxytol-enhanced magnetic resonance imaging assessing inflammation after myocardial infarction. Heart 103(19), 1528–1535 (2017)

    Article  Google Scholar 

  20. Tseng, K.-L., Lin, Y.-L., Hsu, W., Huang, C.-Y.: Joint sequence learning and cross-modality convolution for 3D biomedical segmentation. In: CVPR, pp. 6393–6400 (2017)

    Google Scholar 

  21. Tustison, N.J., Yang, Y., Salerno, M.: Advanced normalization tools for cardiac motion correction. In: Camara, O., Mansi, T., Pop, M., Rhode, K., Sermesant, M., Young, A. (eds.) STACOM 2014. LNCS, vol. 8896, pp. 3–12. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14678-2_1

    Chapter  Google Scholar 

  22. Valindria, V., et al.: Multi-modal learning from unpaired images: application to multi-organ segmentation in CT and MRI. In: WACV (2018)

    Google Scholar 

  23. Zhang, Z., Yang, L., Zheng, Y.: Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: CVPR, pp. 9242–9251 (2018)

    Google Scholar 

  24. Zhuang, X.: Multivariate mixture model for myocardial segmentation combining multi-source images. In: IEEE PAMI (2019)

    Google Scholar 

  25. Zhuang, X., Shen, J.: Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med. Image Anal. 31, 77–87 (2016)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by UK EPSRC (EP/P022928/1) and US National Institutes of Health (1R01HL136578-01), and used resources from the Edinburgh Compute and Data Facility. S.A. Tsaftaris acknowledges the Royal Academy of Engineering and the Research Chairs and Senior Research Fellowships scheme.

Author information

Authors and Affiliations

  1. School of Engineering, University of Edinburgh, Edinburgh, EH9 3FB, UK

    Agisilaos Chartsias & Sotirios A. Tsaftaris

  2. Edinburgh Imaging Facility QMRI, Edinburgh, EH16 4TJ, UK

    Giorgos Papanastasiou, Chengjia Wang, Colin Stirrat, Scott Semple & David Newby

  3. Centre for Cardiovascular Science, Edinburgh, EH16 4TJ, UK

    Giorgos Papanastasiou, Chengjia Wang, Colin Stirrat, Scott Semple & David Newby

  4. The Alan Turing Institute, London, UK

    Sotirios A. Tsaftaris

  5. Cedars Sinai Medical Center, Los Angeles, CA, USA

    Rohan Dharmakumar

Authors
  1. Agisilaos Chartsias
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giorgos Papanastasiou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengjia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Colin Stirrat
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Scott Semple
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. David Newby
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rohan Dharmakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sotirios A. Tsaftaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Agisilaos Chartsias .

Editor information

Editors and Affiliations

  1. University of Toronto, Toronto, ON, Canada

    Mihaela Pop

  2. Inria, Sophia Antipolis, France

    Maxime Sermesant

  3. Pompeu Fabra University, Barcelona, Spain

    Oscar Camara

  4. Fudan University, Shanghai, China

    Xiahai Zhuang

  5. St Joseph’s Health Care, London, ON, Canada

    Shuo Li

  6. King’s College London, London, UK

    Alistair Young

  7. Siemens Medical Solutions USA, Inc., Princeton, NJ, USA

    Tommaso Mansi

  8. University of Auckland, Auckland, New Zealand

    Avan Suinesiaputra

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chartsias, A. et al. (2020). Multimodal Cardiac Segmentation Using Disentangled Representation Learning. In: Pop, M., et al. Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges. STACOM 2019. Lecture Notes in Computer Science(), vol 12009. Springer, Cham. https://doi.org/10.1007/978-3-030-39074-7_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-39074-7_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-39073-0

  • Online ISBN: 978-3-030-39074-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation