Self-supervised Recurrent Neural Network for 4D Abdominal and In-utero MR Imaging

  • Tong ZhangEmail author
  • Laurence H. Jackson
  • Alena Uus
  • James R. Clough
  • Lisa Story
  • Mary A. Rutherford
  • Joseph V. Hajnal
  • Maria Deprez
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11905)


Accurately estimating and correcting the motion artifacts are crucial for 3D image reconstruction of the abdominal and in-utero magnetic resonance imaging (MRI). The state-of-art methods are based on slice-to-volume registration (SVR) where multiple 2D image stacks are acquired in three orthogonal orientations. In this work, we present a novel reconstruction pipeline that only needs one orientation of 2D MRI scans and can reconstruct the full high-resolution image without masking or registration steps. The framework consists of two main stages: the respiratory motion estimation using a self-supervised recurrent neural network, which learns the respiratory signals that are naturally embedded in the asymmetry relationship of the neighborhood slices and cluster them according to a respiratory state. Then, we train a 3D deconvolutional network for super-resolution (SR) reconstruction of the sparsely selected 2D images using integrated reconstruction and total variation loss. We evaluate the classification accuracy on 5 simulated images and compare our results with the SVR method in adult abdominal and in-utero MRI scans. The results show that the proposed pipeline can accurately estimate the respiratory state and reconstruct 4D SR volumes with better or similar performance to the 3D SVR pipeline with less than 20% sparsely selected slices. The method has great potential to transform the 4D abdominal and in-utero MRI in clinical practice.



This work was supported by the National Institutes of Health Human Placenta Project [1U01HD087202-01], by the Wellcome Trust IEH Award [102431], by the Wellcome/EPSRC Centre for Medical Engineering [WT203148/Z/16/Z] and by the National Institute for Health Research (NIHR) Biomedical Research Centre at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London. The authors also thank NVIDIA Corporation for the GPU grant.


  1. 1.
    Lloyd, D.F.A., et al.: Three-dimensional visualisation of the fetal heart using prenatal MRI with motion corrected slice-volume registration. Lancet 393, 1619–1627 (2018)CrossRefGoogle Scholar
  2. 2.
    Story, L., Zhang, T., Aljabar, P., Hajnal, J., Shennan, A., Rutherford, M.: Magnetic resonance imaging assessment of lung volumes in fetuses at high risk of preterm birth. BJOG Int. J. Obstet. Gynaecol. 124, 24 (2017)CrossRefGoogle Scholar
  3. 3.
    Story, L., Hutter, J., Zhang, T., Shennan, A.H., Rutherford, M.: The use of antenatal fetal magnetic resonance imaging in the assessment of patients at high risk of preterm birth. Eur. J. Obstet. Gynecol. Reprod. Biol. 222, 134–141 (2018)CrossRefGoogle Scholar
  4. 4.
    Story, L., et al.: Magnetic resonance imaging assessment of lung: body volume ratios in fetuses at high risk of preterm birth. BJOG Int. J. Obstet. Gynaecol. 126, 8 (2019)CrossRefGoogle Scholar
  5. 5.
    Gholipour, A., Estroff, J.A., Warfield, S.K.: Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain MRI. IEEE Trans. Med. Imaging 29(10), 1739–1758 (2010)CrossRefGoogle Scholar
  6. 6.
    Kuklisova-Murgasova, M., Quaghebeur, G., Rutherford, M.A., Hajnal, J.V., Schnabel, J.A.: Reconstruction of fetal brain MRI with intensity matching and complete outlier removal. Med. Image Anal. 16(8), 1550–1564 (2012)CrossRefGoogle Scholar
  7. 7.
    Kainz, B., et al.: Fast volume reconstruction from motion corrupted stacks of 2D slices. IEEE Trans. Med. Imaging 34(9), 1901–1913 (2015)CrossRefGoogle Scholar
  8. 8.
    Ebner, M., et al.: An automated localization, segmentation and reconstruction framework for fetal brain MRI. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 313–320. Springer, Cham (2018). Scholar
  9. 9.
    Torrents-Barrena, J., et al.: Fully automatic 3D reconstruction of the placenta and its peripheral vasculature in intrauterine fetal MRI. Med. Image Anal. 54, 263–279 (2019)CrossRefGoogle Scholar
  10. 10.
    Ramanathan, V., Tang, K., Mori, G., Fei-Fei, L.: Learning temporal embeddings for complex video analysis. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4471–4479 (2015)Google Scholar
  11. 11.
    Fernando, B., Bilen, H., Gavves, E., Gould, S.: Self-supervised video representation learning with odd-one-out networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3636–3645 (2017)Google Scholar
  12. 12.
    Wei, D., Lim, J., Zisserman, A., Freeman, W.T.: Learning and using the arrow of time, pp. 8052–8060 (2018)Google Scholar
  13. 13.
    Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18(5–6), 602–610 (2005)CrossRefGoogle Scholar
  14. 14.
    Jackson, L.H., et al.: Respiration resolved imaging using continuous steady state multiband excitation with linear frequency sweeps. In: ISMRM, Paris, ISMRM, pp. 5–7 (2018)Google Scholar
  15. 15.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  16. 16.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tong Zhang
    • 1
    Email author
  • Laurence H. Jackson
    • 1
  • Alena Uus
    • 1
  • James R. Clough
    • 1
  • Lisa Story
    • 2
  • Mary A. Rutherford
    • 1
  • Joseph V. Hajnal
    • 1
  • Maria Deprez
    • 1
  1. 1.School of Biomedical Engineering and Imaging SciencesKing’s College LondonLondonUK
  2. 2.Department of Women and Children’s Health, School of Life Course SciencesKing’s College LondonLondonUK

Personalised recommendations