Abstract
The core problem of Magnetic Resonance Imaging (MRI) is the trade off between acceleration and image quality. Image reconstruction and super-resolution are two crucial techniques in Magnetic Resonance Imaging (MRI). Current methods are designed to perform these tasks separately, ignoring the correlations between them. In this work, we propose an end-to-end task transformer network (T\(^2\)Net) for joint MRI reconstruction and super-resolution, which allows representations and feature transmission to be shared between multiple task to achieve higher-quality, super-resolved and motion-artifacts-free images from highly undersampled and degenerated MRI data. Our framework combines both reconstruction and super-resolution, divided into two sub-branches, whose features are expressed as queries and keys. Specifically, we encourage joint feature learning between the two tasks, thereby transferring accurate task information. We first use two separate CNN branches to extract task-specific features. Then, a task transformer module is designed to embed and synthesize the relevance between the two tasks. Experimental results show that our multi-task model significantly outperforms advanced sequential methods, both quantitatively and qualitatively.
C.-M. Feng and Y. Yan are contributed equally to this work. This work was done during the internship of C.-M. Feng at Inception Institute of Artificial Intelligence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cai, J., Han, H., Shan, S., Chen, X.: FCSR-GAN : joint face completion and super-resolution via multi-task learning. IEEE Trans. Biomet, Behav. Identity Sci. 2(2), 109–121 (2019)
Chaudhari, A.S., et al.: Super-resolution musculoskeletal MRI using deep learning. Magn. Resonan. Med. 80(5), 2139–2154 (2018)
Chen, Y., Shi, F., Christodoulou, A.G., Xie, Y., Zhou, Z., Li, D.: Efficient and accurate MRIi super-resolution using a generative adversarial network and 3D multi-level densely connected network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 91–99. Springer (2018)
Collobert, R., Weston, J.: A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, pp. 160–167 (2008)L
Feng, C.M., Fu, H., Yuan, S., Xu, Y.: Multi-contrast MRI super-resolution via a multi-stage integration network. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (2021)
Feng, C.M., Wang, K., Lu, S., Xu, Y., Li, X.: Brain MRI super-resolution using coupled-projection residual network. Neurocomputing 456, 190–199 (2021)
Feng, C.M., Yan, Y., Chen, G., Fu, H., Xu, Y., Shao, L.: Accelerated multi-modal MR imaging with transformers (2021)
Feng, C.M., Yang, Z., Chen, G., Xu, Y., Shao, L.: Dual-octave convolution for accelerated parallel MR image reconstruction. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI) (2021)
Feng, C.M., Yang, Z., Fu, H., Xu, Y., Yang, J., Shao, L.: DONet: dual-octave network for fast MR image reconstruction. IEEE Trans. Neural Netw. Learn. Syst. (2021)
Hammernik, K., et al.: Learning a variational network for reconstruction of accelerated MRI data. Magn, Resonan. Med. 79(6), 3055–3071 (2018)
Huang, Q., Yang, D., Wu, P., Qu, H., Yi, J., Metaxas, D.: MRI reconstruction via cascaded channel-wise attention network. In: International Symposium on Biomedical Imaging (ISBI 2019), pp. 1622–1626. IEEE (2019)
Kim, S., Hori, T., Watanabe, S.: Joint CTC-attention based end-to-end speech recognition using multi-task learning. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4835–4839. IEEE (2017)
Lai, Z., et al.: Image reconstruction of compressed sensing MRI using graph-based redundant wavelet transform. Med. Image Anal. 27, 93–104 (2016)
Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)
Liu, S., Johns, E., Davison, A.J.: End-to-end multi-task learning with attention. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1871–1880 (2019)
Lyu, Q., Shan, H., Wang, G.: MRI super-resolution with ensemble learning and complementary priors. IEEE Trans. Comput. Imag. 6, 615–624 (2020)
Lyu, Q., You, C., Shan, H., Zhang, Y., Wang, G.: Super-resolution MRI and CT through gan-circle. In: Developments in X-Ray Tomography XII, vol. 11113, p. 111130X. International Society for Optics and Photonics (2019)
Mahapatra, D., Bozorgtabar, B., Garnavi, R.: Image super-resolution using progressive generative adversarial networks for medical image analysis. Comput. Med. Imag. Graph. 71, 30–39 (2019)
Mardani, M., Gong, E., Cheng, J.Y., Vasanawala, S.S., Zaharchuk, G., Xing, L., Pauly, J.M.: Deep generative adversarial neural networks for compressive sensing MRI. IEEE Trans. Med. Imaging 38(1), 167–179 (2018)
Nakarmi, U., Wang, Y., Lyu, J., Liang, D., Ying, L.: A kernel-based low-rank (KLR) model for low-dimensional manifold recovery in highly accelerated dynamic MRI. IEEE Trans. Medi. Imag. 36(11), 2297–2307 (2017)
Oktay, O., Bai, W., Lee, M., Guerrero, R., Kamnitsas, K., Caballero, J., de Marvao, A., Cook, S., O’Regan, D., Rueckert, D.: Multi-input cardiac image super-resolution using convolutional neural networks. In: International Conference on Medical Image Computing and Computer Assisted Intervention. pp. 246–254. Springer (2016). https://doi.org/10.1007/10704282
Qin, C., Schlemper, J., Caballero, J., Price, A.N., Hajnal, J.V., Rueckert, D.: Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Trans. Medical Imag. 38(1), 280–290 (2018)
Ravishankar, S., Bresler, Y.: MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imag. 30(5), 1028–1041 (2010)
Shi, J., Liu, Q., Wang, C., Zhang, Q., Ying, S., Xu, H.: Super-resolution reconstruction of MR image with a novel residual learning network algorithm. Phys. Med. Biol. 63(8), 085011 (2018)
Shin, P.J., Larson, P.E., Ohliger, M.A., Elad, M., Pauly, J.M., Vigneron, D.B., Lustig, M.: Calibrationless parallel imaging reconstruction based on structured low-rank matrix completion. Magn. Resonan. Med. 72(4), 959–970 (2014)
Wang, S., Cheng, H., Ying, L., Xiao, T., Ke, Z., Zheng, H., Liang, D.: Deepcomplexmri: exploiting deep residual network for fast parallel MR imaging with complex convolution. Magn. Resonan. Imag. 68, 136–147 (2020)
Wang, Y.H., Qiao, J., Li, J.B., Fu, P., Chu, S.C., Roddick, J.F.: Sparse representation-based MRI super-resolution reconstruction. Measurement 47, 946–953 (2014)
Yang, G., Yu, S., Dong, H., Slabaugh, G., Dragotti, P.L., Ye, X., Liu, F., Arridge, S., Keegan, J., Guo, Y., et al.: Dagan: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction. IEEE Trans. Med. Imag. 37(6), 1310–1321 (2017)
Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-net for compressive sensing MRI. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 10–18 (2016)
Zhan, Z., Cai, J.F., Guo, D., Liu, Y., Chen, Z., Qu, X.: Fast multiclass dictionaries learning with geometrical directions in MRI reconstruction. IEEE Trans. Biomed. Eng. 63(9), 1850–1861 (2015)
Zhang, M., Liu, W., Ma, H.: Joint license plate super-resolution and recognition in one multi-task gan framework. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1443–1447. IEEE (2018)
Zhang, X., Lam, E.Y., Wu, E.X., Wong, K.K.Y.: Application of Tikhonov regularization to super-resolution reconstruction of brain MRI images. In: Gao, X., Müller, H., Loomes, M.J., Comley, R., Luo, S. (eds.) MIMI 2007. LNCS, vol. 4987, pp. 51–56. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-79490-5_8
Zhu, B., Liu, J.Z., Cauley, S.F., Rosen, B.R., Rosen, M.S.: Image reconstruction by domain-transform manifold learning. Nature 555(7697), 487–492 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Feng, CM., Yan, Y., Fu, H., Chen, L., Xu, Y. (2021). Task Transformer Network for Joint MRI Reconstruction and Super-Resolution. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12906. Springer, Cham. https://doi.org/10.1007/978-3-030-87231-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-87231-1_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87230-4
Online ISBN: 978-3-030-87231-1
eBook Packages: Computer ScienceComputer Science (R0)
