Abstract
In the field of medical imaging, 3D deep learning models play a crucial role in building powerful predictive models of disease progression. However, the size of these models presents significant challenges, both in terms of computational resources and data requirements. Moreover, achieving high-quality pretraining of 3D models proves to be even more challenging. To address these issues, hybrid 2.5D approaches provide an effective solution for utilizing 3D volumetric data efficiently using 2D models. Combining 2D and 3D techniques offers a promising avenue for optimizing performance while minimizing memory requirements. In this paper, we explore 2.5D architectures based on a combination of convolutional neural networks (CNNs), long short-term memory (LSTM), and Transformers. In addition, leveraging the benefits of recent non-contrastive pretraining approaches in 2D, we enhanced the performance and data efficiency of 2.5D techniques even further. We demonstrate the effectiveness of architectures and associated pretraining on a task of predicting progression to wet age-related macular degeneration (AMD) within a six-month period on two large longitudinal OCT datasets.
T. Emre and M. Oghbaie—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
NCT00891735. https://clinicaltrials.gov/ct2/show/NCT00891735.
References
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: ViVit: a video vision transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836–6846 (2021)
Balestriero, R., et al.: A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 (2023)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Chen, Z., Xie, L., Niu, J., Liu, X., Wei, L., Tian, Q.: Visformer: the vision-friendly transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 589–598 (2021)
Das, V., Prabhakararao, E., Dandapat, S., Bora, P.K.: B-scan attentive cnn for the classification of retinal optical coherence tomography volumes. IEEE Signal Process. Lett. 27, 1025–1029 (2020)
Emre, T., Chakravarty, A., Rivail, A., Riedl, S., Schmidt-Erfurth, U., Bogunović, H.: TINC: temporally informed non-contrastive learning for disease progression modeling in retinal OCT volumes. In: Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, 18–22 September 2022, Proceedings, Part II, pp. 625–634. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-16434-7_60
Fan, H., et al.: Multiscale vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6824–6835 (2021)
Farsiu, S., Chiu, S., O’Connell, R., Folgar, F.: Quantitative classification of Eyes with and without intermediate age-related macular degeneration using optical coherence tomography. Ophthalmology 121(1), 162–172 (2014). www.sciencedirect.com/science/article/pii/S016164201300612X
Fazekas, B., Lachinov, D., Aresta, G., Mai, J., Schmidt-Erfurth, U., Bogunovic, H.: Segmentation of bruch’s membrane in retinal oct with amd using anatomical priors and uncertainty quantification. IEEE J. Biomed. Health Inf. 27(1), 41–52 (2023). https://doi.org/10.1109/JBHI.2022.3217962
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Kurmann, T., Márquez-Neila, P., Yu, S., Munk, M., Wolf, S., Sznitman, R.: Fused detection of retinal biomarkers in OCT volumes. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 255–263. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_29
Lee, S.H., Lee, S., Song, B.C.: Vision transformer for small-size datasets. arXiv preprint arXiv:2112.13492 (2021)
Li, H., et al.: DT-MIL: deformable transformer for multi-instance learning on histopathological image. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 206–216. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_20
Maintainers, T.: Contributors: torchvision: pytorch’s computer vision library (2016). www.github.com/pytorch/vision
Neimark, D., Bar, O., Zohar, M., Asselmann, D.: Video transformer network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 3163–3172 (2021)
Shao, Z., et al.: Transmil: transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural Inf. Process. Syst. 34, 2136–2147 (2021)
Sutton, J., et al.: Developing and validating a multivariable prediction model which predicts progression of intermediate to late age-related macular degeneration-the pinnacle trial protocol. In: Eye, pp. 1–9 (2022)
Tang, Y., et al.: Self-supervised pre-training of swin transformers for 3d medical image analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20730–20740 (2022)
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, pp. 10347–10357. PMLR (2021)
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., Girshick, R.: Early convolutions help transformers see better. Adv. Neural Inf. Process. Syst. 34, 30392–30400 (2021)
Acknowledgements
This work was supported in part by Wellcome Trust Collaborative Award (PINNACLE) Ref. 210572/Z/18/Z, Christian Doppler Research Association, and FWF (Austrian Science Fund; grant no. FG 9-N).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Emre, T. et al. (2023). Pretrained Deep 2.5D Models for Efficient Predictive Modeling from Retinal OCT: A PINNACLE Study Report. In: Antony, B., Chen, H., Fang, H., Fu, H., Lee, C.S., Zheng, Y. (eds) Ophthalmic Medical Image Analysis. OMIA 2023. Lecture Notes in Computer Science, vol 14096. Springer, Cham. https://doi.org/10.1007/978-3-031-44013-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-44013-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44012-0
Online ISBN: 978-3-031-44013-7
eBook Packages: Computer ScienceComputer Science (R0)
