Multi-label classification of retinal lesions in diabetic retinopathy for automatic analysis of fundus fluorescein angiography based on deep learning

Abstract

Purpose

To automatically detect and classify the lesions of diabetic retinopathy (DR) in fundus fluorescein angiography (FFA) images using deep learning algorithm through comparing 3 convolutional neural networks (CNNs).

Methods

A total of 4067 FFA images from Eye Center at the Second Affiliated Hospital of Zhejiang University School of Medicine were annotated with 4 kinds of lesions of DR, including non-perfusion regions (NP), microaneurysms, leakages, and laser scars. Three CNNs including DenseNet, ResNet50, and VGG16 were trained to achieve multi-label classification, which means the algorithms could identify 4 retinal lesions above at the same time.

Results

The area under the curve (AUC) of DenseNet reached 0.8703, 0.9435, 0.9647, and 0.9653 for detecting NP, microaneurysms, leakages, and laser scars, respectively. For ResNet50, AUC was 0.8140 for NP, 0.9097 for microaneurysms, 0.9585 for leakages, and 0.9115 for laser scars. And for VGG16, AUC was 0.7125 for NP, 0.5569 for microaneurysms, 0.9177 for leakages, and 0.8537 for laser scars.

Conclusions

Experimental results demonstrate that DenseNet is a suitable model to automatically detect and distinguish retinal lesions in the FFA images with multi-label classification, which lies the foundation of automatic analysis for FFA images and comprehensive diagnosis and treatment decision-making for DR.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. 1.

    Huang Y (2017) International Diabetes Federation (IDF) (2017) IDF diabetes atlas. 8th edition. http://www.diabetesatlas.org/resources/2017-atlas.html. Accessed 9 Jun 2019

  2. 2.

    Li Z, Keel S, Liu C, He Y, Meng W, Scheetz J, Lee PY, Shaw J, Ting D, Wong TY, Taylor H, Chang R, He M (2018) An automated grading system for detection of vision-threatening referable diabetic retinopathy on the basis of color fundus photographs. Diabetes Care 41:2509–2516. https://doi.org/10.2337/dc18-0147

    Article  PubMed  Google Scholar 

  3. 3.

    Lamoke F, Shaw S, Yuan J, Ananth S, Duncan M, Martin P, Bartoli M (2015) Increased oxidative and nitrative stress accelerates aging of the retinal vasculature in the diabetic retina. PLoS One 10:e0139664. https://doi.org/10.1371/journal.pone.0139664

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  4. 4.

    Dodo Y, Murakami T, Uji A, Yoshitake S, Yoshimura N (2015) Disorganized retinal lamellar structures in nonperfused areas of diabetic retinopathy. Invest Ophthalmol Vis Sci 56:2012–2020. https://doi.org/10.1167/iovs.14-15924

    Article  PubMed  Google Scholar 

  5. 5.

    Unoki N, Nishijima K, Sakamoto A, Kita M, Watanabe D, Hangai M, Kimura T, Kawagoe N, Ohta M, Yoshimura N (2007) Retinal sensitivity loss and structural disturbance in areas of capillary nonperfusion of eyes with diabetic retinopathy. Am J Ophthalmol 144:755–760. https://doi.org/10.1016/j.ajo.2007.07.011

    Article  PubMed  Google Scholar 

  6. 6.

    Kozak I, El-Emam SY, Cheng L, Bartsch DU, Chhablani J, Freeman WR, Arevalo JF (2014) Fluorescein angiography versus optical coherence tomography-guided planning for macular laser photocoagulation in diabetic macular edema. Retina 34:1600–1605. https://doi.org/10.1097/iae.0000000000000120

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Zhang XH, Chutatape O, Ieee (2004) Detection and classification of bright lesions in color fundus images. Icip: 2004 international conference on image processing 1–5: 139–142

  8. 8.

    van Ginneken B (2017) Fifty years of computer analysis in chest imaging: rule-based, machine learning, deep learning. Radiol Phys Technol 10:23–32. https://doi.org/10.1007/s12194-017-0394-5

    Article  PubMed  PubMed Central  Google Scholar 

  9. 9.

    Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N (2017) Can machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS One 12:e0174944. https://doi.org/10.1371/journal.pone.0174944

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  10. 10.

    Bejnordi BE, Zuidhof G, Balkenhol M, Hermsen M, Bult P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J (2017) Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J Med Imaging (Bellingham) 4:044504. https://doi.org/10.1117/1.jmi.4.4.044504

    Article  Google Scholar 

  11. 11.

    Mercan C, Aksoy S, Mercan E, Shapiro LG, Weaver DL, Elmore JG (2018) Multi-instance multi-label learning for multi-class classification of whole slide breast histopathology images. IEEE Trans Med Imaging 37:316–325. https://doi.org/10.1109/tmi.2017.2758580

    Article  PubMed  Google Scholar 

  12. 12.

    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Corrigendum: dermatologist-level classification of skin cancer with deep neural networks. Nature 546:686. https://doi.org/10.1038/nature22985

    CAS  Article  PubMed  Google Scholar 

  13. 13.

    Abramoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, Niemeijer M (2016) Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci 57:5200–5206. https://doi.org/10.1167/iovs.16-19964

    Article  PubMed  Google Scholar 

  14. 14.

    Vidal-Alaball J, Royo Fibla D (2019) Artificial intelligence for the detection of diabetic retinopathy in primary care: protocol for algorithm development. JMIR Res Protoc 8:e12539. https://doi.org/10.2196/12539

    Article  PubMed  PubMed Central  Google Scholar 

  15. 15.

    Han SS, Kim MS, Lim W, Park GH, Park I, Chang SE (2018) Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm. J Invest Dermatol 138:1529–1538. https://doi.org/10.1016/j.jid.2018.01.028

    CAS  Article  PubMed  Google Scholar 

  16. 16.

    Arslan Tuncer S, Akilotu B, Toraman S (2019) A deep learning-based decision support system for diagnosis of OSAS using PTT signals. Med Hypotheses 127:15–22. https://doi.org/10.1016/j.mehy.2019.03.026

    Article  PubMed  Google Scholar 

  17. 17.

    Kamel M, Belkassim S, Mendonca AM, Campilho A, Ieee; Ieee; Ieee I (2001) A neural network approach for the automatic detection of microaneurysms in retinal angiograms. Ijcnn'01: International Joint Conference on Neural Networks 1–4, Proceedings: 2695–2699

  18. 18.

    Hafez M, Azeem SA, Ieee; Ieee; Ieee I (2002) Using adaptive edge technique for detecting microaneurysms in fluorescein angiograms of the ocular fundus. 11th Ieee Mediterranean Electrotechnical Conference, Proceedings: 479–483

  19. 19.

    Tavakoli M, Shahri RP, Pourreza H, Mehdizadeh A, Banaee T, Toosi MHB (2013) A complementary method for automated detection of microaneurysms in fluorescein angiography fundus images to assess diabetic retinopathy. Pattern Recogn 46:2740–2753. https://doi.org/10.1016/j.patcog.2013.03.011

    Article  Google Scholar 

  20. 20.

    Frame AJ, Undrill PE, Cree MJ, Olson JA, McHardy KC, Sharp PF, Forrester JV (1998) A comparison of computer based classification methods applied to the detection of microaneurysms in ophthalmic fluorescein angiograms. Comput Biol Med 28:225–238. https://doi.org/10.1016/s0010-4825(98)00011-0

    CAS  Article  PubMed  Google Scholar 

  21. 21.

    Tan JH, Fujita H, Sivaprasad S, Bhandary SV, Rao AK, Kuang CC, Acharya UR (2017) Automated segmentation of exudates, haemorrhages, microaneurysms using single convolutional neural network. Inf Sci 420:S0020025517308927

    Article  Google Scholar 

  22. 22.

    Lam C, Yu C, Huang L, Rubin D (2018) Retinal lesion detection with deep learning using image patches. Invest Ophthalmol Vis Sci 59:590–596. https://doi.org/10.1167/iovs.17-22721

    Article  PubMed  PubMed Central  Google Scholar 

  23. 23.

    Lu W, Tong Y, Yu Y, Xing Y, Chen C, Shen Y (2018) Deep learning-based automated classification of multi-categorical abnormalities from optical coherence tomography images. Transl Vis Sci Technol 7:41. https://doi.org/10.1167/tvst.7.6.41

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  24. 24.

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J, Kim R, Raman R, Nelson PC, Mega JL, Webster DR (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama 316:2402–2410. https://doi.org/10.1001/jama.2016.17216

    Article  PubMed  Google Scholar 

  25. 25.

    Cao W, Czarnek N, Shan J, Li L (2018) Microaneurysm detection using principal component analysis and machine learning methods. IEEE Trans Nanobiosci 17:191–198. https://doi.org/10.1109/tnb.2018.2840084

    Article  Google Scholar 

  26. 26.

    Khojasteh P, Aliahmad B, Kumar DK (2018) Fundus images analysis using deep features for detection of exudates, hemorrhages and microaneurysms. BMC Ophthalmol 18:288. https://doi.org/10.1186/s12886-018-0954-4

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work was financially supported by Zhejiang Provincial Key Research and Development Plan (grant number 2019C03020), the Natural Science Foundation of China (grant number 81670888), and the Natural Science Foundation of China (grant number 81870635).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Juan Ye.

Ethics declarations

This study was conducted in compliance with the principles of the Declaration of Helsinki and was approved by the Ethics Committees of the Second Affiliated Hospital of Zhejiang University School of Medicine. Written informed consents were obtained from all subjects for the publication of this study and accompanying images.

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Pan, X., Jin, K., Cao, J. et al. Multi-label classification of retinal lesions in diabetic retinopathy for automatic analysis of fundus fluorescein angiography based on deep learning. Graefes Arch Clin Exp Ophthalmol 258, 779–785 (2020). https://doi.org/10.1007/s00417-019-04575-w

Download citation

Keywords

  • Diabetic retinopathy
  • Fundus fluorescein angiography
  • Deep learning
  • Multi-label classification