Crowdsourcing and Automated Retinal Image Analysis for Diabetic Retinopathy
Purpose of Review
As the number of people with diabetic retinopathy (DR) in the USA is expected to increase threefold by 2050, the need to reduce health care costs associated with screening for this treatable disease is ever present. Crowdsourcing and automated retinal image analysis (ARIA) are two areas where new technology has been applied to reduce costs in screening for DR. This paper reviews the current literature surrounding these new technologies.
Crowdsourcing has high sensitivity for normal vs abnormal images; however, when multiple categories for severity of DR are added, specificity is reduced. ARIAs have higher sensitivity and specificity, and some commercial ARIA programs are already in use. Deep learning enhanced ARIAs appear to offer even more improvement in ARIA grading accuracy.
The utilization of crowdsourcing and ARIAs may be a key to reducing the time and cost burden of processing images from DR screening.
KeywordsCrowdsourcing Diabetic retinopathy Amazon Mechanical Turk Automated retinal image analysis Telemedicine
This publication was made possible by the Johns Hopkins Institute for Clinical and Translational Research (ICTR) which is funded in part by Grant Number KL2TR001077 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research, and by the Johns Hopkins-University of Maryland Diabetes Research Center Pilot and Feasibility Award under National Institute of Diabetes and Digestion P30DK079637 (Hussain, PI).
Compliance with Ethical Standards
Conflict of Interest
Lucy I. Mudie, Xueyang Wang, David S. Friedman, and Christopher J. Brady declare that they have no conflict of interest.
Human and Animal Rights and Informed Consent
This manuscript does not include any studies with human or animal subjects performed by any of the authors. The Johns Hopkins University School of Medicine Institutional Review Board (IRB) has deemed the crowdsourcing work of the authors IRB-exempt as non-human subjects research. Disclaimer
Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the Johns Hopkins ICTR, NCATS or NIH.
Papers of particular interest, published recently, have been highlighted as: ••Of major importance
- 1.Russell S, Norvig P. Artificial intelligence: a modern approach. 3rd ed. Essex: Pearson; 2009.Google Scholar
- 3.Brabham DC. Crowdsourcing. Massachusetts: MIT Press; 2013.Google Scholar
- 5.•• Brady CJ, Villanti AC, Pearson JL, et al. Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res. 2014;16(10):e233. This is the first paper describing the use of crowdsourcing specifically for grading images for diabetic retinopathy CrossRefPubMedPubMedCentralGoogle Scholar
- 7.•• Mitry D, Zutis K, Dhillon B, Peto T, Hayat S, Khaw KT, et al. The accuracy and reliability of crowdsource annotations of digital retinal images. Transl Vis Sci Technol. 2016;5(5):6. Erratum in: Transl Vis Sci Technol. 2016;5(6):9. This is the most recent evaluation of crowdsourcing for using in grading retinal images for pathology. It is also the first paper to report using crowdsourcing to annotate retinal images with areas of pathology associated with diabetic retinopathy CrossRefPubMedPubMedCentralGoogle Scholar
- 9.Wang X, Mudie LI, Baskaran M, Cheng CY, Alward WL, Friedman DS, et al. Crowdsourcing to evaluate fundus photographs for the presence of glaucoma. J Glaucoma. 2017.Google Scholar
- 10.de Alfaro L, Shavlovsky M. Crowdsourcing quantitative evaluation: algorithms and empirical results. Technical Report UCSC-SOE-14-03, School of Engineering, UC Santa Cruz. https://pdfs.semanticscholar.org/cdc8/5dda05559884af5b05f5015288cff5fbaaab.pdf. Accessed 27 Apr 2017.
- 16.Tufail A, Rudisill C, Egan C, Kapetanakis VV, Salas-Vega S, Owen CG, et al. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders. Ophthalmology. 2017;124(3):343–51. https://doi.org/10.1016/j.ophtha.2016.11.014.CrossRefPubMedGoogle Scholar
- 17.MacGillivray TJ, Cameron JR, Zhang Q, El-Medany A, Mulholland C, Sheng Z, et al. Suitability of UK biobank retinal images for automatic analysis of morphometric properties of the vasculature. PLoS One. 2015;10(5):e0127914. https://doi.org/10.1371/journal.pone.0127914. eCollection 2015CrossRefPubMedPubMedCentralGoogle Scholar
- 18.•• Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402–10. https://doi.org/10.1001/jama.2016.17216. This high-profile paper described the use of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, and has a good description of deep learning for a general medical audience CrossRefPubMedGoogle Scholar
- 19.Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57(13):5200–6. https://doi.org/10.1167/iovs.16-19964.CrossRefPubMedGoogle Scholar
- 21.Philip S, Fleming AD, Goatman KA, Fonseca S, McNamee P, Scotland GS, et al. The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol. 2007;91(11):1512–7. https://doi.org/10.1136/bjo.2007.119453.CrossRefPubMedPubMedCentralGoogle Scholar