Current Diabetes Reports

, 17:106 | Cite as

Crowdsourcing and Automated Retinal Image Analysis for Diabetic Retinopathy

  • Lucy I. Mudie
  • Xueyang Wang
  • David S. Friedman
  • Christopher J. Brady
Microvascular Complications—Retinopathy (JK Sun and PS Silva, Section Editors)
Part of the following topical collections:
  1. Topical Collection on Microvascular Complications—Retinopathy


Purpose of Review

As the number of people with diabetic retinopathy (DR) in the USA is expected to increase threefold by 2050, the need to reduce health care costs associated with screening for this treatable disease is ever present. Crowdsourcing and automated retinal image analysis (ARIA) are two areas where new technology has been applied to reduce costs in screening for DR. This paper reviews the current literature surrounding these new technologies.

Recent Findings

Crowdsourcing has high sensitivity for normal vs abnormal images; however, when multiple categories for severity of DR are added, specificity is reduced. ARIAs have higher sensitivity and specificity, and some commercial ARIA programs are already in use. Deep learning enhanced ARIAs appear to offer even more improvement in ARIA grading accuracy.


The utilization of crowdsourcing and ARIAs may be a key to reducing the time and cost burden of processing images from DR screening.


Crowdsourcing Diabetic retinopathy Amazon Mechanical Turk Automated retinal image analysis Telemedicine 


Funding Information

This publication was made possible by the Johns Hopkins Institute for Clinical and Translational Research (ICTR) which is funded in part by Grant Number KL2TR001077 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research, and by the Johns Hopkins-University of Maryland Diabetes Research Center Pilot and Feasibility Award under National Institute of Diabetes and Digestion P30DK079637 (Hussain, PI).

Compliance with Ethical Standards

Conflict of Interest

Lucy I. Mudie, Xueyang Wang, David S. Friedman, and Christopher J. Brady declare that they have no conflict of interest.

Human and Animal Rights and Informed Consent

This manuscript does not include any studies with human or animal subjects performed by any of the authors. The Johns Hopkins University School of Medicine Institutional Review Board (IRB) has deemed the crowdsourcing work of the authors IRB-exempt as non-human subjects research. Disclaimer


Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the Johns Hopkins ICTR, NCATS or NIH.


Papers of particular interest, published recently, have been highlighted as: ••Of major importance

  1. 1.
    Russell S, Norvig P. Artificial intelligence: a modern approach. 3rd ed. Essex: Pearson; 2009.Google Scholar
  2. 2.
    Brabham DC, Ribisl KM, Kirchner TR, Bernhardt JM. Crowdsourcing applications for public health. Am J Prev Med. 2014;46:179–87.CrossRefPubMedGoogle Scholar
  3. 3.
    Brabham DC. Crowdsourcing. Massachusetts: MIT Press; 2013.Google Scholar
  4. 4.
    Brinkmann BH, Wagenaar J, Abbot D, Adkins P, Bosshard SC, Chen M, et al. Crowdsourcing reproducible seizure forecasting in human and canine epilepsy. Brain. 2016;139(Pt 6):1713–22.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    •• Brady CJ, Villanti AC, Pearson JL, et al. Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res. 2014;16(10):e233. This is the first paper describing the use of crowdsourcing specifically for grading images for diabetic retinopathy CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Mitry D, Peto T, Hayat S, et al. Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk Cohort on behalf of the UKBiobank eye and vision consortium. PLoS One. 2013;8(8):e71154.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    •• Mitry D, Zutis K, Dhillon B, Peto T, Hayat S, Khaw KT, et al. The accuracy and reliability of crowdsource annotations of digital retinal images. Transl Vis Sci Technol. 2016;5(5):6. Erratum in: Transl Vis Sci Technol. 2016;5(6):9. This is the most recent evaluation of crowdsourcing for using in grading retinal images for pathology. It is also the first paper to report using crowdsourcing to annotate retinal images with areas of pathology associated with diabetic retinopathy CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    Mitry D, Peto T, Hayat S, et al. Crowdsourcing as a screening tool to detect clinical features of glaucomatous optic neuropathy from digital photography. PLoS One. 2015;10(2):e0117401.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Wang X, Mudie LI, Baskaran M, Cheng CY, Alward WL, Friedman DS, et al. Crowdsourcing to evaluate fundus photographs for the presence of glaucoma. J Glaucoma. 2017.Google Scholar
  10. 10.
    de Alfaro L, Shavlovsky M. Crowdsourcing quantitative evaluation: algorithms and empirical results. Technical Report UCSC-SOE-14-03, School of Engineering, UC Santa Cruz. Accessed 27 Apr 2017.
  11. 11.
    Katz N, Goldbaum M, Nelson M, et al. An image processing system for automatic retina diagnosis. SPIE. 1988;902:131–7. Scholar
  12. 12.
    Yamamoto S, Yokouchi H. Automatic recognition of color fundus photographs. In: Preston K, Onoe M, editors. Digital processing of biomedical images [internet]. Boston: Springer US; 1976. p. 385–98. Scholar
  13. 13.
    Trucco E, Ruggeri A, Karnowski T, Giancardo L, Chaum E, Hubschman JP, et al. Validating retinal fundus image analysis algorithms: issues and a proposal. Invest Ophthalmol Vis Sci. 2013;54(5):3546–59. Scholar
  14. 14.
    Walton OB 4th, Garoon RB, Weng CY, Gross J, Young AK, Camero KA, et al. Evaluation of automated teleretinal screening program for diabetic retinopathy. JAMA Ophthalmol. 2016;134(2):204–9. Scholar
  15. 15.
    Sim DA, Keane PA, Tufail A, Egan CA, Aiello LP, Silva PS. Automated retinal image analysis for diabetic retinopathy in telemedicine. Curr Diab Rep. 2015;15(3):14. Scholar
  16. 16.
    Tufail A, Rudisill C, Egan C, Kapetanakis VV, Salas-Vega S, Owen CG, et al. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared with human graders. Ophthalmology. 2017;124(3):343–51. Scholar
  17. 17.
    MacGillivray TJ, Cameron JR, Zhang Q, El-Medany A, Mulholland C, Sheng Z, et al. Suitability of UK biobank retinal images for automatic analysis of morphometric properties of the vasculature. PLoS One. 2015;10(5):e0127914. eCollection 2015CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    •• Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402–10. This high-profile paper described the use of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, and has a good description of deep learning for a general medical audience CrossRefPubMedGoogle Scholar
  19. 19.
    Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci. 2016;57(13):5200–6. Scholar
  20. 20.
    Oliveira CM, Cristovao LM, Ribeiro ML, Abreu JR. Improved automated screening of diabetic retinopathy. Ophthalmologica. 2011;226(4):191–7. Scholar
  21. 21.
    Philip S, Fleming AD, Goatman KA, Fonseca S, McNamee P, Scotland GS, et al. The efficacy of automated “disease/no disease” grading for diabetic retinopathy in a systematic screening programme. Br J Ophthalmol. 2007;91(11):1512–7. Scholar
  22. 22.
    Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda S, et al. Automated diabetic retinopathy screening and monitoring using retinal fundus image analysis. J Diabetes Sci Technol. 2016;10(2):254–61. Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Lucy I. Mudie
    • 1
  • Xueyang Wang
    • 1
  • David S. Friedman
    • 1
  • Christopher J. Brady
    • 1
  1. 1.Wilmer Eye InstituteJohns Hopkins University School of MedicineBaltimoreUSA

Personalised recommendations