Computerized labeling tools are often used to systematically record the assessment for fundus images. Carefully designed labeling tools not only save time and enable comprehensive and thorough assessment at clinics, but also economize large-scale data collection processes for the development of automatic algorithms. To realize efficient and thorough fundus assessment, we developed a new labeling tool with novel schemes - stepwise labeling and regional encoding. We have used our tool in a large-scale data annotation project in which 318,376 annotations for 109,885 fundus images were gathered with a total duration of 421 h. We believe that the fundamental concepts in our tool would inspire other data collection processes and annotation procedure in different domains.


Weakly labeled data Data annotations Fundoscopic images 


  1. 1.
    Indian diabetic retinopathy image dataset (IDRiD). Accessed 29 May 2018
  2. 2.
    Kaggle diabetic retinopathy detection competition. Accessed 29 May 2018
  3. 3.
    Carmona, E.J., Rincón, M., García-Feijoó, J., Martínez-de-la Casa, J.M.: Identification of the optic nerve head with genetic algorithms. Artif. Intell. Med. 43(3), 243–259 (2008)CrossRefGoogle Scholar
  4. 4.
    Decencière, E., et al.: Teleophta: machine learning and image processing methods for teleophthalmology. IRBM 34(2), 196–203 (2013)CrossRefGoogle Scholar
  5. 5.
    Decencière, E., et al.: Feedback on a publicly distributed image database: the Messidor database. Image Anal. Stereol. 33(3), 231–234 (2014)CrossRefGoogle Scholar
  6. 6.
    Detry-Morel, M., Zeyen, T., Kestelyn, P., Collignon, J., Goethals, M., Belgian Glaucoma Society: Screening for glaucoma in a general population with the non-mydriatic fundus camera and the frequency doubling perimeter. Eur. J. Ophthalmol. 14(5), 387–393 (2004)CrossRefGoogle Scholar
  7. 7.
    Gargeya, R., Leng, T.: Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124(7), 962–969 (2017)CrossRefGoogle Scholar
  8. 8.
    ETDRS Group, et al.: Grading diabetic retinopathy from stereoscopic color fundus photographsan extension of the modified airlie house classification: ETDRS report number 10. Ophthalmology 98(5), 786–806 (1991)Google Scholar
  9. 9.
    Gulshan, V., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22), 2402–2410 (2016)CrossRefGoogle Scholar
  10. 10.
    Kempen, J.H., et al.: The prevalence of diabetic retinopathy among adults in the United States. Arch. Ophthalmol. (Chicago, Ill.: 1960) 122(4), 552–563 (2004)CrossRefGoogle Scholar
  11. 11.
    Lin, D.Y., Blumenkranz, M.S., Brothers, R.J., Grosvenor, D.M.: The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography1. Am. J. Ophthalmol. 134(2), 204–213 (2002)CrossRefGoogle Scholar
  12. 12.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  13. 13.
    Ting, D.S.W., et al.: Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318(22), 2211–2223 (2017)CrossRefGoogle Scholar
  14. 14.
    Park, S.J., Shin, J.Y., Kim, S., Son, J., Jung, K.H., Park, K.H.: A novel fundus image reading tool for efficient generation of a multi-dimensional categorical image database for machine learning algorithm training. J. Korean Med. Sci. 33(43) (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jaemin Son
    • 1
  • Sangkeun Kim
    • 1
  • Sang Jun Park
    • 2
  • Kyu-Hwan Jung
    • 1
    Email author
  1. 1.VUNO Inc.SeoulKorea
  2. 2.Department of OphthalmologySeoul National University College of Medicine, Seoul National University Bundang HospitalSeongnamKorea

Personalised recommendations