Widely-used public benchmarks are of huge importance to computer vision and machine learning research, especially with the computational resources required to reproduce state of the art results quickly becoming untenable. In medical image computing, the wide variety of image modalities and problem formulations yields a huge task-space for benchmarks to cover, and thus the widespread adoption of standard benchmarks has been slow, and barriers to releasing medical data exacerbate this issue. In this paper, we examine the role that publicly available data has played in MICCAI papers from the past five years. We find that more than half of these papers are based on private data alone, although this proportion seems to be decreasing over time. Additionally, we observed that after controlling for open access publication and the release of code, papers based on public data were cited over 60% more per year than their private-data counterparts. Further, we found that more than 20% of papers using public data did not provide a citation to the dataset or associated manuscript, highlighting the “second-rate” status that data contributions often take compared to theoretical ones. We conclude by making recommendations for MICCAI policies which could help to better incentivise data sharing and move the field toward more efficient and reproducible science.



Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA225435. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.


  1. 1.
    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). Scholar
  2. 2.
    Clark, K., et al.: The cancer imaging archive (tcia): maintaining and operating a public information repository. J. Digit. Imaging 26(6), 1045–1057 (2013)CrossRefGoogle Scholar
  3. 3.
    Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., McGillivray, B.: The citation advantage of linking publications to research data. arXiv preprint arXiv:1907.02565 (2019)
  4. 4.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)Google Scholar
  5. 5.
    Dixon, W.J., Yuen, K.K.: Trimming and winsorization: a review. Statistische Hefte 15(2–3), 157–170 (1974)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Drachen, T., Ellegaard, O., Larsen, A., Dorch, S.: Sharing data increases citations. Liber Q. 26(2) (2016)CrossRefGoogle Scholar
  7. 7.
    Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. CRC Press, Boca Raton (1994)zbMATHGoogle Scholar
  8. 8.
    Erickson, B.J., Korfiatis, P., Akkus, Z., Kline, T.L.: Machine learning for medical imaging. Radiographics 37(2), 505–515 (2017)CrossRefGoogle Scholar
  9. 9.
    Eysenbach, G.: Citation advantage of open access articles. PLoS Biol. 4(5), e157 (2006)CrossRefGoogle Scholar
  10. 10.
    Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)CrossRefGoogle Scholar
  11. 11.
    Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report, Citeseer (2009)Google Scholar
  12. 12.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  13. 13.
    Piwowar, H.A., Vision, T.J.: Data reuse and the open data citation advantage. PeerJ 1, e175 (2013)CrossRefGoogle Scholar
  14. 14.
    Roth, H.R., et al.: DeepOrgan: multi-level deep convolutional networks for automated pancreas segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 556–564. Springer, Cham (2015). Scholar
  15. 15.
    Sekara, V., Deville, P., Ahnert, S.E., Barabási, A.L., Sinatra, R., Lehmann, S.: The chaperone effect in scientific publishing. Proc. Natl. Acad. Sci. 115(50), 12603–12607 (2018)CrossRefGoogle Scholar
  16. 16.
    Thelwall, M., Wilson, P.: Regression for citation data: an evaluation of different methods. J. Informetrics 8(4), 963–971 (2014)CrossRefGoogle Scholar
  17. 17.
    Vandewalle, P.: Code sharing is associated with research impact in image processing. Comput. Sci. Eng. 14(4), 42–47 (2012)CrossRefGoogle Scholar
  18. 18.
    Wilkinson, M.D., et al.: The fair guiding principles for scientific data management and stewardship. Sci. Data 3 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nicholas Heller
    • 1
    Email author
  • Jack Rickman
    • 1
  • Christopher Weight
    • 1
  • Nikolaos Papanikolopoulos
    • 1
  1. 1.University of Minnesota – Twin CitiesMinneapolisUSA

Personalised recommendations