Advertisement

CENTRE@CLEF 2019

  • Nicola Ferro
  • Norbert Fuhr
  • Maria MaistroEmail author
  • Tetsuya Sakai
  • Ian Soboroff
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11438)

Abstract

Reproducibility of experimental results has recently become a primary issue in the scientific community at large, and in the information retrieval community as well, where initiatives and incentives to promote and ease reproducibility are arising. In this context, CENTRE is a joint CLEF/TREC/NTCIR lab which aims at raising the attention on this topic and involving the community in a shared reproducibility exercise. In particular, CENTRE focuses on three objectives, e.g. replicability, reproducibility and generalizability, and for each of them a dedicated task is designed. We expect that CENTRE may impact on the validation of some key achievement in IR, help in designing shared protocols for reproducibility, and improve the understanding on generalization across collections and on the additivity issue.

Notes

Acknowledgments

AMAOS (Advanced Machine Learning for Automatic Omni-Channel Support). Funded by: Innovationsfonden, Denmark.

References

  1. 1.
    Allan, J., et al.: Research frontiers in information retrieval - report from the third strategic workshop on information retrieval in lorne (SWIRL 2018). SIGIR Forum 52(1), 34–90 (2018)CrossRefGoogle Scholar
  2. 2.
    Arguello, J., Crane, M., Diaz, F., Lin, J., Trotman, A.: Report on the SIGIR 2015 workshop on reproducibility, inexplicability, and generalizability of results (RIGOR). SIGIR Forum 49(2), 107–116 (2015)CrossRefGoogle Scholar
  3. 3.
    Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Has adhoc retrieval improved since 1994? In: Allan, J., Aslam, J.A., Sanderson, M., Zhai, C., Zobel, J. (eds.) Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2009), pp. 692–693. ACM Press, New York (2009)Google Scholar
  4. 4.
    Armstrong, T.G., Moffat, A., Webber, W., Zobel, J.: Improvements that don’t add up: ad-hoc retrieval results since 1998. In: Cheung, D.W.L., Song, I.Y., Chu, W.W., Hu, X., Lin, J.J. (eds.) Proceedings of the 18th International Conference on Information and Knowledge Management (CIKM 2009), pp. 601–610. ACM Press, New York (2009)Google Scholar
  5. 5.
    Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.): CLEF 2018 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org) (2018). http://ceur-ws.org/Vol-2125/. ISSN 1613–0073
  6. 6.
    Ferro, N., et al.: Report on ECIR 2016: 38th european conference on information retrieval. SIGIR Forum 50(1), 12–27 (2016)CrossRefGoogle Scholar
  7. 7.
    Ferro, N., et al.: Manifesto from Dagstuhl perspectives workshop 17442 - building a predictive science for performance of information retrieval, recommender systems, and natural language processing applications. Dagstuhl Manifestos, Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany 7(1) (2018)Google Scholar
  8. 8.
    Ferro, N., Fuhr, N., Rauber, A.: Introduction to the special issue on reproducibility in information retrieval: evaluation campaigns, collections, and analyses. ACM J. Data Inf. Qual. (JDIQ) 10(3), 9:1–9:4 (2018)Google Scholar
  9. 9.
    Ferro, N., Fuhr, N., Rauber, A.: Introduction to the special issue on reproducibility in information retrieval: tools and infrastructures. ACM J. Data Inf. Qual. (JDIQ) 10(4), 1–4 (2018)CrossRefGoogle Scholar
  10. 10.
    Ferro, N., Kelly, D.: SIGIR initiative to implement ACM artifact review and badging. SIGIR Forum 52(1), 4–10 (2018)CrossRefGoogle Scholar
  11. 11.
    Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: CENTRE@CLEF2018: overview of the replicability task. In: Cappellato et al. [5]Google Scholar
  12. 12.
    Ferro, N., Maistro, M., Sakai, T., Soboroff, I.: Overview of CENTRE@CLEF 2018: a first tale in the systematic reproducibility realm. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 239–246. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-98932-7_23CrossRefGoogle Scholar
  13. 13.
    Freire, J., Fuhr, N., Rauber, A. (eds.): Report from Dagstuhl Seminar 16041: Reproducibility of Data-Oriented Experiments in e-Science. Dagstuhl Reports, vol. 6, no. 1, Schloss Dagstuhl-Leibniz-Zentrum für Informatik, Germany (2016)Google Scholar
  14. 14.
    Fuhr, N.: Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum 51(3), 32–41 (2017)CrossRefGoogle Scholar
  15. 15.
    Jungwirth, M., Hanbury, A.: Replicating an experiment in cross-lingual information retrieval with explicit semantic analysis. In: Cappellato et al. [5]Google Scholar
  16. 16.
    Kharazmi, S., Scholer, F., Vallet, D., Sanderson, M.: Examining additivity and weak baselines. ACM Trans. Inf. Syst. (TOIS) 34(4), 23:1–23:18 (2016)CrossRefGoogle Scholar
  17. 17.
    Lin, J., et al.: Toward reproducible baselines: the open-source IR reproducibility challenge. In: Ferro, N., et al. (eds.) ECIR 2016. LNCS, vol. 9626, pp. 408–420. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-30671-1_30CrossRefGoogle Scholar
  18. 18.
    Munafò, M.R., et al.: A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021:1–0021:9 (2017)CrossRefGoogle Scholar
  19. 19.
    Zobel, J., Webber, W., Sanderson, M., Moffat, A.: Principles for robust evaluation infrastructure. In: Agosti, M., Ferro, N., Thanos, C. (eds.) Proceedings of the Workshop on Data Infrastructures for Supporting Information Retrieval Evaluation (DESIRE 2011), pp. 3–6. ACM Press, New York (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nicola Ferro
    • 1
  • Norbert Fuhr
    • 2
  • Maria Maistro
    • 1
    • 3
    Email author
  • Tetsuya Sakai
    • 4
  • Ian Soboroff
    • 5
  1. 1.University of PaduaPaduaItaly
  2. 2.University Duisburg-EssenDuisburgGermany
  3. 3.University of CopenhagenCopenhagenDenmark
  4. 4.Waseda UniversityTokyoJapan
  5. 5.National Institute of Standards and Technology (NIST)GaithersburgUSA

Personalised recommendations