Abstract
Data and software are immaterial. Scientists in computer science hence have the unique chance to let other scientists easily reproduce their findings. Similarly, and with the same ease, the organization of shared tasks, i.e., the collaborative search for new algorithms given a predefined problem, is possible. Experience shows that the potential of reproducibility is hardly tapped in either case. Based on this observation, and driven by the ambitious goal to find the best solutions for certain problems in our research field, we have been developing the TIRA Integrated Research Architecture. Within TIRA, the reproducibility requirement got top priority right from the start. This chapter introduces the platform, its design requirements, its workflows from both the participants’ and the organizers’ perspectives, alongside a report on user experience and usage scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Claerbout J, Karrenbach M (1992) Electronic documents give reproducible research a new meaning, pp 601–604. https://doi.org/10.1190/1.1822162
Donoho D, Maleki A, Rahman I, Shahram M, Stodden V (2009) Reproducible research in computational harmonic analysis. Comput Sci Eng 11:8–18. https://doi.org/10.1109/MCSE.2009.15
Fokkens A, van Erp M, Postma M, Pedersen T, Vossen P, Freire N (2013) Offspring from reproduction problems: What replication failure teaches us. In: Proceedings of the 51st annual meeting of the association for computational linguistics (Long papers), vol 1. Association for Computational Linguistics, Sofia, pp 1691–1701. http://www.aclweb.org/anthology/P13-1166
Freire J, Fuhr N, Rauber A (2016) Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Rep. 6(1):108–159. https://doi.org/10.4230/DagRep.6.1.108
Gollub T, Stein B, Burrows S (2012a) Ousting Ivory tower research: towards a web framework for providing experiments as a service. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, pp 1125–1126. https://doi.org/10.1145/2348283.2348501
Gollub T, Stein B, Burrows S, Hoppe D (2012b) TIRA: configuring, executing, and disseminating information retrieval experiments. In: Tjoa A, Liddle S, Schewe KD, Zhou X (eds) 9th international workshop on text-based information retrieval (TIR 2012) at DEXA. IEEE, Los Alamitos, pp 151–155. https://doi.org/10.1109/DEXA.2012.55
Gollub T, Potthast M, Beyer A, Busse M, Rangel Pardo F, Rosso P, Stamatatos E, Stenno B (2013) Recent trends in digital text forensics and its evaluation—plagiarism detection, author identification, and author profiling. In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) Information access evaluation meets multilinguality, multimodality, and visualization. Proceedings of the fourth international conference of the clef initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138, Springer, Heidelberg, pp 282–302
Hanbury A, Müller H, Balog K, Brodt T, Cormack G, Eggel I, Gollub T, Hopfgartner F, Kalpathy-Cramer J, Kando N, Krithara A, Lin J, Mercer S, Potthast M (2015) Evaluation-as-a-service: overview and outlook. http://arxiv.org/abs/1512.07454
Pedersen T (2008) Empiricism is not a matter of faith. Comput Linguist 34(3):465–470 https://doi.org/10.1162/coli.2008.34.3.465
Plesser HE (2018) Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in Neuroinformatics 11:76. https://doi.org/10.3389/fninf.2017.00076
Potthast M, Hagen M, Stein B, Graßegger J, Michel M, Tippmann M, Welsch C (2012) ChatNoir: A search engine for the ClueWeb09 corpus. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, p 1004. https://doi.org/10.1145/2348283.2348429
Potthast M, Gollub T, Hagen M, Tippmann M, Kiesel J, Rosso P, Stamatatos E, Stein B (2013) Overview of the 5th international competition on plagiarism detection. In: Forner P, Navigli R, Tufis D (eds) Working notes papers of the CLEF 2013 evaluation labs. http://www.clef-initiative.eu/publication/working-notes
Potthast M, Gollub T, Rangel Pardo F, Rosso P, Stamatatos E, Stein B (2014) Improving the reproducibility of PAN’s shared tasks: plagiarism detection, author identification, and author profiling. In: Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E (eds) Information access evaluation—multilinguality, multimodality, and interaction. Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014). Lecture notes in computer science (LNCS), vol 8685, Springer, Heidelberg, pp 268–299
Potthast M, Göring S, Rosso P, Stein B (2015) Towards data submissions for shared tasks: first experiences for the task of text alignment. In: Working notes papers of the CLEF 2015 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings. http://www.clef-initiative.eu/publication/working-notes
Potthast M, Hagen M, Stein B (2016) Author obfuscation: attacking the state of the art in authorship verification. In: Working notes papers of the CLEF 2016 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings, vol 1609. http://ceur-ws.org/Vol-1609/
Stodden V (2010) The scientific method in practice: reproducibility in the computational sciences. Tech. Rep. MIT Sloan Research Paper No. 4773-10. https://doi.org/10.2139/ssrn.1550193
Zeman D, Popel M, Straka M, Hajic J, Nivre J, Ginter F, Luotolahti J, Pyysalo S, Petrov S, Potthast M, Tyers F, Badmaeva E, Gokirmak M, Nedoluzhko A, Cinkova S, Hajic jr J, Hlavacova J, Kettnerová V, Uresova Z, Kanerva J, Ojala S, Missilä A, Manning C, Schuster S, Reddy S, Taji D, Habash N, Leung H, de Marneffe MC, Sanguinetti M, Simi M, Kanayama H, de Paiva V, Droganova K, Martínez Alonso H, Çöltekin Ç, Sulubacak U, Uszkoreit H, Macketanz V, Burchardt A, Harris K, Marheinecke K, Rehm G, Kayadelen T, Attia M, Elkahky A, Yu Z, Pitler E, Lertpradit S, Mandl M, Kirchner J, Fernandez Alcalde H, Strnadová J, Banerjee E, Manurung R, Stella A, Shimada A, Kwak S, Mendonca G, Lando T, Nitisaroj R, Li J (2017) CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. In: Proceedings of the CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. Association for Computational Linguistics, pp 1–19. https://doi.org/10.18653/v1/K17-3001
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Potthast, M., Gollub, T., Wiegmann, M., Stein, B. (2019). TIRA Integrated Research Architecture. In: Ferro, N., Peters, C. (eds) Information Retrieval Evaluation in a Changing World. The Information Retrieval Series, vol 41. Springer, Cham. https://doi.org/10.1007/978-3-030-22948-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-22948-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-22947-4
Online ISBN: 978-3-030-22948-1
eBook Packages: Computer ScienceComputer Science (R0)