Advertisement

TIRA Integrated Research Architecture

  • Martin PotthastEmail author
  • Tim Gollub
  • Matti Wiegmann
  • Benno Stein
Chapter
Part of the The Information Retrieval Series book series (INRE, volume 41)

Abstract

Data and software are immaterial. Scientists in computer science hence have the unique chance to let other scientists easily reproduce their findings. Similarly, and with the same ease, the organization of shared tasks, i.e., the collaborative search for new algorithms given a predefined problem, is possible. Experience shows that the potential of reproducibility is hardly tapped in either case. Based on this observation, and driven by the ambitious goal to find the best solutions for certain problems in our research field, we have been developing the TIRA Integrated Research Architecture. Within TIRA, the reproducibility requirement got top priority right from the start. This chapter introduces the platform, its design requirements, its workflows from both the participants’ and the organizers’ perspectives, alongside a report on user experience and usage scenarios.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Claerbout J, Karrenbach M (1992) Electronic documents give reproducible research a new meaning, pp 601–604. https://doi.org/10.1190/1.1822162
  2. Donoho D, Maleki A, Rahman I, Shahram M, Stodden V (2009) Reproducible research in computational harmonic analysis. Comput Sci Eng 11:8–18.  https://doi.org/10.1109/MCSE.2009.15 CrossRefGoogle Scholar
  3. Fokkens A, van Erp M, Postma M, Pedersen T, Vossen P, Freire N (2013) Offspring from reproduction problems: What replication failure teaches us. In: Proceedings of the 51st annual meeting of the association for computational linguistics (Long papers), vol 1. Association for Computational Linguistics, Sofia, pp 1691–1701. http://www.aclweb.org/anthology/P13-1166
  4. Freire J, Fuhr N, Rauber A (2016) Reproducibility of data-oriented experiments in e-science (dagstuhl seminar 16041). Dagstuhl Rep. 6(1):108–159.  https://doi.org/10.4230/DagRep.6.1.108 Google Scholar
  5. Gollub T, Stein B, Burrows S (2012a) Ousting Ivory tower research: towards a web framework for providing experiments as a service. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, pp 1125–1126. https://doi.org/10.1145/2348283.2348501 Google Scholar
  6. Gollub T, Stein B, Burrows S, Hoppe D (2012b) TIRA: configuring, executing, and disseminating information retrieval experiments. In: Tjoa A, Liddle S, Schewe KD, Zhou X (eds) 9th international workshop on text-based information retrieval (TIR 2012) at DEXA. IEEE, Los Alamitos, pp 151–155.  https://doi.org/10.1109/DEXA.2012.55
  7. Gollub T, Potthast M, Beyer A, Busse M, Rangel Pardo F, Rosso P, Stamatatos E, Stenno B (2013) Recent trends in digital text forensics and its evaluation—plagiarism detection, author identification, and author profiling. In: Forner P, Müller H, Paredes R, Rosso P, Stein B (eds) Information access evaluation meets multilinguality, multimodality, and visualization. Proceedings of the fourth international conference of the clef initiative (CLEF 2013). Lecture notes in computer science (LNCS), vol 8138, Springer, Heidelberg, pp 282–302Google Scholar
  8. Hanbury A, Müller H, Balog K, Brodt T, Cormack G, Eggel I, Gollub T, Hopfgartner F, Kalpathy-Cramer J, Kando N, Krithara A, Lin J, Mercer S, Potthast M (2015) Evaluation-as-a-service: overview and outlook. http://arxiv.org/abs/1512.07454
  9. Pedersen T (2008) Empiricism is not a matter of faith. Comput Linguist 34(3):465–470  https://doi.org/10.1162/coli.2008.34.3.465 CrossRefGoogle Scholar
  10. Plesser HE (2018) Reproducibility vs. replicability: a brief history of a confused terminology. Frontiers in Neuroinformatics 11:76.  https://doi.org/10.3389/fninf.2017.00076
  11. Potthast M, Hagen M, Stein B, Graßegger J, Michel M, Tippmann M, Welsch C (2012) ChatNoir: A search engine for the ClueWeb09 corpus. In: Hersh B, Callan J, Maarek Y, Sanderson M (eds) 35th international ACM conference on research and development in information retrieval (SIGIR 2012). ACM, New York, p 1004. https://doi.org/10.1145/2348283.2348429 Google Scholar
  12. Potthast M, Gollub T, Hagen M, Tippmann M, Kiesel J, Rosso P, Stamatatos E, Stein B (2013) Overview of the 5th international competition on plagiarism detection. In: Forner P, Navigli R, Tufis D (eds) Working notes papers of the CLEF 2013 evaluation labs. http://www.clef-initiative.eu/publication/working-notes
  13. Potthast M, Gollub T, Rangel Pardo F, Rosso P, Stamatatos E, Stein B (2014) Improving the reproducibility of PAN’s shared tasks: plagiarism detection, author identification, and author profiling. In: Kanoulas E, Lupu M, Clough P, Sanderson M, Hall M, Hanbury A, Toms E (eds) Information access evaluation—multilinguality, multimodality, and interaction. Proceedings of the fifth international conference of the CLEF initiative (CLEF 2014). Lecture notes in computer science (LNCS), vol 8685, Springer, Heidelberg, pp 268–299Google Scholar
  14. Potthast M, Göring S, Rosso P, Stein B (2015) Towards data submissions for shared tasks: first experiences for the task of text alignment. In: Working notes papers of the CLEF 2015 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings. http://www.clef-initiative.eu/publication/working-notes
  15. Potthast M, Hagen M, Stein B (2016) Author obfuscation: attacking the state of the art in authorship verification. In: Working notes papers of the CLEF 2016 evaluation labs, CLEF and CEUR-WS.org, CEUR workshop proceedings, vol 1609. http://ceur-ws.org/Vol-1609/
  16. Stodden V (2010) The scientific method in practice: reproducibility in the computational sciences. Tech. Rep. MIT Sloan Research Paper No. 4773-10.  https://doi.org/10.2139/ssrn.1550193
  17. Zeman D, Popel M, Straka M, Hajic J, Nivre J, Ginter F, Luotolahti J, Pyysalo S, Petrov S, Potthast M, Tyers F, Badmaeva E, Gokirmak M, Nedoluzhko A, Cinkova S, Hajic jr J, Hlavacova J, Kettnerová V, Uresova Z, Kanerva J, Ojala S, Missilä A, Manning C, Schuster S, Reddy S, Taji D, Habash N, Leung H, de Marneffe MC, Sanguinetti M, Simi M, Kanayama H, de Paiva V, Droganova K, Martínez Alonso H, Çöltekin Ç, Sulubacak U, Uszkoreit H, Macketanz V, Burchardt A, Harris K, Marheinecke K, Rehm G, Kayadelen T, Attia M, Elkahky A, Yu Z, Pitler E, Lertpradit S, Mandl M, Kirchner J, Fernandez Alcalde H, Strnadová J, Banerjee E, Manurung R, Stella A, Shimada A, Kwak S, Mendonca G, Lando T, Nitisaroj R, Li J (2017) CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. In: Proceedings of the CoNLL 2017 shared task: multilingual parsing from raw text to universal dependencies. Association for Computational Linguistics, pp 1–19. https://doi.org/10.18653/v1/K17-3001

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Martin Potthast
    • 1
    Email author
  • Tim Gollub
    • 2
  • Matti Wiegmann
    • 3
    • 4
  • Benno Stein
    • 2
  1. 1.Leipzig UniversityLeipzigGermany
  2. 2.Bauhaus-Universität WeimarWeimarGermany
  3. 3.Bauhaus-Universität WeimarWeimarGermany
  4. 4.German Aerospace Center (DLR)JenaGermany

Personalised recommendations