Provenance and Reproducibility
A computational experiment composed by a sequence of steps S created at time T, on environment (hardware and operating system) E, using data D is reproducible if it can be executed with a sequence of steps S′ (modified from or equal to S) at time T′ > T, on environment E′ (potentially different than E), using data D′ that is similar to (or the same as) D with consistent results . Replication is a special case of reproducibility where S′ = S and D′ = D. While there is substantial disagreement on how to define reproducibility , in particular across different domains, in this entry, we focus on computational reproducibility, i.e., reproducibility for computational experiments or processes.
The information needed to reproduce an experiment can be obtained from its provenance: the details of how the experiment was carried out and the results it derived. For computational experiments, provenance can be systematically and transparently...
- 1.Baker M. Muddled meanings hamper efforts to fix reproducibility crisis. Nature News & Comment. 14 Jun 2006 (2016).Google Scholar
- 2.Bonnet P, Manegold S, Bjørling M, Cao W, Gonzalez J, Granados J, Hall N, Idreos S, Ivanova M, Johnson R, Koop D, Kraska T, Müller R, Olteanu D, Papotti P, Reilly C, Tsirogiannis D, Yu C, Freire J, Shasha D. Repeatability and workability evaluation of SIGMOD’2011. ACM SIGMOD Rec. 2011;40(2):45–8.CrossRefGoogle Scholar
- 3.Claerbout J, Karrenbach M. Electronic documents give reproducible research a new meaning. In: Proceedings of the 62nd Annual International Meeting of the Society of Exploration Geophysics; 1992. p. 601–4.Google Scholar
- 4.Collberg C, Proebsting T, Warren AM. Repeatability and benefaction in computer systems research. Technical report. TR 14-04, University of Arizona; 2015.Google Scholar
- 5.Freire J, Bonnet P, Shasha D. Computational reproducibility: state-of-the-art, challenges, and database research opportunities. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data; 2012. p. 593–6.Google Scholar
- 7.Kovacevic J. How to encourage and publish reproducible research. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing; 2007. p. IV-1273–6.Google Scholar