Reproducible Research Using Biomodels
Like other types of computational research, modeling and simulation of biological processes (biomodels) is still largely communicated without sufficient detail to allow independent reproduction of results. But reproducibility in this area of research could easily be achieved by making use of existing resources, such as supplying models in standard formats and depositing code, models, and results in public repositories.
KeywordsReproducibility Biomodels Standards
Reproducibility is at the core of the scientific process. A basic aspect of science is the ability to establish reproducible effects. Karl Popper’s notion of falsifiability depends on reproducibility: a theory is falsified when reproducible effects refute it (Popper 1959). This means that results of scientific investigations must be reproducible or else support for “discoveries” becomes discredited (e.g., see Maddox et al. 1988). Therefore, scientific reports must describe experiments in sufficient detail to allow other researchers to reproduce them. A recent survey suggests that there is a problem with lack of reproducibility in a large proportion of scientific articles (Baker 2016).
This reproducibility crisis has been highlighted mainly for experimental research. At first glance, it could appear that computational research would not suffer from such problems since, after all, computers follow specific sets of instructions (programs) and thus can be run in a reproducible manner. This idea was formalized in the 1990s by Claerbout and Karrenbach (1992) who described how electronic publications could easily be made into reproducible publications. For example, they could allow the reader to re-execute analyses and plots directly from the data through an appropriately associated program. These authors already highlight, though, that a critical issue underlying computational reproducibility is that the programs used should be open source in order to be available to all readers. Claerbout and Karrenbach were essentially optimistic in how computational resources were going to make publications more reproducible, writing that “With workstations becoming widespread and software available, the burdens imposed on the author to create reproducible results are little more than the task of filing everything systematically” (Claerbout and Karrenbach 1992). Unfortunately this optimism did not materialize.
It is now widely accepted that the problem with reproducibility extends also to computational research (Mesirov 2010; Peng 2011; Stodden et al. 2016). And while some journals have adopted a few measures to minimize this problem (Greenbaum et al. 2017; Guerreiro 2017; Loew et al. 2015; Peng 2009), a large proportion of articles describing computational research are still hard or impossible to reproduce (Hothorn et al. 2009; Hothorn and Leisch 2011; Hübner et al. 2011; Stodden et al. 2018).
2 Which Reproducibility?
reproducibility of methods requires one to be able to exactly reproduce the results using the same methods on the same data;
reproducibility of results requires one to obtain similar results in an independent study applying similar procedures;
reproducibility of inferences requires the same conclusions to be reached in an independent replication potentially following a different methodology.
A subset of computational research concerns the use of dynamic models of biological systems (‘biomodels’). This includes a wide range of models, from those representing basic physicochemical phenomena, such as ligand-receptor binding, all the way to models of disease transmission in populations or of entire ecosystems. Models of biochemical reaction networks or pathways are perhaps the most widely represented in this class. Computational biomodels have been reported since the dawn of computers (e.g., Chance et al. 1960) and are now increasingly used to help understand phenomena and make predictions, as can be witnessed by reading the pages of this journal and many others.
Research with biomodels may be better equipped to deal with reproducibility than other types of computational research. A number of researchers have agreed on various standards and principles that promote reproducibility. Most notorious of all is the systems biology markup language (SBML, Hucka et al. 2003) a specification for biomodels that many software packages can read and write, thus promoting reproducibility [see also CellML (Lloyd et al. 2004), NeuroML (Gleeson et al. 2010), PharmML (Swat et al. 2015)]. Standards have also been proposed for model diagrams [SBGN (Novère et al. 2009)], simulation specifications [SED-ML (Waltemath et al. 2011)], and data [SBRML (Dada et al. 2010)]. Finally, there are also minimal information recommendations for publishing models [MIRIAM (Novère et al. 2005)] and simulations [MIASE (Waltemath et al. 2011)] that, if followed, assure a good level of reproducibility — essentially formalizing the process of “filing everything systematically” advocated by Claerbout and Karrenbach (1992).
SBML allows biomodels to be specified in a manner that describes the biology and mathematics without prescribing the algorithms to be applied. An SBML file describes the transformations (“reactions”) that variables (“species”) can undergo; the kinetics of the transformations are well specified with all necessary constants, and the initial state of the system is also included. In essence an SBML file contains all that is needed for software to construct and solve the equations. Because the algorithms are not prescribed, this allows SBML models to be used in different contexts and analyzed with different formalisms in addition to those used in the original research.
With all the standards mentioned above, computational results obtained with biomodels can easily be made reproducible. Publishing the biomodels as SBML files, either as attached supplementary material or by inclusion in a database like BioModels (Chelliah et al. 2015), satisfies reproducibility of results because it makes the model immediately available for simulation with a range of software [e.g., COPASI (Hoops et al. 2006), VCell (Moraru et al. 2008), and many others]. Then, by using the same software as the authors used originally, even reproducibility of methods is achieved. Because SBML does not prescribe the mathematical formalism, in some circumstances it can also facilitate reproducibility of inferences. For example, conclusions may have been obtained from an analysis using the linear noise approximation (e.g., Pahle et al. 2012), and others may reproduce the same conclusions using the Gillespie stochastic simulation algorithm (Gillespie 1976). This is possible because both methods are implemented in software packages that read SBML.
4 Publication of Electronic Materials
The most basic aspect to promote reproducibility of research using biomodels is to publish the electronic materials used. This includes any programs used, the input data (usually constants and initial conditions), and results. Where should these materials be published? There are a number of options in current use: public repositories; journal website as supplementary materials; author’s Web site; supplied by author upon reader’s request—listed in decreasing order of utility to the research community.
The choice of supplying materials only when requested is not practical, and there is plenty of evidence that too frequently it is not honored by authors (Stodden et al. 2018); additionally those materials become inevitably lost as authors retire or die. Any results relying on materials “supplied upon request” should be considered non-reproducible, and journal editors should simply not allow this practice.
Publication of materials in authors’ Web sites is only minimally better: for a short while the materials are indeed immediately available to all, but dead links appear at a fast pace. This is due to frequent website redesigns, authors moving to other institutions, retirement, etc.
Publication of materials in public repositories and in journal Web sites (as supplementary materials) is much better because the materials are immediately available and likely to be findable for a longer time. Both options have some advantages over the other, and it is unclear which of the two may be better. Thus, it is recommended that authors follow both whenever possible.
Several public repositories are conveniently available and are generally being used by a growing number of authors. For code, the most popular are GitHub (https://www.github.com) and CRAN (for R programs, https://cran.r-project.org/), though there are several are other options. For models, the most widely used repository is the BioModels database (https://www.ebi.ac.uk/biomodels-main/), which has the advantage of having curators that ensure the models do indeed produce the results that are described in the publications (Le Novère et al. 2006). In the many cases when this is not true, they contact authors and correct the issues. Thus, submission of models to BioModels already ensures a major verification of reproducibility of methods (Chelliah et al. 2015). For data of any kind (which could include programs and models), other repositories could be used: Zenodo (https://zenodo.org/), Dryad (https://datadryad.org//), and FigShare (https://figshare.com/), all of which issue digital object identifiers (DOI) for data sets.
5 Some Recommendations
Whenever possible use existing peer-reviewed, actively maintained and open-source software to create and analyze models. That way the algorithms and their implementation have already been reviewed, are available to all readers, and most likely will make step 3 trivial. Remember to cite the software and mention the version number used.
If the research required specially written software, deposit the code in a public repository or at a minimum include it as supplementary material in the manuscript. This includes code that may require proprietary software (such as MATLAB, Comsol); your programs need to be published!
Whenever possible, encode the model in an accepted standard (SBML, CellML, etc.), include it as supplementary material, and submit it to a repository.
If it is not possible to encode the model in a standard, then include the full set of equations, parameter values, and initial conditions in the manuscript (at least in a supplement). Make sure that all algorithms used are specified unequivocally, as well as the software used (including version number).
Publish the numerical results as data files, either in a repository, or as supplemental files. (Note that the generic repositories mentioned above allow very large data sets to be deposited.)
Reproducibility is clearly an important aspect of science for both experimental and computational researches. Research using biomodels should be communicated in ways that make it reproducible too. A few actions can be taken that will greatly facilitate this objective. Scientific journals should not publish non-reproducible research and thus should promote, or even enforce, such actions.
Funding was provided by National Institute of General Medical Sciences (Grant No. GM080219).
- Chance B, Garfinkel D, Higgins J, Hess B (1960) Metabolic control mechanisms. V. A solution for the equations representing interaction between glycolysis and respiration in ascites tumor cells. J Biol Chem 235(8):2426–2439Google Scholar
- Chelliah V, Juty N, Ajmera I, Ali R, Dumousseau M, Glont M, Hucka M, Jalowicki G, Keating S, Knight-Schrijver V, Lloret-Villas A, Natarajan KN, Pettit JB, Rodriguez N, Schubert M, Wimalaratne SM, Zhao Y, Hermjakob H, Le Novère N, Laibe C (2015) BioModels: ten-year anniversary. Nucleic Acids Res 43:D542–548. https://doi.org/10.1093/nar/gku1181 CrossRefGoogle Scholar
- Claerbout JF, Karrenbach M (1992) Electronic documents give reproducible research a new meaning. Soc Explor Geophys. https://doi.org/10.1190/1.1822162
- Gleeson P, Crook S, Cannon RC, Hines ML, Billings GO, Farinella M, Morse TM, Davison AP, Ray S, Bhalla US, Barnes SR, Dimitrova YD, Silver RA (2010) NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Comput Biol 6(6):e1000815. https://doi.org/10.1371/journal.pcbi.1000815 MathSciNetCrossRefGoogle Scholar
- Guerreiro M (2017) Forking software used in eLife papers to GitHub . https://elifesciences.org/inside-elife/dbcb6949/forking-software-used-in-elife-papers-to-github
- Hucka M, Finney A, Sauro HM, Bolouri H, Doyle JC, Kitano H, Arkin AP, Bornstein BJ, Bray D, Cornish-Bowden A, Cuellar AA, Dronov S, Gilles ED, Ginkel M, Gor V, Goryanin II, Hedley WJ, Hodgman TC, Hofmeyr JH, Hunter PJ, Juty NS, Kasberger JL, Kremling A, Kummer U, Le Novère N, Loew LM, Lucio D, Mendes P, Minch E, Mjolsness ED, Nakayama Y, Nelson MR, Nielsen PF, Sakurada T, Schaff JC, Shapiro BE, Shimizu TS, Spence HD, Stelling J, Takahashi K, Tomita M, Wagner J, Wang J (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4):524–531. https://doi.org/10.1093/bioinformatics/btg015 CrossRefGoogle Scholar
- Le Novère N, Bornstein B, Broicher A, Courtot M, Donizelli M, Dharuri H, Li L, Sauro H, Schilstra M, Shapiro B, Snoep JL, Hucka M (2006) BioModels database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res 34(Database issue):D689–91 10.1093/nar/gkj092CrossRefGoogle Scholar
- Le Novère N, Finney A, Hucka M, Bhalla US, Campagne F, Collado-Vides J, Crampin EJ, Halstead M, Klipp E, Mendes P, Nielsen P, Sauro H, Shapiro B, Snoep JL, Spence HD, Wanner BL (2005) Minimum information requested in the annotation of biochemical models (MIRIAM). Natute Biotechnol 23(12):1509–15. https://doi.org/10.1038/nbt1156 CrossRefGoogle Scholar
- Le Novère N, Hucka M, Mi H, Moodie S, Schreiber F, Sorokin A, Demir E, Wegner K, Aladjem MI, Wimalaratne SM, Bergman FT, Gauges R, Ghazal P, Kawaji H, Li L, Matsuoka Y, Villéger A, Boyd SE, Calzone L, Courtot M, Dogrusoz U, Freeman TC, Funahashi A, Ghosh S, Jouraku A, Kim S, Kolpakov F, Luna A, Sahle S, Schmidt E, Watterson S, Wu G, Goryanin I, Kell DB, Sander C, Sauro H, Snoep JL, Kohn K, Kitano H (2009) The systems biology graphical notation. Nat Biotechnol 27(8):735–741. https://doi.org/10.1038/nbt.1558 CrossRefGoogle Scholar
- Swat M, Moodie S, Wimalaratne S, Kristensen N, Lavielle M, Mari A, Magni P, Smith M, Bizzotto R, Pasotti L, Mezzalana E, Comets E, Sarr C, Terranova N, Blaudez E, Chan P, Chard J, Chatel K, Chenel M, Edwards D, Franklin C, Giorgino T, Glont M, Girard P, Grenon P, Harling K, Hooker A, Kaye R, Keizer R, Kloft C, Kok J, Kokash N, Laibe C, Laveille C, Lestini G, Mentré F, Munafo A, Nordgren R, Nyberg H, Parra-Guillen Z, Plan E, Ribba B, Smith G, Trocóniz I, Yvon F, Milligan P, Harnisch L, Karlsson M, Hermjakob H, Le Novère N (2015) Pharmacometrics markup language (PharmML): opening new perspectives for model exchange in drug development: PharmML - pharmacometrics markup language. CPT: Pharmacometr Syst Pharmacol 4(6):316–319. https://doi.org/10.1002/psp4.57 CrossRefGoogle Scholar
- Waltemath D, Adams R, Beard DA, Bergmann FT, Bhalla US, Britten R, Chelliah V, Cooling MT, Cooper J, Crampin EJ, Garny A, Hoops S, Hucka M, Hunter P, Klipp E, Laibe C, Miller AK, Moraru I, Nickerson D, Nielsen P, Nikolski M, Sahle S, Sauro HM, Schmidt H, Snoep JL, Tolle D, Wolkenhauer O, Le Novère N (2011) Minimum information about a simulation experiment (MIASE). PLoS Comput Biol 7(4):e1001122. https://doi.org/10.1371/journal.pcbi.1001122 CrossRefGoogle Scholar
- Waltemath D, Adams R, Bergmann FT, Hucka M, Kolpakov F, Miller AK, Moraru II, Nickerson D, Sahle S, Snoep JL, Le Novère N (2011) Reproducible computational biology experiments with SED-ML: the simulation experiment description markup language. BMC Syst Biol 5(1):198. https://doi.org/10.1186/1752-0509-5-198 CrossRefGoogle Scholar