Abstract
Like other types of computational research, modeling and simulation of biological processes (biomodels) is still largely communicated without sufficient detail to allow independent reproduction of results. But reproducibility in this area of research could easily be achieved by making use of existing resources, such as supplying models in standard formats and depositing code, models, and results in public repositories.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baker M (2016) 1,500 scientists lift the lid on reproducibility. Nature 533(7604):452–454. https://doi.org/10.1038/533452a
Chance B, Garfinkel D, Higgins J, Hess B (1960) Metabolic control mechanisms. V. A solution for the equations representing interaction between glycolysis and respiration in ascites tumor cells. J Biol Chem 235(8):2426–2439
Chelliah V, Juty N, Ajmera I, Ali R, Dumousseau M, Glont M, Hucka M, Jalowicki G, Keating S, Knight-Schrijver V, Lloret-Villas A, Natarajan KN, Pettit JB, Rodriguez N, Schubert M, Wimalaratne SM, Zhao Y, Hermjakob H, Le Novère N, Laibe C (2015) BioModels: ten-year anniversary. Nucleic Acids Res 43:D542–548. https://doi.org/10.1093/nar/gku1181
Claerbout JF, Karrenbach M (1992) Electronic documents give reproducible research a new meaning. Soc Explor Geophys. https://doi.org/10.1190/1.1822162
Dada JO, Spasić I, Paton NW, Mendes P (2010) SBRML: a markup language for associating systems biology data with models. Bioinformatics 26(7):932–938. https://doi.org/10.1093/bioinformatics/btq069
Gillespie DT (1976) A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. J Comput Phys 22:403–434
Gleeson P, Crook S, Cannon RC, Hines ML, Billings GO, Farinella M, Morse TM, Davison AP, Ray S, Bhalla US, Barnes SR, Dimitrova YD, Silver RA (2010) NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Comput Biol 6(6):e1000815. https://doi.org/10.1371/journal.pcbi.1000815
Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Sci Transl Med 8(341):341ps12. https://doi.org/10.1126/scitranslmed.aaf5027
Greenbaum D, Rozowsky J, Stodden V, Gerstein M (2017) Structuring supplemental materials in support of reproducibility. Genome Biol 18:64. https://doi.org/10.1186/s13059-017-1205-3
Guerreiro M (2017) Forking software used in eLife papers to GitHub . https://elifesciences.org/inside-elife/dbcb6949/forking-software-used-in-elife-papers-to-github
Hoops S, Sahle S, Gauges R, Lee C, Pahle J, Simus N, Singhal M, Xu L, Mendes P, Kummer U (2006) COPASI: a COmplex PAthway SImulator. Bioinformatics 22(24):3067–3074. https://doi.org/10.1093/bioinformatics/btl485
Hothorn T, Held L, Friede T (2009) Biometrical journal and reproducible research. Biom J 51(4):553–555. https://doi.org/10.1002/bimj.200900154
Hothorn T, Leisch F (2011) Case studies in reproducibility. Brief Bioinform 12(3):288–300. https://doi.org/10.1093/bib/bbq084
Hübner K, Sahle S, Kummer U (2011) Applications and trends in systems biology in biochemistry: systems biology in biochemical research. FEBS J 278(16):2767–2857. https://doi.org/10.1111/j.1742-4658.2011.08217.x
Hucka M, Finney A, Sauro HM, Bolouri H, Doyle JC, Kitano H, Arkin AP, Bornstein BJ, Bray D, Cornish-Bowden A, Cuellar AA, Dronov S, Gilles ED, Ginkel M, Gor V, Goryanin II, Hedley WJ, Hodgman TC, Hofmeyr JH, Hunter PJ, Juty NS, Kasberger JL, Kremling A, Kummer U, Le Novère N, Loew LM, Lucio D, Mendes P, Minch E, Mjolsness ED, Nakayama Y, Nelson MR, Nielsen PF, Sakurada T, Schaff JC, Shapiro BE, Shimizu TS, Spence HD, Stelling J, Takahashi K, Tomita M, Wagner J, Wang J (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4):524–531. https://doi.org/10.1093/bioinformatics/btg015
Le Novère N, Bornstein B, Broicher A, Courtot M, Donizelli M, Dharuri H, Li L, Sauro H, Schilstra M, Shapiro B, Snoep JL, Hucka M (2006) BioModels database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res 34(Database issue):D689–91 10.1093/nar/gkj092
Le Novère N, Finney A, Hucka M, Bhalla US, Campagne F, Collado-Vides J, Crampin EJ, Halstead M, Klipp E, Mendes P, Nielsen P, Sauro H, Shapiro B, Snoep JL, Spence HD, Wanner BL (2005) Minimum information requested in the annotation of biochemical models (MIRIAM). Natute Biotechnol 23(12):1509–15. https://doi.org/10.1038/nbt1156
Le Novère N, Hucka M, Mi H, Moodie S, Schreiber F, Sorokin A, Demir E, Wegner K, Aladjem MI, Wimalaratne SM, Bergman FT, Gauges R, Ghazal P, Kawaji H, Li L, Matsuoka Y, Villéger A, Boyd SE, Calzone L, Courtot M, Dogrusoz U, Freeman TC, Funahashi A, Ghosh S, Jouraku A, Kim S, Kolpakov F, Luna A, Sahle S, Schmidt E, Watterson S, Wu G, Goryanin I, Kell DB, Sander C, Sauro H, Snoep JL, Kohn K, Kitano H (2009) The systems biology graphical notation. Nat Biotechnol 27(8):735–741. https://doi.org/10.1038/nbt.1558
Lloyd CM, Halstead MD, Nielsen PF (2004) CellML: its future, present and past. Prog Biophys Mol Biol 85(2–3):433–450. https://doi.org/10.1016/j.pbiomolbio.2004.01.004
Loew L, Beckett D, Egelman EH, Scarlata S (2015) Reproducibility of research in biophysics. Biophys J 108(7):E1. https://doi.org/10.1016/j.bpj.2015.03.002
Maddox J, Randi J, Stewart WW (1988) High-dilution experiments a delusion. Nature 334(6180):287–290. https://doi.org/10.1038/334287a0
Mesirov JP (2010) Accessible reproducible research. Science 327(5964):415–416. https://doi.org/10.1126/science.1179653
Moraru I, Morgan F, Li Y, Loew L, Schaff J, Lakshminarayana A, Slepchenko B, Gao F, Blinov M (2008) Virtual cell modelling and simulation software environment. IET Syst Biol 2(5):352–362. https://doi.org/10.1049/iet-syb:20080102
Pahle J, Challenger JD, Mendes P, McKane AJ (2012) Biochemical fluctuations, optimisation and the linear noise approximation. BMC Syst Biol 6(1):86. https://doi.org/10.1186/1752-0509-6-86
Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10(3):405–408. https://doi.org/10.1093/biostatistics/kxp014
Peng RD (2011) Reproducible research in computational science. Science 334(6060):1226–1227. https://doi.org/10.1126/science.1213847
Piccolo SR, Frampton MB (2016) Tools and techniques for computational reproducibility. GigaScience. https://doi.org/10.1186/s13742-016-0135-4
Plesser HE (2018) Reproducibility vs. replicability: a brief history of a confused terminology. Front Neuroinform 11:76. https://doi.org/10.3389/fninf.2017.00076
Popper K (1959) The logic of scientific discovery. Hutchinson, London
Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten simple rules for reproducible computational research. PLoS Comput Biol 9(10):e1003285. https://doi.org/10.1371/journal.pcbi.1003285
Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JPA, Taufer M (2016) Enhancing reproducibility for computational methods. Science 354(6317):1240–1241. https://doi.org/10.1126/science.aah6168
Stodden V, Seiler J, Ma Z (2018) An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci 115(11):2584–2589. https://doi.org/10.1073/pnas.1708290115
Swat M, Moodie S, Wimalaratne S, Kristensen N, Lavielle M, Mari A, Magni P, Smith M, Bizzotto R, Pasotti L, Mezzalana E, Comets E, Sarr C, Terranova N, Blaudez E, Chan P, Chard J, Chatel K, Chenel M, Edwards D, Franklin C, Giorgino T, Glont M, Girard P, Grenon P, Harling K, Hooker A, Kaye R, Keizer R, Kloft C, Kok J, Kokash N, Laibe C, Laveille C, Lestini G, Mentré F, Munafo A, Nordgren R, Nyberg H, Parra-Guillen Z, Plan E, Ribba B, Smith G, Trocóniz I, Yvon F, Milligan P, Harnisch L, Karlsson M, Hermjakob H, Le Novère N (2015) Pharmacometrics markup language (PharmML): opening new perspectives for model exchange in drug development: PharmML - pharmacometrics markup language. CPT: Pharmacometr Syst Pharmacol 4(6):316–319. https://doi.org/10.1002/psp4.57
Waltemath D, Adams R, Beard DA, Bergmann FT, Bhalla US, Britten R, Chelliah V, Cooling MT, Cooper J, Crampin EJ, Garny A, Hoops S, Hucka M, Hunter P, Klipp E, Laibe C, Miller AK, Moraru I, Nickerson D, Nielsen P, Nikolski M, Sahle S, Sauro HM, Schmidt H, Snoep JL, Tolle D, Wolkenhauer O, Le Novère N (2011) Minimum information about a simulation experiment (MIASE). PLoS Comput Biol 7(4):e1001122. https://doi.org/10.1371/journal.pcbi.1001122
Waltemath D, Adams R, Bergmann FT, Hucka M, Kolpakov F, Miller AK, Moraru II, Nickerson D, Sahle S, Snoep JL, Le Novère N (2011) Reproducible computational biology experiments with SED-ML: the simulation experiment description markup language. BMC Syst Biol 5(1):198. https://doi.org/10.1186/1752-0509-5-198
Funding
Funding was provided by National Institute of General Medical Sciences (Grant No. GM080219).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Mendes, P. Reproducible Research Using Biomodels. Bull Math Biol 80, 3081–3087 (2018). https://doi.org/10.1007/s11538-018-0498-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11538-018-0498-z