Issues in Reproducible Simulation Research
In recent years, serious concerns have arisen about reproducibility in science. Estimates of the cost of irreproducible preclinical studies range from 28 billion USD per year in the USA alone (Freedman et al. in PLoS Biol 13(6):e1002165, 2015) to over 200 billion USD per year worldwide (Chalmers and Glasziou in Lancet 374:86–89, 2009). The situation in the social sciences is not very different: Reproducibility in psychological research, for example, has been estimated to be below 50% as well (Open Science Collaboration in Science 349:6251, 2015). Less well studied is the issue of reproducibility of simulation research. A few replication studies of agent-based models, however, suggest the problem for computational modeling may be more severe than for laboratory experiments (Willensky and Rand in JASSS 10(4):2, 2007; Donkin et al. in Environ Model Softw 92:142–151, 2017; Bajracharya and Duboz in: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11, 2013). In this perspective, we discuss problems of reproducibility in agent-based simulations of life and social science problems, drawing on best practices research in computer science and in wet-lab experiment design and execution to suggest some ways to improve simulation research practice.
KeywordsAgent-based models Simulation reproducibility Validation Test-driven development Version control Computational lab notebook
This work was partially supported by National Institute on Drug Abuse grant 1R43DA041760-01.
- Bajracharya K, Duboz R (2013) Comparison of three agent-based platforms on the basis of a simple epidemiological model (WIP). In: Proceedings of the symposium on theory of modeling and simulation—DEVS integrative M&S symposium, pp 6–11Google Scholar
- Beck K (2003) Test-Driven Development: By Example. Pearson, BostonGoogle Scholar
- Freedman LP, Cockburn IM, Simcoe TS (2015) The economics of reproducibility. PLoS Biol 13(6):e1002165. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002165. Accessed 10 June 2015
- Mäkinen S, Munch J (2014) Effects of test-driven development: a comparative analysis of empirical studies. In: Winkler D, Bifll S, Bergsmann J (eds) Software quality: model-based approaches for advanced software and systems engineering. Springer, ChamGoogle Scholar
- Martin RC (2008) Clean code: a handbook of agile software craftmanship. Pearson, BostonGoogle Scholar
- Smith, R. (2017). Personal communicationGoogle Scholar
- Wilenksy U, Rand W (2007) Making models match: replicating an agent-based model. JASSS 10(4):2Google Scholar