Skip to main content
SpringerLink
Log in

Valid and Reproducible Simulation Studies—Making It Explicit

  • Chapter
  • First Online:
Computer Simulation Validation

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

The validation of complex simulation models is a challenging task. To increase the trust into the model, diverse simulation experiments are executed to explore the behavior of the model and to check its plausibility. Thus, these simulation experiments present an important information about the validity of the model, similarly as the data used for calibration, as input for the model, and for testing its predictiveness. Simulation models are rarely developed from scratch but by reusing existing models, e.g., by extending or composing them, or for cross-validation. These models and their validity provide further details about the validity of a model. Thus, a multitude of artifacts contribute intricately related to the final simulation model and our “gut feelings” about it. To make these artifacts and their relations explicit and accessible, we will apply a declarative formal modeling language, a declarative language for specifying and executing diverse simulation experiments, and a provenance model to relate the diverse artifacts in telling the validation tale of an agent-based migration model.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Agha, G., & Palmskog, K. (2018). A survey of statistical model checking. ACM Transactions on Modeling and Computer Simulation, 28(1), 6:1–6:39.

    Google Scholar 

  • Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211.

    Article  Google Scholar 

  • Balci, O. (1997). Verification, validation and accreditation of simulation models. In Proceedings of the 1997 winter simulation conference, pp. 135–141. IEEE.

    Google Scholar 

  • Bandourian, R., McDonald, J., & Turley, R. S. (2002). A comparison of parametric models of income distribution across countries and over time. SSRN Scholarly Paper ID 324900, Social Science Research Network, Rochester, NY, June 2002.

    Google Scholar 

  • Beauchemin, C. (2015). Migration between Africa and Europe (MAFE): Looking beyond immigration to understand international migration. Population, 70(1), 13–38.

    Article  Google Scholar 

  • Bergmann, F. T., Cooper, J., Le Novere, N., Nickerson, D., & Waltemath, D. (2015). Simulation experiment description markup language (SED-ML) Level 1 Version 2. Journal of Integrative Bioinformatics (JIB), 12(2), 119–212.

    Google Scholar 

  • Bochner, C., Gude, R.,& Schreiber, A. (2008). A python library for provenance recording and querying. Provenance and Annotation of Data and Processes, pp. 229–240.

    Google Scholar 

  • Coale, A. J., & Mcneil, D. R. (1972). The distribution by age of the frequency of first marriage in a female cohort. Journal of the American Statistical Association, 67(340), 743–749.

    Article  Google Scholar 

  • Conte, R., et al. (2012). Manifesto of computational social science. European Physical Journal-Special Topics, 214, 325.

    Google Scholar 

  • Davies, J. B., Sandström, S., Shorrocks, A. B., & Wolff, E. N. (2011). The level and distribution of global household wealth. The Economic Journal, 121(551), 223–254.

    Article  Google Scholar 

  • De Nies, T., et al. (2013). Git2PROV: Exposing version control system content as w3c prov. In Proceedings of the 2013th International Conference on Posters & Demonstrations Track-Volume 1035 (pp. 125–128). CEUR-WS. org.

    Google Scholar 

  • Ewald, R., & Uhrmacher, A. M. (2014). SESSL: A domain-specific language for simulation experiments. ACM Transactions on Modeling and Computer Simulation, 24(2), 11:1–11:25.

    Google Scholar 

  • Gray, J., Hilton, J., & Bijak, J. (2017). Choosing the choice: Reflections on modelling decisions and behaviour in demographic agent-based models. Population Studies, 71(sup1), 85–97

    Google Scholar 

  • Grimm, V., et al. (2006). A standard protocol for describing individual-based and agent-based models. Ecological Modelling, 198(1), 115–126.

    Google Scholar 

  • Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. F. (2010). The ODD protocol: A review and first update. Ecological Modelling, 221(23), 2760–2768.

    Article  Google Scholar 

  • Groth, P.,& Moreau, L. (2013). PROV-overview. An overview of the PROV family of documents.

    Google Scholar 

  • Harris, L. A., et al. (2016). Bionetgen 2.2: Advances in rule-based modeling. Bioinformatics, 32(21), 3366–3368.

    Article  Google Scholar 

  • Heligman, L., & Pollard, J. H. (1980). The age pattern of mortality. Journal of the Institute of Actuaries, 107(1), 49–80.

    Article  Google Scholar 

  • Helms, T., Himmelspach, J., Maus, C., Röwer, O., Schützel, J., & Uhrmacher, A. M. (2012). Toward a language for the flexible observation of simulations. In Proceedings of the 2012 Winter Simulation Conference, pp. 418:1–418:12. IEEE.

    Google Scholar 

  • IEEE. (2009). IEEE Standard VHDL Language Reference Manual. IEEE Std 1076-2008 (Revision of IEEE Std 1076-2002) (pp. c1–626).

    Google Scholar 

  • Klabunde, A., Willekens, F., Zinn, S., & Leuchter, M. (2015). An agent-based decision model of migration, embedded in the life course—Model description in ODD+D format. Max Planck Institute for Demographic Research, Rostock, Germany: Technical report.

    Google Scholar 

  • Klabunde, A., Zinn, S., Willekens, F., & Leuchter, M. (2016). Multistate modeling extended by behavioral rules (Version 6). https://www.openabm.org/model/5146/version/6/view.

  • Klabunde, A., Zinn, S., Willekens, F., & Leuchter, M. (2017). Multistate modeling extended by behavioral rules—an example of migration. Population Studies, 71(sup1), 51–67

    Google Scholar 

  • Klügl, F. (2008). A validation methodology for agent-based simulations. In Proceedings of the 2008 ACM Symposium on Applied Computing, SAC’08 (pp. 39–43). New York: ACM.

    Google Scholar 

  • Kossow, C., Helms, T., Kreutzer, J. M., Martens, A., & Uhrmacher, A. M. (2016) Evaluating different modeling languages based on a user study. In Proceedings of the 49th Annual Simulation Symposium, ANSS ’16 (pp. 18:1–18:8). Society for Computer Simulation International, San Diego, CA, USA.

    Google Scholar 

  • Laurent, J., Medina-Abarca, H. F., Boutillier, P., Yang, J.,& Fontana, W. (2018). A trace query language for rule-based models. In Computational Methods in Systems Biology (CMSB 2018), Lecture Notes in Bioinformatics. Cham: Springer.

    Google Scholar 

  • Leye, S., Himmelspach, J., Uhrmacher, A. M. (2009). A discussion on experimental model validation. In 2009 11th International Conference on Computer Modelling and Simulation (pp. 161–167).

    Google Scholar 

  • Lim, C., Lu, S., Chebotko, A.,& Fotouhi, F. (2011). OPQL: A first OPM-level query language for scientific workflow provenance. In 2011 IEEE International Conference on Services Computing, pp. 136–143. IEEE.

    Google Scholar 

  • Lukasiewycz, M., Glaß, M., Reimann, F.,& Teich, J. (2011). Opt4J—a modular framework for meta-heuristic optimization. In Proceedings of the Genetic and Evolutionary Computing Conference (GECCO 2011) (pp. 1723–1730), Dublin, Ireland.

    Google Scholar 

  • Maler, O., & Nickovic, D. (2004). Monitoring temporal properties of continuous signals. Formal Techniques (pp. 152–166), Modelling and Analysis of Timed and Fault-Tolerant Systems, Lecture Notes in Computer Science. Heidelberg: Springer.

    Google Scholar 

  • Maus, C., Rybacki, S., & Uhrmacher, A. M. (2011). Rule-based multi-level modeling of cell biological systems. BMC Systems Biology, 5, 166.

    Article  Google Scholar 

  • Moreau, L., et al. (2011). The open provenance model core specification (v1. 1). Future Generation Computer Systems, 27(6), 743–756.

    Article  Google Scholar 

  • Murta, L., Braganholo, V., Chirigati, F., Koop, D.,& Freire, J. (2014). Noworkflow: Capturing and analyzing provenance of scripts. In International Provenance and Annotation Workshop (pp. 71–83). Springer.

    Google Scholar 

  • Noble, E. et al. (2012). Linked lives: The Utility of an agent-based approach to modeling partnership and household formation in the context of social care. In Proceedings of the 2012 Winter Simulation Conference, pp. 93:1–93:12. IEEE.

    Google Scholar 

  • Novère, N. L., Finney, A., Hucka, M., Bhalla, U. S., Campagne, F., Collado-Vides, J., et al. (2005). Minimum information requested in the annotation of biochemical models (MIRIAM). Nature Biotechnology, 23(12), 1509–1515.

    Article  Google Scholar 

  • Peng, D., Warnke, T., Haack, F., & Uhrmacher, A. M. (2016). Reusing simulation experiment specifications to support developing models by successive extension. Simulation Modelling Practice and Theory, 68, 33–53.

    Article  Google Scholar 

  • Peng, D., Warnke, T., Haack, F., & Uhrmacher, A. M. (2017). Reusing simulation experiment specifications in developing models by successive composition—a case study of the wnt/\(\beta \)-catenin signaling pathway. SIMULATION, 93(8), 659–677.

    Article  Google Scholar 

  • Pierce, M. E., Krumme, U., & Uhrmacher, A. M. (2018). Building simulation models of complex ecological systems by successive composition and reusing simulation experiments. In Proceedings of the 2018 Winter Simulation Conference. IEEE.

    Google Scholar 

  • Rahmandad, H., & Sterman, J. D. (2012). Reporting guidelines for simulation-based research in social sciences. System Dynamics Review, 28(4), 396–411.

    Article  Google Scholar 

  • Reinhardt, O., & Uhrmacher, A. M. (2017). An efficient simulation algorithm for continuous-time agent-based linked lives models. In Proceedings of the 50th Annual Simulation Symposium, ANSS’17, pp. 9:1–9:12, San Diego, CA, USA. Society for Computer Simulation International.

    Google Scholar 

  • Reinhardt, O., Hilton, J., Warnke, T., Bijak, J., & Uhrmacher, A. M. (2018a). Streamlining simulation experiments with agent-based models in demography. Journal of Artificial Societies and Social Simulation, 21(3), 9.

    Google Scholar 

  • Reinhardt, O., Ruscheinski, A., & Uhrmacher, A. M. (2018b). Odd+p: Complementing the odd protocol with provenance information. In Proceedings of the 2018 Winter Simulation Conference. IEEE.

    Google Scholar 

  • Rozier, K. Y. (2011). Linear temporal logic symbolic model checking. Computer Science Review, 5(2), 163–203.

    Article  Google Scholar 

  • Ruscheinski, A., & Uhrmacher, A. M. (2017). Provenance in modeling and simulation studies-bridging gaps. In Proceedings of the 2017 Winter Simulation Conference. IEEE Press.

    Google Scholar 

  • Rybacki, S., Leye, S., Himmelspach, J., & Uhrmacher, A. M. (2012). Template and frame based experiment workflows in modeling and simulation software with worms. In 2012 IEEE Eighth World Congress on Services (SERVICES) (pp. 25–32). IEEE.

    Google Scholar 

  • Scheidegger, C., Koop, D., Santos, E., Vo, H., Callahan, S., Freire, J., et al. (2008). Tackling the provenance challenge one layer at a time. Concurrency and Computation: Practice and Experience, 20(5), 473–483.

    Article  Google Scholar 

  • Schützel, J., Peng, D., Uhrmacher, A. M., & Perrone, L. F. (2014). Perspectives on languages for specifying simulation experiments. In Proceedings of the 2014 Winter Simulation Conference (pp. 2836–2847). IEEE Press (2014).

    Google Scholar 

  • Sheppard,C. J.,& Railsback, S. (2015). Time Extension for NetLogo (Version 1.2) [Software]. https://github.com/colinsheppard/time.

  • Steiniger, A., Uhrmacher, A. M., Zinn, S., Gampe, J.,& Willekens, F. (2014). The role of languages for modeling and simulating continuous-time multi-level models in demography. In Proceedings of the 2014 Winter Simulation Conference, pp. 2978–2989. IEEE.

    Google Scholar 

  • Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (ed.), 18th European Simulation Multiconference. Networked Simulations and Simulation Networks (pp. 265–270).

    Google Scholar 

  • Troitzsch, K. G. (2017). Using empirical data for designing, calibrating and validating simulation models. Advances in Social Simulation 2015 (pp. 413–427). Advances in Intelligent Systems and Computing. Cham: Springer.

    Google Scholar 

  • Uhrmacher, A. M., Brailsford, S., Liu, J., Rabe, M.,& Tolk,A. (2016). Panel—reproducible research in discrete event simulation—a must or rather a maybe? In Proceedings of the 2016 Winter Simulation Conference, pp. 1301–1315. IEEE.

    Google Scholar 

  • van Deursen, A., Klint, P., & Visser, J. (2000). Domain-specific languages: An annotated bibliography. SIGPLAN Notices, 35(6), 26–36.

    Article  Google Scholar 

  • Waltemath, D., Adams, R., Bergmann, F. T., Hucka, M., Kolpakov, F., Miller, A. K., et al. (2011). Reproducible computational biology experiments with SED-ML—the simulation experiment description markup language. BMC Systems Biology, 5(1), 198.

    Article  Google Scholar 

  • Warnke, T., Reinhardt, O., Klabunde, A., Willekens, F., & Uhrmacher, A. M. (2017). Modelling and simulating decision processes of linked lives—an approach based on concurrent processes and stochastic race. Population Studies, 71(sup1), 69–83.

    Google Scholar 

  • Wittgenstein, L. (1922). Tractatus Logico-Philosophicus. London: Routledge & Kegan Paul.

    MATH  Google Scholar 

  • Zeigler, B. P., Praehofer, H., & Kim, T. G. (2000). Theory of Modeling and Simulation (2nd ed.). San Diego: Academic Press.

    MATH  Google Scholar 

  • Zinn, S. (2012). A mate-matching algorithm for continuous-time microsimulation models. International Journal of Microsimulation, 5(1), 31–51.

    Google Scholar 

Download references

Acknowledgements

This research was supported by the German Research Foundation (DFG) via research grant UH-66/15 and research grant UH-66/18.

Author information

Authors and Affiliations

  1. University of Rostock, Institute of Computer Science, Albert-Einstein-Straße 22, 18059, Rostock, Germany

    Oliver Reinhardt, Tom Warnke, Andreas Ruscheinski & Adelinde M. Uhrmacher

Authors
  1. Oliver Reinhardt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tom Warnke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andreas Ruscheinski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adelinde M. Uhrmacher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Oliver Reinhardt .

Editor information

Editors and Affiliations

  1. University of Bern, Bern, Switzerland

    Claus Beisbart

  2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Nicole J. Saam

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Reinhardt, O., Warnke, T., Ruscheinski, A., Uhrmacher, A.M. (2019). Valid and Reproducible Simulation Studies—Making It Explicit. In: Beisbart, C., Saam, N. (eds) Computer Simulation Validation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-70766-2_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70766-2_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70765-5

  • Online ISBN: 978-3-319-70766-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation