Abstract
The use of computational models in science end engineering is increasingly becoming pervasive. However, there is a growing credibility gap due to widespread, relaxed attitudes in communication of experiments, models, and validation of simulations used in computational research. Consequent disputes and article retractions due to unverified code and data suggest a pressing need for greater transparency. We introduce the e Portfolio concept, which is an ensemble documents that interweave the conceptual model, simulator design, experimental frames, and scientific workflow specifications. Strategies and potential mechanisms are delineated to enable authors, publishers, funding agencies, journals, and the broader scientific community to cooperate and establish a sustained model base, simulations, experiments, and documentation, so that scientists can build on each other’s work and achievements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Alberts, B.: Editorial Expression of Concern. Science 327(5962), 144 (2010), http://www.sciencemag.org/content/327/5962/144.1.full
Altman, M., King, G.: A Proposed Standard for the Scholarly Citation of Quantitative Data. D-Lib Magazine (March/April 2007), http://www.dlib.org/dlib/march07/altman/03altman.html
Anand, M.K., Bowers, S., McPhillips, T., Ludäscher, B.: Exploring Scientific Workflow Provenance Using Hybrid Queries over Nested Data and Lineage Graphs. In: Winslett, M. (ed.) SSDBM 2009. LNCS, vol. 5566, pp. 237–254. Springer, Heidelberg (2009)
Annotea. The Annotea Project (2011), http://www.w3.org/2001/Annotea/
Baggerly, K., Coombes, K.: Deriving Chemosensitivity from Cell Lines: Forensic Bioinformatics and Reproducible Research in High-Throughput Biology. Annals Applied Statistics 3(4), 1309–1334 (2009)
Beck, K.: Test-Driven Development by Example. Addison Wesley (2003)
Borgman, C.L.: Scholarship in the Digital Age: Information, Infrastructure, and the Internet. The MIT Press, Cambridge (2007)
Chang, C., et al.: Retraction. Science 314(5807), 1875 (2006), http://www.sciencemag.org/cgi/content/full/314/5807/1875b
CreativeCommons (2011), http://www.creativecommons.org
Cummings, T.P., Glotzer, S.C.: Inventing a New America through Discovery and Innovation in Science, Engineering, and Medicine. World Technology Evaluation Center (2010)
David, A.P., Spence, M.: Toward Institutional Infrastructures for e-Science: The Scope of Challenges. Technical Report: Oxford Technical Institute (May 2003)
Data Cite (2011), http://thedata.org/citation/tech
De Roure, D., Goble, C., Stevens, R.: The Design and Realisation of the myExperiment Virtual Research Environment for Social Sharing of Workflows. Future Generation Computer Systems 25, 561–567 (2009)
Donoho, D., Maleki, A., Rahman, I., Shahram, M., Stodden, V.: Reproducible Research in Computational Harmonic Analysis. Computing in Science & Engineering 11(1), 818 (2009)
Economist, The Clouds of Unknowing. The Economist (March 18, 2010), http://www.economist.com/node/15719298
Edwards, H.S.: Improving Student Performance by Evaluating How Well Students Test Their Own Programs. Journal of Educational Resources in Computing 3(3), 1–24 (2003)
Fenton, E.N., Bieman, J.: Software Metrics: A Rigorous and Practical Approach. Chapman & Hall (2010)
Fomel, S., Hennenfent, G.: Reproducible Computational Experiments using Scons. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. 1257–1260 (2009)
Fomel, S., Claerbout, J.: Guest Editors’ Introduction: Reproducible Research. Computing in Science and Engineering 11(1), 5–7 (2009)
Foster, I.: Service-oriented Science. Science 308, 814–817 (2005)
Gentleman, R.: Reproducible Research: A Bioinformatics Case Study. Statistical Applications in Genetics and Molecular Biology 4(1), 123–137 (2005)
Kiczales, G., Des Rivieres, J., Bobrow, D.G.: The Art of the Meta-object Protocol. MIT Press (1992)
King, D.R., Turnitsa, C.D.: The Landscape of Assumptions. In: Proceedings of the Spring Simulation Multiconference, pp. 81–88 (2008)
Leisch, F.: Sweave and Beyond: Computations on Text Documents. In: Proceedings of the 3rd Intl Workshop on Distributed Statistical Computing, pp. 87–96 (2003)
Mesirov, J.: Accessible Reproducible Research. Science 327(5964), 415–416 (2010)
NSF. Simulation-Based Engineering Science, Revolutionizing Engineering Science through Simulation. National Science Foundation (2006)
NSF. Changing the Conduct of Science in the Information Age. Summary Report of Workshop Held on November 12, 2010. National Science Foundation (June 28, 2011), http://www.nsf.gov/pubs/2011/oise11003/
Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., Glover, K.: Taverna: A Tool for the Composition and Enactment of Bioinformatics Workflows. Bioinformatics 20(17), 3045–3054 (2004)
Overstreet, M.C., Nance, R.E., Balci, O.: Issues in Enhancing Model Reuse. In: Proceedings of the First International Conference on Grand Challenges for Modeling and Simulation (2002)
Ören, T.I., Zeigler, B.P.: Concepts for Advanced Simulation Methodologies. Simulation 32(3), 69–82 (1979)
Ören, T.I.: The Richness of Modeling and Simulation and its Body of Knowledge. In: Proceedings of SIMULTECH 2012, 2nd International Conference on Simulation and Modeling Methodologies, Technologies and Applications, Rome, Italy, July 28-31 (invited keynote paper - in press, 2012)
Peng, R.: Reproducible Research and Biostatistics. Biostatistics 10(3), 405–408 (2009)
reST. reStructuredText: Markup Syntax and Parser Component of Docutils (2011), http://docutils.sourceforge.net/rst.html
Saeki, M., Hiroi, T., Ugai, T.: Reflective Specification: Applying a Reflective Language to Formal Specification. In: Proceedings of the 7th International Workshop on Software Specification and Design, pp. 204–213. IEEE CS Press (1993)
Schwab, M., Karrenbach, M., Claerbout, J.: Making Scientific Computations Reproducible. Computing in Science and Engineering 2, 61–67 (2000)
SCS. The Code of Professional Ethics for Simulationists (2012), http://www.scs.org/upload/03-Code_0.pdf
Stodden, V.: The Legal Framework for Reproducible Scientific Research: Licensing and Copyright. Computing in Science & Engineering 11(1), 35–40 (2009)
Stodden, V.: The Scientific Method in Practice: Reproducibility in the Computational Sciences, Technical report, MIT Sloan Research (February 2010)
Sphinx. Python Document Generator (2011), http://sphinx.pocoo.org
Wilensky, U., Rand, W.: Making Models Match: Replicating an Agent-Based Model. Journal of Artificial Societies and Social Simulation 10(42) (2007), http://jasss.soc.surrey.ac.uk/10/4/2.html
WTEC. Panel Report on International Assessment of Research and Development in Simulation-Based Engineering and Science. World Technology Evaluation Center (2009)
Yilmaz, L.: On the Need for Contextualized Introspective Simulation Models to Improve Reuse and Composability of Defense Simulations. Journal of Defense Modeling and Simulation 1(3), 135–145 (2005)
Yilmaz, L.: Reproducibility in Modeling & Simulation Research. Simulation 87(1), 3–6 (2011a)
Yilmaz, L.: Simulation Reproducibility: Implications for Theory, Methodology, and Model Development Environments. In: Epistemology of Modeling & Simulation Conference, Pittsburgh, April 1-3 (2011b), http://www.modelingepistemology.pitt.edu/program/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Yilmaz, L., Ören, T. (2013). Toward Replicability-Aware Modeling and Simulation: Changing the Conduct of M&S in the Information Age. In: Tolk, A. (eds) Ontology, Epistemology, and Teleology for Modeling and Simulation. Intelligent Systems Reference Library, vol 44. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31140-6_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-31140-6_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31139-0
Online ISBN: 978-3-642-31140-6
eBook Packages: EngineeringEngineering (R0)