Skip to main content
SpringerLink
Log in
Book cover

Computer Simulation Validation pp 69–97Cite as

Simulation Accuracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective

  • Chapter
  • First Online:

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

Most computational analysts, as well as most governmental policy-makers and the public, view computational simulation accuracy as a good agreement of simulation results with empirical measurements. However, decision-makers, such as business managers and safety regulators who rely on simulation for decision support, view computational simulation accuracy as much more than agreement of simulation results with experimental data . Decision-makers’ concept of accuracy is better captured by the term predictive capability of the simulation. Predictive capability meaning the use of a computational model to foretell or forecast the response of a system to conditions without available experimental data , even for system responses that have never occurred in nature. This chapter makes this important distinction by discussing the crucial ingredients needed for predictive capability : code verification , solution (or calculation) verification, model validation, model calibration , and predictive uncertainty estimation. Each of these ingredients is required, whether the simulation results are used in the generation of new knowledge , or for decision support by business managers, government policy-makers, or safety regulators.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Abbreviations

CFD:

Computational fluid dynamics

MFE:

Model form error

MMS:

Method of manufactured solutions

PDE:

Partial differential equation

SRQ:

System response quantity

SQA:

Software quality assurance

References

  • Abanto, J., Pelletier, D., Garon, A., Trepanier, J.-Y., & Reggio, M. (2005). Verification of some commercial CFD codes on atypical CFD problems. Paper presented at the 43rd AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV.

    Google Scholar 

  • Aeschliman, D. P., & Oberkampf, W. L. (1998). Experimental methodology for computational fluid dynamics code validation. AIAA Journal, 36(5), 733–741.

    Article  Google Scholar 

  • AIAA. (1998). Guide for the verification and validation of computational fluid dynamics simulations (AIAA-G-077-1998). Retrieved from Reston, VA.

    Google Scholar 

  • Ainsworth, M., & Oden, J. T. (2000). A posteriori error estimation in finite element analysis. New York: Wiley.

    Book  MATH  Google Scholar 

  • Anderson, M. G., & Bates, P. D. (Eds.). (2001). Model validation: Perspectives in hydrological science. New York, NY: Wiley.

    Google Scholar 

  • ASME. (2006). Guide for verification and validation in computational solid mechanics (ASME Standard V&V 10-2006). Retrieved from New York, NY.

    Google Scholar 

  • ASME. (2009). Standard for verification and validation in computational fluid dynamics and heat transfer (ASME Standard V&V 20-2009). Retrieved from New York, NY.

    Google Scholar 

  • ASME. (2012). An illustration of the concepts of verification and validation in computational solid mechanics (ASME Standard V&V 10.1-2012). Retrieved from New York, NY.

    Google Scholar 

  • Augustin, T., Coolen, F. P. A., de Cooman, G., & Troffaes, M. C. M. (Eds.). (2014). Introduction to imprecise probabilities. Chichester, UK: Wiley.

    MATH  Google Scholar 

  • Babuska, I., & Strouboulis, T. (2001). The finite element method and its reliability. Oxford, U.K.: Oxford University Press.

    MATH  Google Scholar 

  • Babuska, I., Nobile, F., & Tempone, R. (2008). A systematic approach to model validation based on bayesian updates and prediction related rejection criteria. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2517–2539.

    Article  MATH  Google Scholar 

  • Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., et al. (2007). A framework for validation of computer models. Technometrics, 49(2), 138–154.

    Article  MathSciNet  Google Scholar 

  • Bernardini, A., & Tonon, F. (2010). Bounding uncertainty in civil engineering. Berlin: Springer.

    Book  MATH  Google Scholar 

  • Beven, K. (2002). Towards a coherent philosophy of modelling the environment. Proceedings of the Royal Society of London Series A, 458(2026), 2465–2484.

    Article  MathSciNet  MATH  Google Scholar 

  • Bossel, H. (1994). Modeling and simulation (1st ed.). Wellesley, MA: A. K. Peters.

    Book  MATH  Google Scholar 

  • Chen, W., Xiong, Y., Tsui, K.-L., & Wang, S. (2008). A design-driven validation approach using bayesian prediction models. Journal of Mechanical Design, 130(2), 021101–021112.

    Google Scholar 

  • Chiles, J.-P., & Delfiner, P. (1999). Geostatistics: Modeling spatial uncertainty. New York: Wiley.

    Book  MATH  Google Scholar 

  • Cooke, R. (2004). The antomy of the squizzel: The role of operational definitions in representing uncertainty. Reliability Engineering and System Safety, 85(1–3), 313–319.

    Article  Google Scholar 

  • Cullen, A. C., & Frey, H. C. (1999). Probabilistic techniques in exposure assessment: A handbook for dealing with variability and uncertainty in models and inputs. New York: Plenum Press.

    Google Scholar 

  • Davey, S., Gordon, N., Holland, I., Rutten, M., & Williams, J. (2016). Bayesian methods in the search for MH370. Springer Nature (Open Access).

    Google Scholar 

  • Donoho, D. L., Maleki, A., Shahram, M., Rahman, I. U., & Stodden, V. (2009). Reproducible research in computational harmonic analysis. Computing in Science & Engineering, 11(1), 8–18.

    Article  Google Scholar 

  • Duggirala, R. K., Roy, C. J., Saeidi, S. M., Khodadadi, J. M., Cahela, D. R., & Tatarchuk, B. J. (2008). Pressure drop predictions in microfibrous materials using computational fluid dynamics. Journal of Fluids Engineering, 130(7), 071302–071313.

    Google Scholar 

  • Eca, L., & Hoekstra, M. (2002). An evaluation of verification procedures for CFD applications. Paper presented at the Proceedings of the 24th Symposium on Naval Hydrodynamics, Fukuoka, Japan.

    Google Scholar 

  • Ferson, S., & Oberkampf, W. L. (2009). Validation of imprecise probability models. International Journal of Reliability and Safety, 3(1–3), 3–22.

    Article  Google Scholar 

  • Ferson, S., Oberkampf, W. L., & Ginzburg, L. (2008). Model validation and predictive capability for the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2408–2430.

    Article  MATH  Google Scholar 

  • Ferziger, J. H., & Peric, M. (2002). Computational methods for fluid dynamics (3rd ed.). New York: Springer.

    Book  MATH  Google Scholar 

  • Fomel, S., & Claerbout, J. F. (2009). Guest editors’ introduction: Reproducible research. Computing in Science & Engineering, 11(1), 5–7.

    Article  Google Scholar 

  • Golub, G. H., & Van Loan, C. F. (2013). Matrix computations (4th ed.). Baltimore, MD: The Johns Hopkins University Press.

    MATH  Google Scholar 

  • Haimes, Y. Y. (2009). Risk modeling, assessment, and management (3rd ed.). New York: Wiley.

    MATH  Google Scholar 

  • Hatton, L. (1997). The T experiments: Errors in scientific software. IEEE Computational Science & Engineering, 4(2), 27–38.

    Article  Google Scholar 

  • Higdon, D., Nakhleh, C., Gattiker, J., & Williams, B. (2008). A bayesian calibration approach to the thermal problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2431–2441.

    Article  MATH  Google Scholar 

  • Jolliffe, I. T., & Stephenson, D. B. (Eds.). (2011). Forecast verification: A practitioner’s guide in atmospheric science (2nd ed.). Hoboken, NJ: Wiley.

    Google Scholar 

  • Kennedy, M. C., & O’Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society Series B-Statistical Methodology, 63(3), 425–450.

    Article  MathSciNet  MATH  Google Scholar 

  • Knupp, P., & Salari, K. (2002). Verification of computer codes in computational science and engineering. Boca Raton, FL: Chapman & Hall/CRC.

    Book  MATH  Google Scholar 

  • Lehmann, E. L., & Romano, J. P. (2005). Testing statistical hypotheses (3rd ed.). Berlin: Springer.

    MATH  Google Scholar 

  • Leijnse, A., & Hassanizadeh, S. M. (1994). Model definition and model validation. Advances in Water Resources, 17, 197–200.

    Article  Google Scholar 

  • Lenhard, J., & Winsberg, E. (2010). Holism, entrenchment, and the future of climate model pluralism. Studies in History and Philosophy of Modern Physics, 41, 253–262.

    Article  Google Scholar 

  • LeVeque, R. J., Mitchell, I. M., & Stodden, V. (2012). Reproducible research for scientific computing: Tools and strategies for changing the culture. Computing in Science & Engineering, 14(4), 13–17.

    Article  Google Scholar 

  • Li, W., Chen, W., Jiang, Z., Lu, Z., & Liu, Y. (2014). New validation metrics for models with multiple correlated responses. Reliability Engineering and System Safety, 127, 1–11.

    Article  Google Scholar 

  • Li, W., Chen, S., Jiang, Z., Apley, D. W., Lu, Z., & Chen, W. (2016). Integrating bayesian calibration, bias correction, and machine learning for the 2014 sandia verification and validation challent problem. Journal of Verification, Validation and Uncertainty Quantification, 1(1), 011004–011012.

    Google Scholar 

  • Liu, F., Bayarri, M. J., Berger, J. O., Paulo, R., & Sacks, J. (2008). A bayesian analysis of the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2457–2466.

    Article  MATH  Google Scholar 

  • Liu, Y., Chen, W., Arendt, P., & Huang, H.-Z. (2011). Toward a better understanding of model validation metrics. Journal of Mechanical Design, 133(13), 071001–071013.

    Google Scholar 

  • Marvin, J. G. (1995). Perspective on computational fluid dynamics validation. AIAA Journal, 33(10), 1778–1787.

    Article  MATH  Google Scholar 

  • McFarland, J., & Mahadevan, S. (2008). Multivariate significance testing and model calibration under uncertainty. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2467–2479.

    Article  MATH  Google Scholar 

  • Morgan, M. G., & Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis (1st ed.). Cambridge, UK: Cambridge University Press.

    Book  Google Scholar 

  • Morrison, J. H. (2014). Statistical analysis of the fifth drag prediction workshop computational fluid dynamics solutions. Journal of Aircraft, 51(4), 1214–1222.

    Article  Google Scholar 

  • Neumann, P. G. (1995). Computer-related risks. New York: ACM Press, Addison-Wesley Publishing Company.

    Google Scholar 

  • Oberkampf, W. L., & Aeschliman, D. P. (1992). Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 1, experimental results. AIAA Journal, 30(8), 2000–2009.

    Article  Google Scholar 

  • Oberkampf, W. L., & Barone, M. F. (2006). Measures of agreement between computation and experiment: Validation metrics. Journal of Computational Physics, 217(1), 5–36.

    Article  MATH  Google Scholar 

  • Oberkampf, W. L., & Trucano, T. G. (2008). Verification and validation benchmarks. Nuclear Engineering and Design, 238(3), 716–743.

    Article  Google Scholar 

  • Oberkampf, W. L., & Roy, C. J. (2010). Verification and validation in scientific computing. Cambridge, UK: Cambridge University Press.

    Book  MATH  Google Scholar 

  • O’Hagan, A. (2006). Bayesian analysis of computer code outputs: A tutorial. Reliability Engineering and System Safety, 91(10–11), 1290–1300.

    Article  Google Scholar 

  • Reason, J. (1997). Managing the risks of organizational accidents. Burlington, VT: Ashgate Publishing Limited.

    Google Scholar 

  • Reason, J. (2008). The human contribution: Unsafe acts, accidents and heroic recoveries. Burlington, VT: Ashgate Publishing Co.

    Google Scholar 

  • Refsgaard, J. C., & Henriksen, H. J. (2004). Modelling guidelines-terminology and guiding principles. Advances in Water Resources, 27(1), 71–82.

    Article  Google Scholar 

  • Roache, P. J. (1972). Computational fluid dynamics. Albuquerque, NM: Hermosa Publishers.

    MATH  Google Scholar 

  • Roache, P. (2009). Fundamentals of verification and validation. Socorro, New Mexico: Hermosa Publishers.

    Google Scholar 

  • Rougier, J. (2007). Probabilistic inference for future climate using an ensemble of climate model evaluations. Climate Change, 81(3–4), 247–264.

    Article  Google Scholar 

  • Roy, C. J. (2010). Review of discretization error estimators in scientific computing. Paper presented at the 48th AIAA Aerospace Sciences Meeting, Orlando, FL.

    Google Scholar 

  • Rumsey, C. L., Reif, B. A. P., & Gatski, T. B. (2006). Arbitrary steady-state solutions with the k–ε model. AAIAA Journal, 44(7), 1586–1592.

    Article  Google Scholar 

  • Rykiel, E. J. (1996). Testing ecological models: The meaning of validation. Ecological Modelling, 90(3), 229–244.

    Article  Google Scholar 

  • Silver, N. (2012). The signal and the noise. New York, NY: Penguin Books.

    Google Scholar 

  • Stodden, V. (2012). Guest editor’s introduction: Reproducible research-tools and strategies for scientific computing. IEEE Computing in Science and Engineering, 14(4), 11–12.

    Article  Google Scholar 

  • Taleb, N. N. (2007). The black swan: The impact of the highly improbable. New York: Random House.

    Google Scholar 

  • Taleb, N. N. (2008). The fourth quadrant: A map of the limits of statistics. Retrieved September 14, 2008, from https://www.edge.org/conversation/the-fourth-quadrant-a-map-of-the-limits-of-statistics.

  • Trucano, T. G., Easterling, R. G., Dowding, K. J., Paez, T. L., Urbina, A., Romero, V. J., … Hills, R. G. (2001). Description of the sandia validation metrics project (SAND2001-1339). Retrieved from Albuquerque, NM.

    Google Scholar 

  • Trucano, T. G., Pilch, M., & Oberkampf, W. L. (2002). General concepts for experimental validation of asci code applications (SAND2002-0341). Retrieved from Albuquerque, NM.

    Google Scholar 

  • Trucano, T. G., Post, D. E., Pilch, M., & Oberkampf, W. L. (2005). Software engineering intersection with verification and validation of higher performance computational science software: Some observations (SAND2005-3662P). Retrieved from Albuquerque, NM.

    Google Scholar 

  • Trucano, T. G., Swiler, L. P., Igusa, T., Oberkampf, W. L., & Pilch, M. (2006). Calibration, validation, and sensitivity analysis: What’s what. Reliability Engineering and System Safety, 91(10–11), 1331–1357.

    Article  Google Scholar 

  • Verfurth, R. (2013). A posteriori error estimation techniques for finite element methods. Oxford, UK: Oxford University Press.

    Book  MATH  Google Scholar 

  • Vose, D. (2008). Risk analysis: A quantitative guide (3rd ed.). New York: Wiley.

    MATH  Google Scholar 

  • Voyles, I. T., & Roy, C. J. (2015). Evaluation of model validation techniques in the presence of aleatory and epistemic input uncertainties. Paper presented at the American Institute of Aeronautics and Astronautics SciTech Conference, Kissimmee, FL.

    Google Scholar 

  • Wang, S., Chen, W., & Tsui, K.-L. (2009). Bayesian validation of computer models. Technometrics, 51(4), 439–451.

    Article  MathSciNet  Google Scholar 

  • Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.

    Book  MATH  Google Scholar 

  • Wilks, D. S. (2011). Statistical methods in the atmospheric sciences (3rd ed.). Amsterdam: Elsevier.

    Google Scholar 

  • Wolpert, R. L. (2004). A conversation with James O. Berger. Statistical Science, 19(1), 205–218.

    Article  MathSciNet  MATH  Google Scholar 

  • Ziliak, S. T., & McCloskey, D. N. (2008). The cult of statistical significance: How the standard error costs us jobs, justice, and lives. Ann Arbor, Michigan: University of Michigan Press.

    MATH  Google Scholar 

Download references

Acknowledgements

I thank Drs. Timothy Trucano, Patrick Roache, and Theodore Kneupper for carefully reviewing an earlier version of this chapter and providing many helpful suggestions for improvements and clarifications.

Author information

Authors and Affiliations

  1. Sandia National Laboratories - retired, Albuquerque, NM, USA

    William L. Oberkampf

Authors
  1. William L. Oberkampf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William L. Oberkampf .

Editor information

Editors and Affiliations

  1. University of Bern, Bern, Switzerland

    Claus Beisbart

  2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Nicole J. Saam

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Oberkampf, W.L. (2019). Simulation Accuracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective. In: Beisbart, C., Saam, N. (eds) Computer Simulation Validation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-70766-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70766-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70765-5

  • Online ISBN: 978-3-319-70766-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation