Abstract
Current quantification of margin and uncertainty (QMU) guidance lacks a consistent framework for communicating the credibility of analysis results. Recent efforts at providing QMU guidance have pushed for broadening the analyses supporting QMU results beyond extrapolative statistical models to include a more holistic picture of risk, including information garnered from both experimental campaigns and computational simulations. Credibility guidance would assist in the consideration of belief-based aspects of an analysis. Such guidance exists for presenting computational simulation-based analyses and is under development for the integration of experimental data into computational simulations (calibration or validation), but is absent for the ultimate QMU product resulting from experimental or computational analyses. A QMU credibility assessment framework comprised of five elements is proposed: requirement definitions and quantity of interest selection, data quality, model uncertainty, calibration/parameter estimation, and validation. Through considering and reporting on these elements during a QMU analysis, the decision-maker will receive a more complete description of the analysis and be better positioned to understand the risks involved with using the analysis to support a decision. A molten salt battery application is used to demonstrate the proposed QMU credibility framework.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Merriam-Webster Dictionary. https://www.merriam-webster.com/dictionary/credibility
National Research Council: Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile Committee on the Evaluation of Quantification of Margins and Uncertainties Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile. National Academies Press, Washington (2008)
Kaplan, S., Garrick, B.J.: On the quantitative definition of risk. Risk Anal. 1(1), 11–27 (1981)
Pilch, M., Trucano, T., Helton, J.: Ideas underlying quantification of margins and uncertainties (QMU): a white paper, Technical Report SAND2006-5001, Sandia National Laboratories (2006)
Newcomer, J.: A new approach to quantification of margins and uncertainties for physical simulation data, Technical Report SAND2012-7912, Sandia National Laboratories (2012)
Hund, L., Schroeder, B., Rumsey, K., Murchison, N.: Robust approaches to quantification of margin and uncertainty for sparse data, Technical Report SAND2017-9960, Sandia National Laboratories (2017)
Roy Dholakia, R., Sternthal, B.: Highly credible sources: persuasive facilitators or persuasive liabilities? J. Consum. Res. 3(4), 223–232 (1977)
Gass, R.H., Seiter, J.S.: Persuasion: Social Influence and Compliance Gaining. Routledge, Abingdon (2015)
Lev-Ari, S., Keysar, B.: Why don’t we believe non-native speakers? The influence of accent on credibility. J. Exp. Soc. Psychol. 46(6), 1093–1096 (2010)
Chaiken, S., Maheswaran, D.: Heuristic processing can bias systematic processing: effects of source credibility, argument ambiguity, and task importance on attitude judgment. J. Pers. Soc. Psychol. 66(3), 460 (1994)
Heesacker, M., Petty, R.E., Cacioppo, J.T.: Field dependence and attitude change: source credibility can alter persuasion by affecting message-relevant thinking. J. Personal. 51(4), 653–666 (1983)
Mehta, U., Romero, V., Eklund, D., Pearce, J., Keim, N.: The JANNAF simulation credibility guide on verification, uncertainty propagation and quantification, and validation. In: 53rd AIAA Aerospace Sciences Meeting, AIAA SciTech Forum (2015)
Oberkampf, W.L., Pilch, M., Trucano, T.G.: Predictive capability maturity model for computational modeling and simulation, Technical Report SAND2007-5948, Sandia National Laboratories (2007)
Beghini, L., Hough, P.: Sandia verification and validation challenge problem: a PCMM-based approach to assessing prediction credibility. J. Verif. Valid. Uncertain. Quantif. 1, 011002 (2016)
Schroeder, B., Silva, H., Smith, K.: Separability of mesh bias and parametric uncertainty for a full system thermal analysis. In: ASME 2018 Verification and Validation Symposium, pp. V001T04A003–V001T04A003. American Society of Mechanical Engineers, May 16, 2018
Oberkampf, W.L., Smith, B.: Assessment criteria for computational fluid dynamics model validation experiments. J. Verif. Valid. Uncertain. Quantif. 2, 031002 (2017)
Kieweg, S.L., Witkowski, W.R.: Experimental credibility and its role in model validation and decision making. In: Model Validation and Uncertainty Quantification, vol. 3, pp. 31–36. Springer, Cham (2019)
Hund, L., Schroeder, B., Rumsey, K., Huerta, G.: Distinguishing between model- and data-driven inferences for high reliability statistical predictions. Reliab. Eng. Syst. Saf. 180, 201–210 (2018)
Hemez, F., Atamturktur, H.S., Unal, C.: Defining predictive maturity for validated numerical simulations. Comput. Struct. 88(7–8), 497–505 (2010)
Pearl, J., Glymour, M., Jewell, N.P.: Causal Inference in Statistics: A Primer. Wiley, Hoboken (2016)
Bareinboim, E., Pearl, J.: Causal inference and the data-fusion problem. Proc. Natl. Acad. Sci. 113(27), 7345–7352 (2016)
Oberkampf, W., Barone, M.: Measures of agreement between computation and experiment: validation metrics. J. Comput. Phys. 217(1), 5–36 (2006)
Liu, Y., Arendt, P., Huang, H.: Toward a better understanding of model validation metrics. J. Mech. Des. 133(7), 071005 (2011)
Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B Methodol. 36, 111–147 (1974)
Efron, B.: Estimating the error rate of a prediction rule: improvement on cross-validation. J. Am. Stat. Assoc. 78(382), 316–331 (1983)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer Series in Statistics. Springer, New York (2009)
Zeng, Z., Di Maio, F., Zio, E., Kang, R.: A hierarchical decision-making framework for the assessment of the prediction capability of prognostic methods. Proc. Inst. Mech. Eng. O: J. Risk Reliab. 231(1), 36–52 (2017)
EricksonKirk, M., et al.: Sensitivity studies of the probabilistic fracture mechanics model used in FAVOR version 03.1. In: NUREG-1808, US Nuclear Regulatory Commission, ADAMS ML, vol. 61580349 (2004)
Coles, S., Bawa, J., Trenner, L., Dorazio, P.: An Introduction to Statistical Modeling of Extreme Values, vol. 208. Springer, London (2001)
Acknowledgements
Review
We would like to thank John R. Lewis and Aubrey C. Eckert-Gallup for their helpful comments that allowed us to refine and improve this paper.
Funding
This work was supported by a Sandia National Laboratories Laboratory Directed Research and Development (LDRD) grant. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2018-11643 C.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Society for Experimental Mechanics, Inc.
About this paper
Cite this paper
Schroeder, B.B., Hund, L., Kittinger, R.S. (2020). The Need for Credibility Guidance for Analyses Quantifying Margin and Uncertainty. In: Barthorpe, R. (eds) Model Validation and Uncertainty Quantification, Volume 3. Conference Proceedings of the Society for Experimental Mechanics Series. Springer, Cham. https://doi.org/10.1007/978-3-030-12075-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-12075-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-12074-0
Online ISBN: 978-3-030-12075-7
eBook Packages: EngineeringEngineering (R0)