The Dangers of Sparse Sampling for Uncertainty Propagation and Model Calibration

Conference paper
Part of the Conference Proceedings of the Society for Experimental Mechanics Series book series (CPSEMS)


Activities such as sensitivity analysis, statistical effect screening, uncertainty propagation, or model calibration have become integral to the Verification and Validation (V&V) of numerical models and computer simulations. Because these analyses involve performing multiple runs of a computer code, they can rapidly become computationally expensive. For example, propagating uncertainty with a 1,000 Monte Carlo samples wrapped around a finite element calculation that takes only 10 minutes to run requires seven days of single-processor time. An alternative is to combine a design of computer experiments to meta-modeling, and replace the potentially expensive computer simulation by a fast-running surrogate. The surrogate can then be used to estimate sensitivities, propagate uncertainty, and calibrate model parameters at a fraction of the cost it would take to wrap a sampling algorithm or optimization solver around the analysis code. In this publication, we focus on the dangers of using too sparsely populated design-of-experiments to propagate uncertainty or train a fast-running surrogate model. One danger for sensitivity analysis or calibration is to develop meta-models that include erroneous sensitivities. This is illustrated with a high-dimensional, non-linear mathematical function in which several parameter effects are statistically insignificant, therefore, mimicking a situation that is often encountered in practice. It is shown that using a sparse design of computer experiments leads to an incorrect approximation of the function. (Publication approved for unlimited, public release on November 4, 2009, LA-UR-09-7227, Unclassified.)


Markov Chain Monte Carlo Alamos National Laboratory Uncertainty Propagation Prediction Uncertainty Gaussian Process Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliographical References

  1. [1]
    Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P., ”Design and Analysis of Computer Experiments,“ Statistical Science, Vol. 4, 1989, pp. 409–423.MathSciNetMATHGoogle Scholar
  2. [2]
    Santner, T.J., Williams, B.J., Notz, W.I., The Design and Analysis of Computer Experiments, Springer–Verlag Publishers, 2003.Google Scholar
  3. [3]
    Maupin, R., Hylok, J., Rutherford, A., Anderson, M., ”Validation of a Threaded Assembly, Part I: Overview,“ 6th European Conference on Structural Dynamics, Paris, France, September 5–7, 2005.Google Scholar
  4. [4]
    Hylok, J., Rutherford, A., Maupin, R., Anderson, M., Groethe, M., ”Validation of a Threaded Assembly, Part II: Experiments,“ 6th European Conference on Structural Dynamics, Paris, France, September 5–7, 2005. (Los Alamos Technical Report LA-UR-05-0931.)Google Scholar
  5. [5]
    Rutherford, A., Maupin, R., Hylok, J., Anderson, M., ”Validation of a Threaded Assembly, Part III: Validation,“ 6th European Conference on Structural Dynamics, Paris, France, September 5–7, 2005. (Los Alamos Technical Report LA-UR-05-2500.)Google Scholar
  6. [6]
    Lam, K., Allen, D., Tippetts, T., ”Engineering Verification and Validation Assessment of Truchas for Induction Heating,“ Technical Report LA-UR-07-7134, Los Alamos National Laboratory, Los Alamos, New Mexico, October 2007.Google Scholar
  7. [7]
    Sigeti, D.E., Buescher, K.L., Vaughan, D.E., Cooley, J.H., Hemez, F.M., ”Simulation of Integrated Effects Tests for the Fiscal Year 2008 ASC Primary Verification and Validation Milestone,“ Technical Report LA-CP-08-1154, Los Alamos National Laboratory, Los Alamos, New Mexico, September 2008. (Not available for public release.)Google Scholar
  8. [8]
    Hemez, F.M., Atamturktur, S.H., Unal, C., ”Prediction with Quantified Uncertainty of Temperature and Rate Dependent Material Behavior,“ 11th AIAA Non-Deterministic Approaches Conference, Palm Springs, California, May 4–7, 2009. (Los Alamos Technical Report LA-UR-08-6741.)Google Scholar
  9. [9]
    Atamturktur, H.S., Lebensohn, R., Higdon, D., Williams, B., Hemez, F.M., Unal, C., ”Predicative Maturity: A Quantitative Metric to Optimize Complex Simulations via Systematic Experimental Validation,“ Technical Report LA-UR-09-7226, Los Alamos National Laboratory, Los Alamos, New Mexico, October 2007.Google Scholar
  10. [10]
    Hemez, F.M., ”A Technical Note on the Goodness-of-fit of Staistical Emulators,“ Technical Memorandum X-3:10-001-C, Los Alamos National Laboratory, Los Alamos, New Mexico, October 2009. (Not available for public release.)Google Scholar
  11. [11]
    Myers, R.H., Montgomery, D.C., Response Surface Methodology: Process and Product Optimization Using Designed Experiments, Wiley Inter-science Publishers, 1995.Google Scholar
  12. [12]
    Saltelli, A., Chan, K., Scott, M., Sensitivity Analysis, John Wiley & Sons Publishers, 2000.Google Scholar
  13. [13]
    McKay, M.D., Beckman, R.J., Conover, W.J., ”A Comparison of Three Methods For Selecting Values of Input Variables in the Analysis of Output From a Computer Code,“ Technometrics, Vol. 21, No. 2, 1979, pp. 239–245.MathSciNetMATHGoogle Scholar
  14. [14]
    Tang, B., ”Orthogonal Array-Based Latin Hypercubes,“ Journal of the American Statistical Association, Vol. 88, 1993, pp. 1392–1397.MATHGoogle Scholar
  15. [15]
    Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., ”Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models,“ Probability and Statistics Series, John Wiley & Sons Publishers, 2004.Google Scholar
  16. [16]
    Rosenbrock, H.H., ”An Automatic Method for Finding the Greatest or Least Value of a Function,“ Computer Journal, Vol. 3, 1960, pp. 175–184.MathSciNetGoogle Scholar
  17. [17]
    Lindman, H.R., Analysis of Variance in Complex Experimental Designs, W.H. Freeman & Co. Ltd Publishers, 1975.Google Scholar
  18. [18]
    Box, G.E., Hunter, J.S., Hunter, W.G., Statistics for Experimenters: Design, Innovation, and Discovery, Wiley-Interscience Publishers, 2 Edition, 2005.Google Scholar
  19. [19]
    Kennedy, M., O’Hagan, A., ”Predicting the Output From a Complex Computer Code When Fast Approximations Are Available,“ Biometrika, Vol. 87, 2000, pp. 1–13.MathSciNetMATHGoogle Scholar
  20. [20]
    Williams, B., Higdon, D., Gattiker, J., Moore, L., McKay, M., Keller-McNulty, S., ”Combining Experimental Data and Computer Simulations With an Application to Flyer Plate Experiments,“ Bayesian Analysis, Vol. 1, 2006, pp. 765–792.MathSciNetGoogle Scholar
  21. [21]
    Higdon, D., Gattiker, J., Williams, B., Rightley, M., ”Computer Model Calibration Using High-Dimensional Output,“ Journal of the American Statistical Association, Vol. 103, No. 482, June 2008, pp. 570–583.MathSciNetMATHCrossRefGoogle Scholar
  22. [22]
    Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller, A., Teller, E., ”Equations of State Calculations by Fast Computing Machines,“ Journal of Chemical Physics, Vol. 21, 1953, pp. 1087–1091.Google Scholar
  23. [23]
    Hastings, W.K., ”Monte Carlo Sampling Methods Using Markov Chains and Their Applications,“ Biometrika, Vol. 57, 1970, pp. 97–109.MATHGoogle Scholar
  24. [24]
    Oakley, J., O’Hagan, A., ”Probabilistic Sensitivity Analysis of Complex Models,“ Journal of the Royal Statistical Society, Vol. 66, 2004, pp. 751–769.MathSciNetMATHGoogle Scholar
  25. [25]
    Williams, B., Gattiker, J., ”Using the Gaussian Process Model for Simulation Analysis (GPM/SA) Code,“ Technical Report LA-UR-06-5431, Los Alamos National Laboratory, Los Alamos, NM, December 2006.Google Scholar
  26. [26]
    Bingham, D., Higdon, D., Williams, B., ”Gaussian Process Models for High Dimensional Computer Experiments,“ Working Technical Paper, Simon-Frasier University, Vancouver, Canada, September 2009.Google Scholar
  27. [27]
    Morris, M.D., ”Factorial Sampling Plans for Preliminary Computational Experiments,“ Technometrics, Vol. 33, 1991.Google Scholar
  28. [28]
    Campolongo, F., Cariboni, J., Saltelli, A., ”An Effective Screening Design for Sensitivity Analysis of Large Models,“ Environmental Modelling and Software, Elsevier Publishers, Vol. 22, 2007, pp. 1510–1518.Google Scholar

Copyright information

© Springer Science+Businees Media, LLC 2011

Authors and Affiliations

  1. 1.X-Division (X-3)Los Alamos National Laboratory, X-3Los AlamosUSA
  2. 2.Civil Engineering DepartmentClemson UniversityClemsonUSA
  3. 3.Los Alamos National LaboratoryLos AlamosUSA

Personalised recommendations