Skip to main content
SpringerLink
Log in

Verification Methods for Surrogate Models

  • Chapter
  • First Online:
Surrogate Model-Based Engineering Design and Optimization

Part of the book series: Springer Tracts in Mechanical Engineering ((STME))

Abstract

A surrogate model built based on a limited number of sample points will inevitably have large prediction uncertainty. Applying such imprecise surrogate models in design and optimization may lead to misleading predictions or optimal solutions located in unfeasible regions (Picheny in Improving accuracy and compensating for uncertainty in surrogate modeling. University of Florida, Gainesville, 2009). Therefore, verifying the accuracy of a surrogate model before using it can ensure the reliability of the design.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Acar E (2010) Various approaches for constructing an ensemble of metamodels using local measures. Struct Multidiscip Optim 42:879–896

    Article  Google Scholar 

  • Acar E (2015) Effect of error metrics on optimum weight factor selection for ensemble of metamodels. Expert Syst Appl 42:2703–2709

    Article  Google Scholar 

  • Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37:279–294

    Article  Google Scholar 

  • Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79

    Article  MathSciNet  MATH  Google Scholar 

  • Barrett JP (1974) The coefficient of determination—some limitations. Am Stat 28:19–20

    MATH  Google Scholar 

  • Bhattacharyya B (2018) A critical appraisal of design of experiments for uncertainty quantification. Arch Comput Methods Eng 25:727–751

    Article  MathSciNet  MATH  Google Scholar 

  • Boopathy K, Rumpfkeil MP (2014) Unified framework for training point selection and error estimation for surrogate models. AIAA J 53:215–234

    Article  Google Scholar 

  • Borra S, Di Ciaccio A (2010) Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods. Comput Stat Data Anal 54:2976–2989

    Article  MathSciNet  MATH  Google Scholar 

  • Breiman L (1996) Heuristics of instability and stabilization in model selection. Ann Stat 24:2350–2383

    Article  MathSciNet  MATH  Google Scholar 

  • Burman P (1989) A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika 76:503–514

    Article  MathSciNet  MATH  Google Scholar 

  • Chai T, Draxler RR (2014) Root mean square error (RMSE) or mean absolute error (MAE)?–Arguments against avoiding RMSE in the literature. Geosci Model Dev 7:1247–1250

    Article  Google Scholar 

  • Du Q, Faber V, Gunzburger M (1999) Centroidal Voronoi tessellations: applications and algorithms. SIAM Rev 41:637–676

    Article  MathSciNet  MATH  Google Scholar 

  • Eason J, Cremaschi S (2014) Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput Chem Eng 68:220–232

    Article  Google Scholar 

  • Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 1–26

    Article  MathSciNet  MATH  Google Scholar 

  • Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78:316–331

    Article  MathSciNet  MATH  Google Scholar 

  • Efron B, Tibshirani R (1997) Improvements on cross-validation: the 632 + bootstrap method. J Am Stat Assoc 92:548–560

    MathSciNet  MATH  Google Scholar 

  • Efron B, Tibshirani RJ (1993a) An introduction to the bootstrap. Number 57 in monographs on statistics and applied probability. Chapman & Hall, New York

    Google Scholar 

  • Efron B, Tibshirani RJ (1993b) An Introduction to the Bootstrap: monographs on Statistics and Applied Probability, vol 57. Chapman and Hall/CRC, New York and London

    Book  MATH  Google Scholar 

  • Franses PH (2016) A note on the mean absolute scaled error. Int J Forecast 32:20–22

    Article  Google Scholar 

  • Fushiki T (2011) Estimation of prediction error by using K-fold cross-validation. Stat Comput 21:137–146

    Article  MathSciNet  MATH  Google Scholar 

  • Goel T, Hafkta RT, Shyy W (2009) Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Struct Multidiscip Optim 38:429

    Article  Google Scholar 

  • Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidiscip Optim 33:199–216

    Article  Google Scholar 

  • Goel T, Stander N (2009) Comparing three error criteria for selecting radial basis function network topology. Comput Methods Appl Mech Eng 198:2137–2150

    Article  MathSciNet  MATH  Google Scholar 

  • Grafton RQ (2012) Coefficient of determination. A dictionary of climate change and the environment. Edward Elgar Publishing Limited

    Google Scholar 

  • Gronau QF, Wagenmakers E-J (2018) Limitations of Bayesian leave-one-out cross-validation for model selection. Comput Brain Behav 1–11

    Google Scholar 

  • Hu J, Yang Y, Zhou Q, Jiang P, Shao X, Shu L, Zhang Y (2018) Comparative studies of error metrics in variable fidelity model uncertainty quantification. J Eng Des 29:512–538

    Article  Google Scholar 

  • Hyndman RJ, Koehler AB (2006) Another look at measures of forecast accuracy. Int J Forecast 22:679–688

    Article  Google Scholar 

  • Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodeling techniques under multiple modelling criteria. Struct Multidiscip Optim 23:1–13

    Article  Google Scholar 

  • Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. Montreal, Canada, Ijcai, pp 1137–1145

    Google Scholar 

  • Larson SC (1931) The shrinkage of the coefficient of multiple correlation. J Educ Psychol 22:45

    Article  Google Scholar 

  • Li Y (2010) Root mean square error. In: Salkind NJ (ed) Encyclopedia of research design. Sage Publications Inc., Thousand Oaks, CA, pp 1288–1289

    Google Scholar 

  • Li J, Heap AD (2011) A review of comparative studies of spatial interpolation methods in environmental sciences: performance and impact factors. Ecol Inform 6:228–241

    Article  Google Scholar 

  • Liu H, Cai J, Ong Y-S (2017) An adaptive sampling approach for kriging metamodeling by maximizing expected prediction error. Comput Chem Eng 106:171–182

    Article  Google Scholar 

  • Liu J, Han Z, Song W (2012) Comparison of infill sampling criteria in kriging-based aerodynamic optimization. In: 28th congress of the international council of the aeronautical sciences, pp 23–28

    Google Scholar 

  • Mao W, Xu J, Wang C, Dong L (2014) A fast and robust model selection algorithm for multi-input multi-output support vector machine. Neurocomputing 130:10–19

    Article  Google Scholar 

  • Meckesheimer M, Booker AJ, Barton RR, Simpson TW (2002) Computationally inexpensive metamodel assessment strategies. AIAA J 40:2053–2060

    Article  Google Scholar 

  • Mehmani A, Chowdhury S, Messac A (2015) Predictive quantification of surrogate model fidelity based on modal variations with sample density. Struct Multidiscip Optim 52:353–373

    Article  Google Scholar 

  • Miller RG (1974) The jackknife-a review. Biometrika 61:1–15

    MathSciNet  MATH  Google Scholar 

  • Nagelkerke NJ (1991) A note on a general definition of the coefficient of determination. Biometrika 78:691–692

    Article  MathSciNet  MATH  Google Scholar 

  • Nguyen HM, Couckuyt I, Knockaert L, Dhaene T, Gorissen D, Saeys Y (2011) An alternative approach to avoid overfitting for surrogate models. In: Proceedings of the winter simulation conference: winter simulation conference, pp 2765–2776

    Google Scholar 

  • Picheny V (2009) Improving accuracy and compensating for uncertainty in surrogate modeling. University of Florida, Gainesville

    Google Scholar 

  • Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28

    Article  Google Scholar 

  • Quenouille MH (1949) Approximate tests of correlation in time-series 3. In: Mathematical proceedings of the Cambridge Philosophical Society. Cambridge University Press, pp 483–484

    Google Scholar 

  • Renaud O, Victoria-Feser M-P (2010) A robust coefficient of determination for regression. J Stat Plan Inference 140:1852–1862

    Article  MathSciNet  MATH  Google Scholar 

  • Rodriguez JD, Perez A, Lozano JA (2010) Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Trans Pattern Anal Mach Intell 32:569–575

    Article  Google Scholar 

  • Romero DA, Marin VE, Amon CH (2015) Error metrics and the sequential refinement of kriging metamodels. J Mech Des 137:011402

    Article  Google Scholar 

  • Salkind NJ (2010) Encyclopedia of research design. Sage

    Google Scholar 

  • Sanchez E, Pintos S, Queipo NV (2008) Toward an optimal ensemble of kernel-based approximations with engineering applications. Struct Multidiscip Optim 36:247–261

    Article  Google Scholar 

  • Shao J (1993) Linear model selection by cross-validation. J Am Stat Assoc 88:486–494

    Article  MathSciNet  MATH  Google Scholar 

  • Shao J (1996) Bootstrap model selection. J Am Stat Assoc 91:655–665

    Article  MathSciNet  MATH  Google Scholar 

  • Shao J, Tu D (2012) The jackknife and bootstrap. Springer Science & Business Media

    Google Scholar 

  • Stone M (1974) Cross-validatory choice and assessment of statistical predictions. J R Stat Soc. Series B (Methodological) 111–147

    MathSciNet  MATH  Google Scholar 

  • Vehtari A, Gelman A, Gabry J (2017) Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput 27:1413–1432

    Article  MathSciNet  MATH  Google Scholar 

  • Viana FA, Haftka RT, Steffen V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39:439–457

    Article  Google Scholar 

  • Wang Y, Liu Q (2006) Comparison of Akaike information criterion (AIC) and Bayesian information criterion (BIC) in selection of stock–recruitment relationships. Fish Res 77:220–225

    Article  Google Scholar 

  • Willmott CJ, Matsuura K (2005) Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Res 30:79–82

    Article  Google Scholar 

  • Yanagihara H, Tonda T, Matsumoto C (2006) Bias correction of cross-validation criterion based on Kullback-Leibler information under a general condition. J Multivar Anal 97:1965–1975

    Article  MathSciNet  MATH  Google Scholar 

  • Yang Y (2007) Consistency of cross validation for comparing regression procedures. Ann Stat 35:2450–2473

    Article  MathSciNet  MATH  Google Scholar 

  • Ye P, Pan G, Dong Z (2018) Ensemble of surrogate based global optimization methods using hierarchical design space reduction. Struct Multidiscip Optim 58:537–554

    Article  MathSciNet  Google Scholar 

  • Zhao D, Xue D (2010) A comparative study of metamodeling methods considering sample quality merits. Struct Multidiscip Optim 42:923–938

    Article  Google Scholar 

  • Zhou Q, Shao X, Jiang P, Gao Z, Zhou H, Shu L (2016) An active learning variable-fidelity metamodelling approach based on ensemble of metamodels and objective-oriented sequential sampling. J Eng Des 27:205–231

    Article  Google Scholar 

  • Zhou Q, Wang Y, Choi S-K, Jiang P, Shao X, Hu J (2017) A sequential multi-fidelity metamodeling approach for data regression. Knowl-Based Syst 134:199–212

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China

    Ping Jiang

  2. School of Aerospace Engineering, Huazhong University of Science and Technology, Wuhan, Hubei, China

    Qi Zhou

  3. The State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China

    Xinyu Shao

Authors
  1. Ping Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyu Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ping Jiang .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Jiang, P., Zhou, Q., Shao, X. (2020). Verification Methods for Surrogate Models. In: Surrogate Model-Based Engineering Design and Optimization. Springer Tracts in Mechanical Engineering. Springer, Singapore. https://doi.org/10.1007/978-981-15-0731-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-0731-1_5

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-0730-4

  • Online ISBN: 978-981-15-0731-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation