Surrogate Models for Mixed Discrete-Continuous Variables

  • Laura P. Swiler
  • Patricia D. Hough
  • Peter Qian
  • Xu Xu
  • Curtis Storlie
  • Herbert Lee
Part of the Studies in Computational Intelligence book series (SCI, volume 539)


Large-scale computational models have become common tools for analyzing complex man-made systems. However, when coupled with optimization or uncertainty quantification methods in order to conduct extensive model exploration and analysis, the computational expense quickly becomes intractable. Furthermore, these models may have both continuous and discrete parameters. One common approach to mitigating the computational expense is the use of response surface approximations. While well developed for models with continuous parameters, they are still new and largely untested for models with both continuous and discrete parameters. In this work, we describe and investigate the performance of three types of response surfaces developed for mixed-variable models: Adaptive Component Selection and Shrinkage Operator, Treed Gaussian Process, and Gaussian Process with Special Correlation Functions. We focus our efforts on test problems with a small number of parameters of interest, a characteristic of many physics-based engineering models. We present the results of our studies and offer some insights regarding the performance of each response surface approximation method.


Mean Square Error Reproduce Kernel Hilbert Space Average Mean Square Error Smoothing Spline Latin Hypercube Design 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adams, B.M., Bohnhoff, W.J., Dalbey, K.R., Eddy, J.P., Eldred, M.S., Gay, D.M., Hough, P.D., Swiler, L.P.: DAKOTA, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: Version 5.1 user’s manual. Technical Report SAND2010-2183, Sandia National Laboratories, Albuquerque, NM (2010),
  2. 2.
    Berlinet, A., Thomas-Agnan, C.: Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, Norwell (2004)CrossRefzbMATHGoogle Scholar
  3. 3.
    Breiman, L., Friedman, J.H., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth, Belmont (1984)zbMATHGoogle Scholar
  4. 4.
    Broderick, T., Gramacy, R.B.: Classification and categorical inputs with treed Gaussian process models. Journal of Classification 28(2), 244–270 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Chipman, H., George, E., McCulloch, R.: Bayesian CART Model Search (with discussion). Journal of the American Statistical Association 93, 930–960Google Scholar
  6. 6.
    Craven, P., Wahba, G.: Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation. Numerical Mathematics 31, 377–403 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Eubank, R.L.: Nonparametric Regression and Spline Smoothing. CRC Press, London (1999)zbMATHGoogle Scholar
  8. 8.
    Gramacy, R.B., Lee, H.K.H.: Bayesian treed gaussian process models with an application to computer modeling. Journal of the American Statistical Association 103, 1119–1130 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Gramacy, R.B., Lee, H.K.H.: Gaussian processes and limiting linear models. Computational Statistics and Data Analysis 53, 123–136 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  10. 10.
    Gramacy, R.B., Taddy, M.: Categorical inputs, sensitivity analysis, optimization and importance tempering with tgp version 2, an r package for treed gaussian process models. Technical report, R manual (2009),
  11. 11.
    Gu, C.: Smoothing Spline ANOVA Models. Springer, New York (2002)CrossRefzbMATHGoogle Scholar
  12. 12.
    Hastie, T., Tibshirani, R.J.: Generalized Additive Models. Chapman & Hall/CRC, London (1990)zbMATHGoogle Scholar
  13. 13.
    Kennedy, M.C., O’Hagan, A.: Bayesian calibration of computer models (with discussion). Journal of the Royal Statistical Society B 63, 425–464 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, vol. 2(12), pp. 1137–1143 (1995)Google Scholar
  15. 15.
    Lin, Y., Zhang, H.: Component selection and smoothing in smoothing spline analysis of variance models. Annals of Statistics 34(5), 2272–2297 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Lophaven, S.N., Neilson, H.B., Sondergaard, J.: Dace - a matlab kriging toolbox. Technical report (2009),
  17. 17.
    McDaniel, W.R., Ankenman, B.E.: A response surface test bed. Quality and Reliability Engineering International 16, 363–372 (2000)CrossRefGoogle Scholar
  18. 18.
    Neter, J., Wasserman, W., Kutner, M.H.: Applied Linear Statistical Models, 2nd edn. Irwin (1985)Google Scholar
  19. 19.
    Qian, P.Z.G.: Sliced latin hypercube designs (2011) (submitted)Google Scholar
  20. 20.
    Qian, P.Z.G., Wu, H., Wu, C.F.J.: Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics 50(3), 383–396 (2008)CrossRefMathSciNetGoogle Scholar
  21. 21.
    Rebonato, R., Jackel, P.: The most general methodology for creating a valid correlation matrix for risk management and option pricing purposes. The Journal of Risk 2, 17–27 (1999)Google Scholar
  22. 22.
    Reich, B.J., Storlie, C.B., Bondell, H.D.: Variable selection in bayesian smoothing spline anova models: Application to deterministic computer codes. Technometrics 51(2), 110–120 (2009)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Sacks, J., Welch, W.J., Mitchel, T.J., Wynn, H.P.: Design and analysis of computer experiments. Statistical Science 4(4), 409–435 (1989)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global sensitivity analysis, The Primer. Wiley and Sons (2008)Google Scholar
  25. 25.
    Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments. Springer, New York (2003)CrossRefzbMATHGoogle Scholar
  26. 26.
    Schimek, M. (ed.): Smoothing and Regression: Approaches, Computation, and Application. John Wiley, New York (2000)Google Scholar
  27. 27.
    Simpson, T.W., Toropov, V., Balabanov, V., Viana, F.A.C.: Design and analysis of computer experiments in multidisciplinary design optimization: A review of how far we have come – or not. In: Proceedings of the 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Victoria, British Columbia, Canada (September 2008) AIAA Paper 2008-5802Google Scholar
  28. 28.
    Storlie, C.B., Bondell, H.D., Reich, B.J.: A locally adaptive penalty for estimation of functions with varying roughness. Journal of Computational and Graphical Statistics 19(3), 569–589 (2010)CrossRefMathSciNetGoogle Scholar
  29. 29.
    Storlie, C.B., Bondell, H.D., Reich, B.J., Zhang, H.H.: Surface estimation, variable selection, and the nonparametric oracle property. Statistica Sinica 21(2), 679–705 (2010)CrossRefMathSciNetGoogle Scholar
  30. 30.
    Storlie, C.B., Helton, J.C.: Multiple predictor smoothing methods for sensitivity analysis: Description of techniques. Reliabililty Engineering and System Safety 93(1), 28–54 (2008)CrossRefGoogle Scholar
  31. 31.
    Storlie, C.B., Swiler, L.P., Helton, J.C., Sallaberry, C.J.: Implementation and evaluation of nonparametric regression procedures for sensitivity analysis of computationally demanding models. Reliability Engineering and System Safety 94(11), 1735–1763 (2009)CrossRefGoogle Scholar
  32. 32.
    Swiler, L.P., Wyss, G.D.: A user’s guide to Sandia’s latin hypercube sampling software: LHS UNIX library and standalone version. Technical Report SAND04-2439, Sandia National Laboratories, Albuquerque, NM (July 2004)Google Scholar
  33. 33.
    Swiler, L.P., Hough, P.D., Qian, P., Xu, X., Storlie, C.B., Lee, H.: Surrogate models for mixed discrete-continuous variables. Technical Report SAND2012-0491, Sandia National Laboratories, Albuquerque, NM (August 2012)Google Scholar
  34. 34.
    Tibshirani, R.J.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society B 58, 267–288 (1996)zbMATHMathSciNetGoogle Scholar
  35. 35.
    Viana, F.A.C., Haftka, R.T., Steffan Jr., V.: Multiple surrogates: How cross-validation errors can help us obtain the best predictor. Structural and Multidisciplinary Optimization 39(4), 439–457 (2009)CrossRefGoogle Scholar
  36. 36.
    Viana, F.A.C., Haftka, R.T., Steffan Jr., V., Butkewitsch, S., Leal, M.F.: Ensemble of surrogates: a framework based on minimization of the mean integrated square error. In: Proceedings of the 49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Schaumburg, IL (April 2008) AIAA Paper 2008–1885Google Scholar
  37. 37.
    Wahba, G.: Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics (1990)Google Scholar
  38. 38.
    Wu, Z.: Multivariate compactly supported positive definite radial functions. Advances in Computational Mathematics 4, 283–292 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  39. 39.
    Zhou, Q., Qian, P.Z.G., Zhou, S.: A simple approach to emulation for computer models with qualitative and quantitative factors. Technometrics 53, 266–273 (2011)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Laura P. Swiler
    • 1
    • 2
  • Patricia D. Hough
    • 1
    • 2
  • Peter Qian
    • 3
  • Xu Xu
    • 3
  • Curtis Storlie
    • 4
  • Herbert Lee
    • 5
  1. 1.Sandia National LaboratoriesAlbuequerqueNew Mexico
  2. 2.Sandia National LaboratoriesLivermoreUSA
  3. 3.University of Wisconsin-MadisonMadisonUSA
  4. 4.Los Alamos National LaboratoryLos AlamosNew Mexico
  5. 5.Univerity of CaliforniaSanta CruzUSA

Personalised recommendations