Some Criterion-Based Experimental Designs

  • Thomas J. Santner
  • Brian J. Williams
  • William I. Notz
Part of the Springer Series in Statistics book series (SSS)


Chapter  5 considered designs that attempt to spread observations “evenly” throughout the experimental region. Such designs were called space-filling designs. Recall that one rationale for using a space-filling design is the following. If it is believed that interesting features of the true model are just as likely to be in one part of the input region as another, observations should be taken in all portions of the input region. There are many heuristic criteria for producing designs that might be considered space-filling; several of these were discussed in Chap.  5. However none of the methods was tied to a statistical justification, and no single criterion was singled out as best.


  1. Balling R (2003) The maximin fitness function: A multiobjective city and regional planning. In: Fonseca C, Fleming P, Zitzler E, Deb K, Thiele L (eds) Evolutionary multi-criterion optimization. Springer, Berlin, pp 1–15zbMATHGoogle Scholar
  2. Bautista DC (2009) A sequential design for approximating the Pareto front using the expected Pareto improvement function. PhD thesis, Department of Statistics, The Ohio State University, Columbus, OHGoogle Scholar
  3. Bernardo MC, Buck RJ, Liu L, Nazaret WA, Sacks J, Welch WJ (1992) Integrated circuit design optimization using a sequential strategy. IEEE Trans Comput Aided Des 11:361–372CrossRefGoogle Scholar
  4. Coello Coello CA, Lamont GB, Van Veldhuizen DA (2006) Evolutionary algorithms for solving multi-objective problems (genetic and evolutionary computation). Springer, New York, NYzbMATHGoogle Scholar
  5. Currin C, Mitchell TJ, Morris MD, Ylvisaker D (1991) Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J Am Stat Assoc 86:953–963MathSciNetCrossRefGoogle Scholar
  6. Emmerich MT, Giannakoglou KC, Naujoks B (2006) Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10(4):421–439CrossRefGoogle Scholar
  7. Fonseca CM, Paquete L, López-Ibáñez M (2006) An improved dimension -sweep algorithm for the hypervolume indicator. In: IEEE congress on evolutionary computation. IEEE Press, New York, NY, pp 1157–1163Google Scholar
  8. Forrester A, Sóbester A, Keane A (2007) Multi-fidelity optimization via surrogate modeling. Proc R Soc A 463(2088):3251–3269CrossRefGoogle Scholar
  9. Gramacy RB, Gray GA, Le Digabel S, Lee HKH, Ranjan P, Wells G, Wild SM (2016) Modeling an augmented Lagrangian for blackbox constrained optimization (with discussion). Technometrics 58(1):1–29MathSciNetCrossRefGoogle Scholar
  10. Johnson ME, Moore LM, Ylvisaker D (1990) Minimax and maximin distance designs. J Stat Plann Inf 26:131–148MathSciNetCrossRefGoogle Scholar
  11. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13:455–492MathSciNetCrossRefGoogle Scholar
  12. Keane AJ (2006) Statistical improvement criteria for use in multiobjective design optimization. AIAA J 44:879–891CrossRefGoogle Scholar
  13. Lam CQ, Notz WI (2008) Sequential adaptive designs in computer experiments for response surface model fit. Stat Appl 6:207–233Google Scholar
  14. Leatherman ER, Dean AM, Santner TJ (2014) Computer experiment designs via particle swarm optimization. In: Melas VB, Mignani S, Monari P, Salmaso L (eds) Topics in statistical simulation: research papers from the 7th international workshop on statistical simulation, vol 114. Springer, Berlin, pp 309–317Google Scholar
  15. Leatherman ER, Dean AM, Santner TJ (2017) Designing combined physical and computer experiments to maximize prediction accuracy. Comput Stat Data Anal 113:346–362MathSciNetCrossRefGoogle Scholar
  16. Leatherman ER, Santner TJ, Dean AM (2018) Computer experiment designs for accurate prediction. Stat Comput 28:739–751MathSciNetCrossRefGoogle Scholar
  17. Lindley DV (1956) On a measure of information provided by an experiment. Ann Math Stat 27:986–1005MathSciNetCrossRefGoogle Scholar
  18. Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376MathSciNetCrossRefGoogle Scholar
  19. Loeppky JL, Moore LM, Williams BJ (2010) Batch sequential designs for computer experiments. J Stat Plann Inf 140(6):1452–1464MathSciNetCrossRefGoogle Scholar
  20. Mitchell TJ (1974) An algorithm for the construction of “D-optimal” experimental designs. Technometrics 16:203–210MathSciNetzbMATHGoogle Scholar
  21. Mitchell TJ, Scott DS (1987) A computer program for the design of group testing experiments. Commun Stat Theory Methods 16:2943–2955CrossRefGoogle Scholar
  22. Mockus J, Tiešis V, Žilinskas A (1978) The application of Bayesian methods for seeking the extremum. In: Dixon LCW, Szego GP (eds) Towards global optimisation, vol 2. North Holland, Amsterdam, pp 117–129Google Scholar
  23. Mockus A, Mockus J, Mockus L (1994) Bayesian approach adapting stochastic and heuristic methods of global and discrete optimization. Informatica 5:123–166MathSciNetzbMATHGoogle Scholar
  24. Nocedal J, Wright S (2006) Numerical optimization. Springer series in operations research and financial engineering. Springer, New York, NYGoogle Scholar
  25. Notz WI (2015) Expected improvement designs. In: Bingham D, Dean AM, Morris M, Stufken J (eds) Handbook of design and analysis of experiments. Chapman and Hall, New York, NY, pp 675–716Google Scholar
  26. Picard R, Williams B (2013) Rare event estimation for computer models. Am Stat 67(1):22–32MathSciNetCrossRefGoogle Scholar
  27. Picheny V, Ginsbourger D, Richet Y, Caplin G (2013) Quantile-based optimization of noisy computer experiments with tunable precision. Technometrics 55(1):2–13MathSciNetCrossRefGoogle Scholar
  28. Pukelsheim F (1993) Optimal design of experiments. Wiley, New York, NYzbMATHGoogle Scholar
  29. Ranjan P, Bingham D, Michailidis G (2008) Sequential experiment design for contour estimation from complex computer codes. Technometrics 50(4):527–541MathSciNetCrossRefGoogle Scholar
  30. Roy S, Notz WI (2014) Estimating percentiles in computer experiments: a comparison of sequential-adaptive designs and fixed designs. Stat Theory Practice 8:12–29MathSciNetCrossRefGoogle Scholar
  31. Sacks J, Schiller SB, Welch WJ (1989a) Design for computer experiments. Technometrics 31:41–47MathSciNetCrossRefGoogle Scholar
  32. Schonlau M (1997) Computer experiments and global optimization. PhD thesis, Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, ONGoogle Scholar
  33. Schonlau M, Welch WJ, Jones DR (1998) Global versus local search in constrained optimization of computer models. In: Flournoy N, Rosenberger WF, Wong WK (eds) New developments and applications in experimental design, vol 34. Institute of Mathematical Statistics, pp 11–25Google Scholar
  34. Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27:379–423MathSciNetCrossRefGoogle Scholar
  35. Shewry MC, Wynn HP (1987) Maximum entropy sampling. J Appl Stat 14:165–170CrossRefGoogle Scholar
  36. Svenson JD (2011) Computer experiments: multiobjective optimization and sensitivity analysis. PhD thesis, Department of Statistics, The Ohio State University, Columbus, OHGoogle Scholar
  37. Svenson J, Santner T (2016) Multiobjective optimization of expensive-to-evaluate deterministic computer simulator models. Comput Stat Data Anal 94:250–264MathSciNetCrossRefGoogle Scholar
  38. Weaver BP, Williams BJ, Anderson-Cook CM, Higdon DM (2016) Computational enhancements to Bayesian design of experiments using Gaussian processes. Bayesian Anal 11(1):191–213MathSciNetCrossRefGoogle Scholar
  39. Welch WJ, Buck RJ, Sacks J, Wynn HP, Mitchell TJ, Morris MD (1992) Screening, predicting, and computer experiments. Technometrics 34:15–25CrossRefGoogle Scholar
  40. Williams BJ, Santner TJ, Notz WI (2000) Sequential design of computer experiments to minimize integrated response functions. Stat Sinica 10:1133–1152MathSciNetzbMATHGoogle Scholar
  41. Williams BJ, Santner TJ, Notz WI, Lehman JS (2010) Sequential design of computer experiments for constrained optimization. In: Kneib T, Tutz G (eds) Statistical modelling and regression structures: Festschrift in Honour of Ludwig Fahrmeir. Physica-Verlag, Heidelberg, pp 449–472CrossRefGoogle Scholar
  42. Zitzler E, Knowles J, Thiele L (2008) Quality assessment of Pareto set approximations. In: Branke J, Deb K, Miettinen K, Slowinski R (eds) Multiobjective optimization: interactive and evolutionary approaches. Springer, Berlin, pp 373–404CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Thomas J. Santner
    • 1
  • Brian J. Williams
    • 2
  • William I. Notz
    • 1
  1. 1.Department of StatisticsThe Ohio State UniversityColumbusUSA
  2. 2.Statistical Sciences GroupLos Alamos National LaboratoryLos AlamosUSA

Personalised recommendations