Abstract
This tutorial reviews the design and analysis of simulation experiments. These experiments may have various goals: validation , prediction, sensitivity analysis, optimization (possibly robust), and risk or uncertainty analysis . These goals may be realized through metamodels. Two types of metamodels are the focus of this tutorial: (i) low-order polynomial regression , and (ii) Kriging (or Gaussian processes). The type of metamodel guides the design of the experiment; this design fixes the input combinations of the simulation model. However, before a regression or Kriging metamodel is applied, the many inputs of the underlying realistic simulation model should be screened; the tutorial focuses on sequential bifurcation . Optimization of the simulated system may use either a sequence of low-order polynomials—known as response surface methodology—or Kriging models fitted through sequential designs. Finally, “robust” optimization should account for uncertainty in simulation inputs. The tutorial includes references to earlier Winter Simulation Conference papers.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ankenman B, Nelson B, Staum J (2010) Stochastic Kriging for Simulation Metamodeling. Oper Res 58(2):371–382
Barton RR (1997) Design of experiments for fitting subsystem metamodels. In: Proceedings of the 1997 winter simulation conference, pp 303–310
Bertsimas D, Mišić VV (2017) Robust product line design. Oper Res 65(1):19–37
Biles BE, van Beers WCM, Kleijnen JPC, van Nieuwenhuyse I (2007) Kriging metamodels in constrained simulation optimization: an exploratory study. In: Proceedings of the 2007 winter simulation conference, pp 355–362
Binois M, Gramacy RB, Ludkovskiz M (2016) Practical heteroskedastic Gaussian process modeling for large simulation experiments. ArXiv, 17 Nov 2016
Chen X Kim K-K (2013) Building metamodels for quantile-based measures using sectioning. In: Proceedings of the 2013 winter simulation conference, pp 521–532
Dellino G, Kleijnen JPC, Meloni C (2009) Robust simulation-optimization using metamodels. In: Proceedings 2009 winter simulation conference, pp 540–550
Dellino G, Kleijnen JPC, Meloni C (2010) Parametric and distribution-free bootstrapping in robust simulation optimization. In: Proceedings 2010 winter simulation conference, pp 1283–1294
Erickson CB, Sanchez SM, Ankenman BE (2016) Comparison of Gaussian process modeling software. In: WSC 2016 proceedings, pp 3692–3693
Frazier PI, Jedynak B, Chen L (2012) Sequential screening: a Bayesian dynamic programming analysis. In: Proceedings 2012 winter simulation conference, Berlin, pp 555–566
Govindan E (2016) Yosida approximations of stochastic differential equations in infinite dimensions and applications. Springer
Gramacy RB (2015) laGP: large-scale spatial modeling via local approximate Gaussian processes in R. J Stat Softw (available as a vignette in the laGP package)
Jalali H, Van Nieuwenhuyse I (2015) Simulation optimization in inventory replenishment: a classification. IIE Trans 47(11):1217–1235
Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13:455–492
Kleijnen JPC (1995) Case study: statistical validation of simulation models. Eur J Oper Res 87(1):21–34
Kleijnen JPC (2015) Design and analysis of simulation experiments, 2nd edn. Springer
Kleijnen JPC (2017) Regression and kriging metamodels with their experimental designs in simulation: a review. Eur J Oper Res 256:1–16
Kleijnen JPC, Mehdad E (2013) Conditional simulation for efficient global optimization. In: Proceedings 2013 winter simulation conference, pp 969–979
Kleijnen JPC, Mehdad E, van Beers WCM (2012) Convex and monotonic bootstrapped kriging. In: Proceedings of the 2012 winter simulation conference, pp 543–554
Kleijnen JPC, Shi W (2017) Sequential probability ratio tests: conservative and robust. In: CentER discussion paper, vol 2017-001. CentER, Center for economic research, Tilburg
Law AM (2015) Simulation modeling and analysis, 5th edn. McGraw-Hill, Boston
Liu Z, Rexachs D, Epelde F, Luque E (2017) A simulation and optimization based method for calibrating agent-based emergency department models under data scarcity. Comput Ind Eng 103:300–309
Lophaven SN, Nielsen HB, Sondergaard J (2002) DACE: a matlab kriging toolbox, version 2.0. IMM Technical University of Denmark, Kongens Lyngby
Mitchell TJ, Morris MD (1992) The spatial correlation function approach to response surface estimation. In: Proceedings of the 2012 winter simulation conference, pp 565–571
Montgomery DC (2009) Design and analysis of experiments, 7th edn. Wiley, Hoboken, NJ
Myers RH, Montgomery DC, Anderson-Cook CM (2009) Response surface methodology: process and product optimization using designed experiments, 3rd edn. Wiley, New York
Rosen SL, Slater D, Beeker E, Guharay S, Jacyna G (2015) Critical infrastructure network analysis enabled by simulation metamodeling. In: Proceedings of the 2015 winter simulation conference, pp 2436–2447
Ryzhov IO, Chen Y (2017) Bayesian belief models in simulation-based optimization. In: Advances in modeling and simulation: seminal research from 50 years of winter simulation conferences, pp 181–217
Sargent RG, Goldsman DM, Yaacoub T (2016) A tutorial on the operational validation of simulation models. In: Proceedings of the 2016 winter simulation conference, pp 163–177
Shi W, Kleijnen JPC (2017) Testing the assumptions of sequential bifurcation in factor screening, vol 2017-06. Discussion paper, Center for economic research, Tilburg
Simpson TW, Booker AJ, Ghosh D, Giunta AA, Koch PN, Yang R-J (2004) Approximation methods in multidisciplinary analysis and optimization: a panel discussion. Struct Multidiscip Optim 27(5):302–313
Song E, Nelson BL (2017) Input model risk: where we stand and where we are going. In: Tolk A, Fowler J, Shao G, Yücesan E (eds) Advances in modeling and simulation: seminal research from 50 years of winter simulation conferences, pp 63–80
Timm IJ, Lorig F (2015) A survey on methodological aspects of computer simulation as research technique. In: Proceedings of the 2015 winter simulation conference, pp 2704–2715
Uhrmacher A, Brailsford S, Liu J, Rabe M, Tolk A (2016) Panel—reproducible research in discrete event simulation—a must or rather a maybe? In: Proceedings of the 2016 winter simulation conference, pp 1301–1315
Ulaganathan S, Couckuyt I, Dhaene T, Laermans E (2014) On the use of gradients in kriging surrogate models. In: Proceedings of the 2014 winter simulation conference, pp 2692–2701
Van Beers WCM, Kleijnen JPC (2004) Kriging interpolation in simulation: a survey. In: Proceedings of the 2004 winter simulation conference, pp 113–121
Van Ham G, Rotmans J, Kleijnen JPC (1992) Techniques for sensitivity analysis of simulation models: a case study of the CO2 greenhouse effect. Simulation 58(6):410–417
Wagner H (1975) Principles of operations research; with applications to managerial decisions, 2nd edn. Prentice-Hall Inc, Englewood Cliffs, New Jersey
Wan H, Ankenman BE, Nelson BL (2010) Improving the efficiency and efficacy of controlled sequential bifurcation for simulation factor screening. INFORMS J Comput 22(3):482–492
Yanikoğlu I, den Hertog D, Kleijnen JPC (2016) Robust dual-response optimization. IIE Trans Ind Eng Res Dev 48(3):298–312
Zeigler BP, Praehofer H, Kim TG (2000) Theory of modeling and simulation, 2nd edn. Academic, San Diego
Zhou E, Wu D (2017) Simulation optimization under input uncertainty. In: Tolk A, Fowler J, Shao G, Yücesan E (eds) Advances in modeling and simulation: seminal research from 50 years of winter simulation conferences, pp 219–247
Acknowledgements
I thank the editors for inviting me to contribute a chapter to this book, commemorating the 50th anniversary of the Winter Simulation Conferences.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG (outside the USA)
About this chapter
Cite this chapter
Kleijnen, J.P.C. (2017). Design and Analysis of Simulation Experiments: Tutorial. In: Tolk, A., Fowler, J., Shao, G., Yücesan, E. (eds) Advances in Modeling and Simulation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-64182-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-64182-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-64181-2
Online ISBN: 978-3-319-64182-9
eBook Packages: Computer ScienceComputer Science (R0)