Skip to main content

Sensitivity Analysis and Uncertainty Quantification of State-Based Discrete-Event Simulation Models Through a Stacked Ensemble of Metamodels

  • Conference paper
  • First Online:
Quantitative Evaluation of Systems (QEST 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12289))

Included in the following conference series:

Abstract

Realistic state-based discrete-event simulation models are often quite complex. The complexity frequently manifests in models that (a) contain a large number of input variables whose values are difficult to determine precisely, and (b) take a relatively long time to solve.

Traditionally, models that have a large number of input variables whose values are not well-known are understood through the use of sensitivity analysis (SA) and uncertainty quantification (UQ). However, it can be prohibitively time consuming to perform SA and UQ.

In this work, we present a novel approach we developed for performing fast and thorough SA and UQ on a metamodel composed of a stacked ensemble of regressors that emulates the behavior of the base model. We demonstrate the approach using a previously published botnet model as a test case, showing that the metamodel approach is several orders of magnitude faster than the base model, more accurate than existing approaches, and amenable to SA and UQ.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    A solved Sudoku puzzle is an example of a Latin square.

  2. 2.

    This is similar to the classic “8 rooks” problem in chess.

  3. 3.

    For example, taking a sample at every cell along the diagonal is a valid Latin hypercube sample sequence in two dimensions.

  4. 4.

    The \(\mu ^*\) values and the feature importance errors were rounded to the nearest integer, and all other values in the table were rounded to the nearest hundredth.

References

  1. Campolongo, F., Cariboni, J., Saltelli, A.: An effective screening design for sensitivity analysis of large models. Environ. Model. Softw. 22(10), 1509–1518 (2007)

    Article  Google Scholar 

  2. Cunha, A., Nasser, R., Sampaio, R., Lopes, H., Breitman, K.: Uncertainty quantification through the Monte Carlo method in a cloud computing setting. Comput. Phys. Commun. 185(5), 1355–1363 (2014)

    Article  Google Scholar 

  3. Eisenhower, B., O’Neill, Z., Narayanan, S., Fonoberov, V.A., Mezić, I.: A methodology for meta-model based optimization in building energy models. Energy Build. 47, 292–301 (2012)

    Article  Google Scholar 

  4. Herman, J., Usher, W.: SALib: an open-source Python library for sensitivity analysis. J. Open Source Softw. 2(9) (2017). https://doi.org/10.21105/joss.00097

  5. Iman, R.L., Helton, J.C., Campbell, J.E.: An approach to sensitivity analysis of computer models: Part I - introduction, input variable selection and preliminary variable assessment. J. Qual. Technol. 13(3), 174–183 (1981)

    Article  Google Scholar 

  6. Iooss, B., Lemaître, P.: A review on global sensitivity analysis methods. In: Dellino, G., Meloni, C. (eds.) Uncertainty Management in Simulation-Optimization of Complex Systems: Algorithms and Applications, pp. 101–122. Springer, Boston (2015). https://doi.org/10.1007/978-1-4899-7547-8_5

    Chapter  Google Scholar 

  7. Liu, H., Ong, Y.-S., Cai, J.: A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct. Multi. Optim. 57(1), 393–416 (2017). https://doi.org/10.1007/s00158-017-1739-8

    Article  Google Scholar 

  8. McKay, M.D., Beckman, R.J., Conover, W.J.: Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)

    MathSciNet  MATH  Google Scholar 

  9. Morris, M.D.: Factorial sampling plans for preliminary computational experiments. Technometrics 33(2), 161–174 (1991)

    Article  Google Scholar 

  10. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  11. Rausch, M.: Determining cost-effective intrusion detection approaches for an advanced metering infrastructure deployment using ADVISE. Master’s thesis, University of Illinois at Urbana-Champaign (2016)

    Google Scholar 

  12. Rausch, M., Fawaz, A., Keefe, K., Sanders, W.H.: Modeling humans: a general agent model for the evaluation of security. In: McIver, A., Horvath, A. (eds.) Quantitative Evaluation of Systems. Proceedings of International Conference on Quantitative Evaluation of Systems, pp. 373–388. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99154-2_23

  13. Rausch, M., Feddersen, B., Keefe, K., Sanders, W.H.: A comparison of different intrusion detection approaches in an advanced metering infrastructure network using ADVISE. In: Agha, G., Van Houdt, B. (eds.) QEST 2016. LNCS, vol. 9826, pp. 279–294. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-43425-4_19

    Chapter  Google Scholar 

  14. Razmjoo, A., Xanthopoulos, P., Zheng, Q.P.: Online feature importance ranking based on sensitivity analysis. Expert Syst. Appl. 85, 397–406 (2017)

    Article  Google Scholar 

  15. Risdal, M.: Stacking made easy: an introduction to StackNet by competitions grandmaster Marios Michailidis (KazAnova). http://blog.kaggle.com/2017/06/15/stacking-made-easy-an-introduction-to-stacknet-by-competitions-grandmaster-marios-michailidis-kazanova/. Accessed 13 Dec 2019

  16. Ruitenbeek, E.V., Sanders, W.H.: Modeling peer-to-peer botnets. In: Proceedings of 2008 Fifth International Conference on Quantitative Evaluation of Systems, pp. 307–316, September 2008

    Google Scholar 

  17. Sobol, I.: On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7(4), 86–112 (1967)

    Article  MathSciNet  Google Scholar 

  18. Sobol, I.: Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55(1), 271–280 (2001)

    Article  MathSciNet  Google Scholar 

  19. Tenne, Y.: An optimization algorithm employing multiple metamodels and optimizers. Int. J. Autom. Comput. 10(3), 227–241 (2013)

    Article  Google Scholar 

  20. Viana, F., Gogu, C., Haftka, R.: Making the most out of surrogate models: tricks of the trade. In: Proceedings of the ASME Design Engineering Technical Conference, vol. 1, pp. 587–598 (2010)

    Google Scholar 

  21. Wolpert, D.H.: Stacked generalization. Neural Netw. 5(2), 241–259 (1992)

    Article  Google Scholar 

  22. Xiao, N.C., Zuo, M.J., Zhou, C.: A new adaptive sequential sampling method to construct surrogate models for efficient reliability analysis. Reliab. Eng. Syst. Saf. 169, 330–338 (2018)

    Article  Google Scholar 

  23. Zhou, X.J., Ma, Y.Z., Li, X.F.: Ensemble of surrogates with recursive arithmetic average. Struct. Mult. Optim. 44(5), 651–671 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Jenny Applequist, Lowell Rausch, and the reviewers for their feedback on the paper. This material is based upon work supported by the Maryland Procurement Office under Contract No. H98230-18-D-0007. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Maryland Procurement Office.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Michael Rausch .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rausch, M., Sanders, W.H. (2020). Sensitivity Analysis and Uncertainty Quantification of State-Based Discrete-Event Simulation Models Through a Stacked Ensemble of Metamodels. In: Gribaudo, M., Jansen, D.N., Remke, A. (eds) Quantitative Evaluation of Systems. QEST 2020. Lecture Notes in Computer Science(), vol 12289. Springer, Cham. https://doi.org/10.1007/978-3-030-59854-9_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59854-9_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59853-2

  • Online ISBN: 978-3-030-59854-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics