Skip to main content
SpringerLink
Log in

Ranking and Selection: Efficient Simulation Budget Allocation

  • Chapter
  • First Online:
Handbook of Simulation Optimization

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 216))

Abstract

This chapter reviews the problem of selecting the best of a finite set of alternatives, where best is defined with respect to the highest mean performance, and where the performance is uncertain but may be estimated with simulation. This problem has been explored from several perspectives, including statistical ranking and selection, multiple comparisons, and stochastic optimization. Approaches taken in the literature include frequentist statistics, Bayesian statistics, related heuristics, and asymptotic convergence in probability. This chapter presents algorithms that are derived from Bayesian and related conceptual frameworks to provide empirically effective performance for the ranking and selection problem. In particular, we motivate the optimal computing budget allocation (OCBA) algorithm and expected value of information (EVI) approaches, give example algorithms, and provide pointers to the literature for detailed derivations and extensions of these approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. R. E. Bechhofer, T. J. Santner, and D. M. Goldsman. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley & Sons, New York, 1995.

    Google Scholar 

  2. J. O. Berger and J. Deely. A Bayesian approach to ranking and selection of related means with alternatives to analysis-of-variance methodology. Journal of the American Statistical Association, 83(402):364–373, 1988.

    Article  Google Scholar 

  3. J. Blanchet, J. Liu, and B. Zwart. Large deviations perspective on ordinal optimization of heavy-tailed systems. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 489–494, 2008.

    Google Scholar 

  4. J. Branke, S. E. Chick, and C. Schmidt. Selecting a selection procedure. Management Science, 53:1916–1932, 2007.

    Article  Google Scholar 

  5. M. W. Brantley, L. H. Lee, C. H. Chen, and A. Chen. Efficient simulation budget allocation with regression. IIE Transactions, 45(3):291–308, 2013.

    Article  Google Scholar 

  6. M. W. Brantley, L. H. Lee, C.-H. Chen, and J. Xu. An efficient simulation budget allocation method incorporating regression for partitioned domains. Automatica, 50(5):1391–1400, 2014.

    Google Scholar 

  7. A. Charnes, W. W. Cooper, and E. Rhodes. Measuring the efficiency of decision making units. European Journal of Operational Research, 2:429–444, 1978.

    Article  Google Scholar 

  8. C. H. Chen and L. H. Lee. Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Publishing Co, 2011.

    Google Scholar 

  9. C. H. Chen, D. He, and M. C. Fu. Efficient dynamic simulation allocation in ordinal optimization. IEEE Transactions on Automatic Control, 51(12):2005–2009, 2006.

    Article  Google Scholar 

  10. C. H. Chen, D. He, M. C. Fu, and L. H. Lee. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal of Computing, 20:579–595, 2008.

    Article  Google Scholar 

  11. C. H. Chen, V. Kumar, and Y.C. Luo. Motion planning of walking robots in environments with uncertainty. Journal of Robotic Systems, 16(10):527–545, 1999.

    Article  Google Scholar 

  12. C. H. Chen, J. Lin, E. Yücesan, and S. E. Chick. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems: Theory and Applications, 10:251–270, 2000.

    Article  Google Scholar 

  13. C. H. Chen, E. Yücesan, L. Dai, and H. C. Chen. Efficient computation of optimal budget allocation for discrete event simulation experiment. IIE Transactions, 42(1):60–70, 2010.

    Article  Google Scholar 

  14. S. E. Chick, and P. Frazier. Sequential sampling with economics of selection procedures. Management Science, 58(3):550–569, 2012.

    Article  Google Scholar 

  15. S. E. Chick, and N. Gans. Economic analysis of simulation selection problems. Management Science, 55(3):421–437, 2009.

    Article  Google Scholar 

  16. S. E. Chick and K. Inoue. New two-stage and sequential procedures for selecting the best simulated system. Operations Research, 49(5):732–743, 2001.

    Article  Google Scholar 

  17. S. E. Chick and K. Inoue. New procedures to select the best simulated system using common random numbers. Management Science, 47(8):1133–1149, 2001.

    Article  Google Scholar 

  18. S. E. Chick, J. Branke, and C. Schmidt. Sequential sampling to myopically maximize the expected value of information. INFORMS Journal of Computing, 22:71–80, 2010.

    Article  Google Scholar 

  19. S. E. Chick and Y. Z. Wu. Selection procedures with frequentist expected opportunity cost bounds. Operations Research, 53(5):867–878, 2005.

    Article  Google Scholar 

  20. M. H. de Groot. Optimal Statistical Decisions. McGraw-Hill, New York, 1970.

    Google Scholar 

  21. E. J. Dudewicz and S.R. Dalal. Allocation of observations in ranking and selection with unequal variances. The Indian Journal of Statistics, 37B, 1:28–78, 1975.

    Google Scholar 

  22. P. Frazier, W. B. Powell, and S. Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5):2410–2439, 2008.

    Article  Google Scholar 

  23. P. Frazier, W. B. Powell, and S. Dayanik. The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal of Computing, 21:599–613, 2009.

    Article  Google Scholar 

  24. P. Frazier, J. Xie, and S. E. Chick. Bayesian optimization via simulation with correlated sampling and correlated prior beliefs. In S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, M. Fu, editors, Proceedings of the 2011 Winter Simulation Conference, 3979–3911, 2011.

    Google Scholar 

  25. M. C. Fu, C. H. Chen, and L. Shi. Some topics for simulation optimization. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings the 2008 Winter Simulation Conference, 27–38, 2008.

    Google Scholar 

  26. M. C. Fu, F. W. Glover, and J. April. Simulation optimization: A review, new developments, and applications. In M. E. Kuhl, N. M. Steiger, F. B. Armstrong, J. A. Joines, editors, Proceedings the 2005 Winter Simulation Conference, 83–95, 2005.

    Google Scholar 

  27. M. C. Fu, J. Q. Hu, C. H. Chen, and X. Xiong. Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal of Computing, 19:101–111, 2007.

    Article  Google Scholar 

  28. P. Glynn and S. Juneja. A large deviations perspective on ordinal optimization. In R. G. Ingalls, M. D. Rosetti, J. S. Smith, B. A. Peters, editors, Proceedings of the 2004 Winter Simulation Conference, 577–585, 2004.

    Google Scholar 

  29. D. Goldsman and B. L. Nelson. Comparing systems via simulation. In J. Banks, editor, Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice. John Wiley & Sons, New York, 273–306, 1998.

    Google Scholar 

  30. S. S. Gupta and K. J. Miescke. Bayesian look ahead one stage sampling allocations for selecting the largest normal mean. Statistical Papers, 35:169–177, 1994.

    Article  Google Scholar 

  31. S. S. Gupta and K. J. Miescke. Bayesian look ahead one-stage sampling allocations for selecting the best population. Journal of Statistical Planning and Inference, 54(2):229–244, 1996.

    Article  Google Scholar 

  32. J. M. Hammersley and D.C. Hanscomb. Monte Carlo Methods. Methuen, London, 1964.

    Book  Google Scholar 

  33. D. He, S. E. Chick, and C. H. Chen. The opportunity cost and OCBA selection procedures in ordinal optimization. IEEE Transactions on Systems, Man, and Cybernetics—Part C (Applications and Reviews), 37(4):951–961, 2007.

    Google Scholar 

  34. D. He, L. H. Lee, C. H. Chen, and M. C. Fu, and S. Wasserkrug. Simulation optimization using the cross-entropy method with optimal computing budget allocation. ACM Transactions on Modeling and Computer Simulation, 20(1):Article 4, 2010.

    Google Scholar 

  35. L. J. Hong and B. L. Nelson. A brief introduction to optimization via simulation. In M. D. Rosetti, R. R. Hill, B. Johansson, A. Dunkin, R. Ingalls, editors, Proceedings of the 2009 Winter Simulation Conference, 75–85, 2009.

    Google Scholar 

  36. B. W. Hsieh, C. H. Chen, and S.C. Chang. Scheduling semiconductor wafer fabrication by using ordinal optimization-based simulation. IEEE Transactions on Robotics and Automation, 17(5):599–608, 2001.

    Article  Google Scholar 

  37. Q. S. Jia, E. Zhou, and C. H. Chen. Efficient computing budget allocation for finding simplest good designs. IIE Transactions, 45(7):736–750, 2013.

    Article  Google Scholar 

  38. S.-H. Kim and B. L. Nelson. A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation, 11:251–273, 2001.

    Article  Google Scholar 

  39. S.-H. Kim. and B. L. Nelson. Selecting the best system. In S. G. Henderson, B. L. Nelson, editors, Handbooks in Operations Research and Management Science: Simulation. Elsevier Science, Oxford, 501–534, 2006.

    Google Scholar 

  40. L. W. Koenig and A. M. Law. A procedure for selecting a subset of size m containing the one best of k independent normal populations, with applications to simulation. Communications in Statistics B14, 3:719–734, 1985.

    Google Scholar 

  41. A. M. Law. Simulation Modeling and Analysis, 5th Edition. McGraw-Hill, New York, 2013.

    Google Scholar 

  42. L. H. Lee, E. P. Chew, and P. Manikam. A general framework on the simulation-based optimization under fixed computing budget. European Journal of Operational Research, 174:1828–1841, 2006.

    Article  Google Scholar 

  43. L. H. Lee, E. P. Chew, S. Teng, and D. Goldsman. Optimal computing budget allocation for multi-objective simulation models. In R. G. Ingalls, M. D. Rosetti, J. S. Smith, B. A. Peters, editors, Proceedings of the 2004 Winter Simulation Conference, 586–594, 2004.

    Google Scholar 

  44. L. H. Lee, E. P. Chew, S. Teng, and D. Goldsman. Finding the non-dominated Pareto set for multi-objective simulation models. IIE Transactions, 42:656–674, 2010.

    Article  Google Scholar 

  45. L. H. Lee, N. A. Pujowidianto, L. W. Li, C. H. Chen, and C. M. Yap. Approximate simulation budget allocation for selecting the best system in the presence of stochastic constraints. IEEE Transactions on Automatic Control, 57(11):2940–2945, 2012.

    Article  Google Scholar 

  46. D. J. Morrice, M. W. Brantley, and C. H. Chen. An efficient ranking and selection procedure for a linear transient mean performance measure. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 290–296, 2008.

    Google Scholar 

  47. D. J. Morrice, M. W. Brantley, and C. H. Chen. A transient means ranking and selection procedure with sequential sampling constraints. In M. D. Rosetti, R. R. Hill, B. Johansson, A. Dunkin, R. Ingalls, editors, Proceedings of the 2009 Winter Simulation Conference, 590–600, 2009.

    Google Scholar 

  48. B. L. Nelson, J. Swann, D. Goldsman, and W. M. Song. Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research, 49(6):950–963, 2001.

    Article  Google Scholar 

  49. W. B. Powell and I. O. Ryzhov. Optimal Learning. Wiley, 2012.

    Google Scholar 

  50. Y. Rinott. On two-stage selection procedures and related probability inequalities. Communications in Statistics A7:799–811, 1978.

    Article  Google Scholar 

  51. R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2): 127–190, 1999.

    Article  Google Scholar 

  52. I. O. Ryzhov, W. B. Powell, and P. Frazier. The knowledge-gradient algorithm for a general class of online learning problems. Operations Research, 60(1):180–195, 2012.

    Article  Google Scholar 

  53. L. Shi and C. H. Chen. A new algorithm for stochastic discrete resource allocation optimization. Discrete Event Dynamic Systems: Theory and Applications, 10:271–294, 2000.

    Article  Google Scholar 

  54. J. F. Shortle, C. H. Chen, B. Crain, A. Brodsky, and D. Brod. Optimal splitting for rare-event simulation. IIE Transactions, 44(5):352–367, 2012.

    Article  Google Scholar 

  55. R. Szechtman and E. Yücesan. A new perspective on feasibility determination. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 273–280, 2008.

    Google Scholar 

  56. E. Tekin and I. Sabuncuoglu. Simulation optimization: A comprehensive review on theory and applications. IIE Transactions, 36:1067–1081, 2004.

    Article  Google Scholar 

  57. L. Trailovic and L. Y. Pao. Computing budget allocation for efficient ranking and selection of variances with application to target tracking algorithms. IEEE Transactions on Automatic Control, 49:58–67, 2004.

    Article  Google Scholar 

  58. R. Waeber, P. Frazier, and S. G. Henderson. Performance measures for ranking and selection procedures. In B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, E. Yücesan, editors, Proceedings of the 2010 Winter Simulation Conference, 1235–1245, 2010.

    Google Scholar 

  59. W. P. Wong, W. Jaruphongsa, and L. H. Lee. Budget allocation for effective data collection in predicting an accurate DEA efficiency score. IEEE Transactions on Automatic Control, 56(6):1235–1245, 2011.

    Article  Google Scholar 

  60. S. Yan, E. Zhou, and C. H. Chen. Efficient selection of a set of good enough designs with complexity preference. IEEE Transactions on Automation Science and Engineering, 9(3):596–606, 2012.

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported in part by National Science Foundation under Award CMMI-1233376, Department of Energy under Award DE-SC0002223, NIH under Grant 1R21DK088368-01, National Science Council of Taiwan under Award NSC-100-2218-E-002-027-MY3, and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.

Author information

Authors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Chun-Hung Chen

  2. INSEAD, Fontainebleau, France

    Stephen E. Chick

  3. National University of Singapore, Singapore, Singapore

    Loo Hay Lee

  4. Hewlett-Packard Singapore, Singapore, Singapore

    Nugroho A. Pujowidianto

Authors
  1. Chun-Hung Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen E. Chick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Loo Hay Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nugroho A. Pujowidianto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chun-Hung Chen .

Editor information

Editors and Affiliations

  1. University of Maryland, College Park, Maryland, USA

    Michael C Fu

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer Science+Business Media New York

About this chapter

Cite this chapter

Chen, CH., Chick, S.E., Lee, L.H., Pujowidianto, N.A. (2015). Ranking and Selection: Efficient Simulation Budget Allocation. In: Fu, M. (eds) Handbook of Simulation Optimization. International Series in Operations Research & Management Science, vol 216. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-1384-8_3

Download citation

Publish with us

Policies and ethics

Search

Navigation