Journal of Global Optimization

, Volume 50, Issue 4, pp 575–596 | Cite as

Dynamic sample budget allocation in model-based optimization

  • Jiaqiao Hu
  • Hyeong Soo Chang
  • Michael C. Fu
  • Steven I. Marcus


Model-based search methods are a class of optimization techniques that search the solution space by sampling from an underlying probability distribution “model,” which is updated iteratively after evaluating the performance of the samples at each iteration. This paper aims to improve the sampling efficiency of model-based methods by considering a generalization where a population of distribution models is maintained and subsequently propagated from generation to generation. A key issue in the proposed approach is how to efficiently allocate the sampling budget among the population of models to maximize the algorithm performance. We formulate this problem as a generalized max k-armed bandit problem, and derive an efficient dynamic sample allocation scheme based on Markov decision theory to adaptively allocate computational resources. The proposed allocation scheme is then further used to update the current population to produce an improving population of models. Our preliminary numerical results indicate that the proposed procedure may considerably reduce the number of function evaluations needed to obtain high quality solutions, and thus further enhance the value of model-based methods for optimization problems that require expensive function evaluations for performance evaluation.


Markov decision processes Max k-armed bandit Global optimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bertsekas D.P.: Dynamic Programming and Optimal Control, vol 1 and 2. Athena Scientific, Belmont (1995)Google Scholar
  2. 2.
    Chepuri K., Homem De Mello T.: Solving the vehicle routing problem with stochastic demands using the cross-entropy method. Ann. Oper. Res. 134, 153–181 (2005)CrossRefGoogle Scholar
  3. 3.
    Cicirello, V., Smith, S.F.: The max k-armed bandit: a new model for exploration applied to search heuristic selection. In: Proceedings of the 20th National Conference on Artificial Intelligence (AAAI-05) (2005)Google Scholar
  4. 4.
    Dorigo M., Gambardella L.M.: Ant colony system: a cooperative learning approach to the traveling salesman problem. IEEE Trans. Evol. Comput. 1, 53–66 (1997)CrossRefGoogle Scholar
  5. 5.
    Fu, M.C., Hu, J., Marcus, S. I.: Model-based randomized methods for global optimization. In: Proceedings of the 17th international symposium on mathematical theory of networks and systems. Kyoto, Japan (2006)Google Scholar
  6. 6.
    Glover F.: Tabu search: a tutorial. Interfaces 20, 74–94 (1990)CrossRefGoogle Scholar
  7. 7.
    Hu J., Fu M.C., Marcus S.I.: A model reference adaptive search method for global optimization. Oper. Res. 55, 549–568 (2007)CrossRefGoogle Scholar
  8. 8.
    Kirkpatrick S., Gelatt C.D., Vecchi M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983)CrossRefGoogle Scholar
  9. 9.
    Larrañaga P., Lozano J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation. Kluwer, Boston (2002)Google Scholar
  10. 10.
    Mühlenbein H., Paaß G.: From recombination of genes to the estimation of distributions: I. Binary parameters. In: Voigt, H.-M., Ebeling, W., Rechenberg, I., Schwefel, H.-P. (eds) Parallel Problem Solving from Nature—PPSN IV, pp. 178–187. Springer, Berlin (1996)CrossRefGoogle Scholar
  11. 11.
    Pintér J.D.: Global Optimization in Action. Kluwer, The Netherlands (1996)Google Scholar
  12. 12.
    Ross S.: Stochastic Processes 2nd edn. John Wiley, New York (1995)Google Scholar
  13. 13.
    Royden H.L.: Real Analysis 3rd edn. Prentice-Hall, Englewood Cliffs (1988)Google Scholar
  14. 14.
    Rubinstein R.Y.: Optimization of computer simulation models with rare events. Eur. J. Oper. Res. 99, 89–112 (1997)CrossRefGoogle Scholar
  15. 15.
    Rubinstein R.Y., Kroese D.P.: The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation, and Machine Learning. Springer, New York (2004)Google Scholar
  16. 16.
    Shi L., Ólafsson S.: Nested partitions method for global optimization. Oper. Res. 48, 390–407 (2000)CrossRefGoogle Scholar
  17. 17.
    Srinivas M., Patnaik L.M.: Genetic algorithms: a survey. IEEE Comp. 27, 17–26 (1994)Google Scholar
  18. 18.
    Streeter, M., Smith, S.F.: A simple distribution-free approach to the max k-armed bandit problem. In: Proceedings of the 12th international conference on principles and practice of constraint programming. Lecture Notes in Computer Science 4204, pp. 560–574. Springer, Berlin (2006)Google Scholar
  19. 19.
    Tang Z.: Adaptive partitioned random search to global optimization. IEEE Trans. Autom. Control 39, 2235–2244 (1994)CrossRefGoogle Scholar
  20. 20.
    Zlochin M., Birattari M., Meuleau N., Dorigo M.: Model-based search for combinatorial optimization: a critical survey. Ann. Oper. Res. 131, 373–395 (2004)CrossRefGoogle Scholar
  21. 21.
    Zhang, H., Fu, M.C.: Applying model reference adaptive search to American-style option pricing. In: Proceedings of the 28th winter simulation conference, pp. 711–718 (2006)Google Scholar

Copyright information

© Springer Science+Business Media, LLC. 2009

Authors and Affiliations

  • Jiaqiao Hu
    • 1
  • Hyeong Soo Chang
    • 2
  • Michael C. Fu
    • 3
  • Steven I. Marcus
    • 4
  1. 1.Department of Applied Mathematics and StatisticsState University of New YorkStony BrookUSA
  2. 2.Department of Computer Science and Engineering (200811037)Sogang UniversitySeoulKorea
  3. 3.Robert H. Smith School of Business and The Institute for Systems ResearchUniversity of MarylandCollege ParkUSA
  4. 4.Department of Electrical and Computer Engineering and The Institute for Systems ResearchUniversity of MarylandCollege ParkUSA

Personalised recommendations