A Rigorous Runtime Analysis for Quasi-Random Restarts and Decreasing Stepsize

  • Marc Schoenauer
  • Fabien Teytaud
  • Olivier Teytaud
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7401)


Multi-Modal Optimization (MMO) is ubiquitous in engineering, machine learning and artificial intelligence applications. Many algorithms have been proposed for multimodal optimization, and many of them are based on restart strategies. However, only few works address the issue of initialization in restarts. Furthermore, very few comparisons have been done, between different MMO algorithms, and against simple baseline methods. This paper proposes an analysis of restart strategies, and provides a restart strategy for any local search algorithm for which theoretical guarantees are derived. This restart strategy is to decrease some ’step-size’, rather than to increase the population size, and it uses quasi-random initialization, that leads to a rigorous proof of improvement with respect to random restarts or restarts with constant initial step-size. Furthermore, when this strategy encapsulates a (1+1)-ES with 1/5th adaptation rule, the resulting algorithm outperforms state of the art MMO algorithms while being computationally faster.


Theoretical Guarantee Murder Operator Multimodal Optimization Halton Sequence Random Restart 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Augr, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2005, pp. 1769–1776 (2005)Google Scholar
  2. 2.
    Auger, A., Jebalia, M., Teytaud, O.: Algorithms (X, sigma, eta): Quasi-random Mutations for Evolution Strategies. In: Talbi, E.-G., Liardet, P., Collet, P., Lutton, E., Schoenauer, M. (eds.) EA 2005. LNCS, vol. 3871, pp. 296–307. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Auger, A., Teytaud, O.: Continuous lunches are free plus the design of optimal optimization algorithms. Algorithmica 57(1), 121–146 (2010)Google Scholar
  4. 4.
    Beyer, H.-G., Sendhoff, B.: Covariance Matrix Adaptation Revisited – The CMSA Evolution Strategy –. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN X. LNCS, vol. 5199, pp. 123–132. Springer, Heidelberg (2008)Google Scholar
  5. 5.
    Beyer, H.-G.: The Theory of Evolution Strategies. Springer, Heidelberg (2001)Google Scholar
  6. 6.
    DeJong, K.: The Analysis of the Behavior of a Class of Genetic Adaptive Systems. PhD thesis, University of Michigan, Ann Harbor (1975); Dissertation Abstract International 36(10), 5140B (University Microfilms No 76-9381)Google Scholar
  7. 7.
    Georgieva, A., Jordanov, I.: A hybrid meta-heuristic for global optimisation using low-discrepancy sequences of points. Computers and Operations Research - Special Issue on Hybrid Metaheuristics (in press)Google Scholar
  8. 8.
    Goldberg, D.E., Richardson, J.: Genetic algorithms with sharing for multimodalfunction optimization. In: Grefenstette, J.J. (ed.) ICGA, pp. 41–49. Lawrence Erlbaum Associates (1987)Google Scholar
  9. 9.
    Hansen, N., Kern, S.: Evaluating the CMA Evolution Strategy on Multimodal Test Functions. In: Yao, X., Burke, E.K., Lozano, J.A., Smith, J., Merelo-Guervós, J.J., Bullinaria, J.A., Rowe, J.E., Tiňo, P., Kabán, A., Schwefel, H.-P. (eds.) PPSN VIII. LNCS, vol. 3242, pp. 282–291. Springer, Heidelberg (2004)Google Scholar
  10. 10.
    Harik, G.: Finding multiple solutions in problems of bounded difficulty. Technical Report 94002, University of Illinois at Urbana Champaign (1994)Google Scholar
  11. 11.
    Harik, G.: Finding multimodal solutions using restricted tournament selection. In: Eshelman, L.J. (ed.) Proc. Sixth International Conference on Genetic Algorithms, pp. 24–31. Morgan Kaufmann (1995)Google Scholar
  12. 12.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. of Global Optimization 13(4), 455–492 (1998)Google Scholar
  13. 13.
    Kimura, S., Matsumura, K.: Genetic algorithms using low-discrepancy sequences. In: GECCO, pp. 1341–1346 (2005)Google Scholar
  14. 14.
    L’Ecuyer, P., Lemieux, C.: Recent Advances in Randomized Quasi-Monte Carlo Methods, pp. 419–474. Kluwer Academic (2002)Google Scholar
  15. 15.
    Li, J., Balazs, M., Parks, G., Clarkson, P.: A species conserving genetic algorithm for multimodal function optimization. Evolutionary Computation 10(3), 207–234 (2002)Google Scholar
  16. 16.
    Lindemann, S.R., LaValle, S.M.: Incremental low-discrepancy lattice methods for motion planning. In: Proceedings IEEE International Conference on Robotics and Automation, pp. 2920–2927 (2003)Google Scholar
  17. 17.
    Mahfoud, S.W.: Niching methods for genetic algorithms. PhD thesis, University of Illinois at Urbana-Champaign, Champaign, IL, USA (1995)Google Scholar
  18. 18.
    Mengshoel, O.J., Goldberg, D.E.: Probabilistic crowding: Deterministic crowding with probabilisitic replacement. In: Banzhaf, W., Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, Orlando, Florida, USA, July 13-17, vol. 1, pp. 409–416. Morgan Kaufmann (1999)Google Scholar
  19. 19.
    Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1992)Google Scholar
  20. 20.
    Owen, A.B.: Quasi-Monte Carlo sampling. In: Jensen, H.W. (ed.) Monte Carlo Ray Tracing: Siggraph 2003 Course 44, pp. 69–88. SIGGRAPH (2003)Google Scholar
  21. 21.
    Pétrowski, A.: A clearing procedure as a niching method for genetic algorithms. In: International Conference on Evolutionary Computation, pp. 798–803 (1996)Google Scholar
  22. 22.
    Singh, G., Kalyanmoy Deb, D.: Comparison of multi-modal optimization algorithms based on evolutionary algorithms. In: GECCO 2006: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1305–1312. ACM, New York (2006)Google Scholar
  23. 23.
    Teytaud, F., Teytaud, O.: Log( λ ) Modifications for Optimal Parallelism. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 254–263. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  24. 24.
    Teytaud, O.: When Does Quasi-random Work? In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN X. LNCS, vol. 5199, pp. 325–336. Springer, Heidelberg (2008)Google Scholar
  25. 25.
    Teytaud, O., Gelly, S.: DCMA: yet another derandomization in covariance-matrix-adaptation. In: Thierens, D., et al. (eds.) ACM-GECCO 2007, pp. 955–963. ACM, New York (2007)Google Scholar
  26. 26.
    van den Bergh, F., Engelbrecht, A.: Cooperative learning in neural networks using particle swarm optimizers (2000)Google Scholar
  27. 27.
    Vandewoestyne, B., Cools, R.: Good permutations for deterministic scrambled halton sequences in terms of l2-discrepancy. Computational and Applied Mathematics 189(1,2), 341–361 (2006) bibitemiccama Vandewoestyne, B., Cools, R.: Good permutations for deterministic scrambled halton sequences in terms of l2-discrepancy. Computational and Applied Mathematics 189(1,2), 341–361 (2006)Google Scholar
  28. 28.
    Villemonteix, J., Vazquez, E., Walter, E.: An informational approach to the global optimization of expensive-to-evaluate functions. Journal of Global Optimization 44(4), 509–534 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Yin, X., Germay, N.: A fast genetic algorithm with sharing scheme using cluster analysis methods in multimodal function optimization. In: Albrecht, R.F., Steele, N.C., Reeves, C.R. (eds.) Artificial Neural Nets and Genetic Algorithms, pp. 450–457. Springer (1993)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Marc Schoenauer
    • 1
  • Fabien Teytaud
    • 1
  • Olivier Teytaud
    • 1
  1. 1.TAO (Inria), LRI, UMR 8623CNRS - Univ. Paris-SudOrsayFrance

Personalised recommendations