Implementation Effort and Performance

A Comparison of Custom and Out-of-the-Box Metaheuristics on the Vehicle Routing Problem with Stochastic Demand
  • Paola Pellegrini
  • Mauro Birattari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4638)


In practical applications, one can take advantage of metaheuristics in different ways: To simplify, we can say that metaheuristics can be either used out-of-the-box or a custom version can be developed. The former way requires a rather low effort, and in general allows to obtain fairly good results. The latter implies a larger investment in the design, implementation, and fine-tuning, and can often produce state-of-the-art results.

Unfortunately, most of the research works proposing an empirical analysis of metaheuristics do not even try to quantify the development effort devoted to the algorithms under consideration. In other words, they do not make clear whether they considered out-of-the-box or custom implementations of the metaheuristics under analysis. The lack of this information seriously undermines the generality and utility of these works.

The aim of the paper is to stress that results obtained with out-of-the-box implementations cannot be always generalized to custom ones, and vice versa. As a case study, we focus on the vehicle routing problem with stochastic demand and on five among the most successful metaheuristics—namely, tabu search, simulated annealing, genetic algorithm, iterated local search, and ant colony optimization. We show that the relative performance of these algorithms strongly varies whether one considers out-of-the-box implementations or custom ones, in which the parameters are accurately fine-tuned.


Local Search Simulated Annealing Tabu Search Multivariate Adaptive Regression Spline Local Search Procedure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Glover, F.: Future paths for integer programming and links to artificial intelligence. Computers & Operations Research 13, 533–549 (1986)zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Dorigo, M., Stützle, T.: Ant Colony Optimization. MIT Press, Cambridge (2004)zbMATHGoogle Scholar
  3. 3.
    Bianchi, L., Birattari, M., Chiarandini, M., Manfrin, M., Mastrolilli, M., Paquete, L., Rossi-Doria, O., Schiavinotto, T.: Hybrid metaheuristics for the vehicle routing problem with stochastic demands. Journal of Mathematical Modelling and Algorithms 5(1), 91–110 (2006)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Langdon, W. (ed.) GECCO 2002. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 11–18. Morgan Kaufmann, San Francisco (2002)Google Scholar
  5. 5.
    Birattari, M.: The problem of tuning metaheuristics as seen from a machine learning perspective. PhD thesis, Université Libre de Bruxelles, Brussels, Belgium (2005)Google Scholar
  6. 6.
    Bartz-Beielstein, T.: Experimental analysis of evolution strategies - overview and comprehensive introduction. Technical Report CI-157/03, Interner Bericht des Sonderforschungsbereichs 531 Computational Intelligence, Universität Dortmund, Dortmund, Germany (2003)Google Scholar
  7. 7.
    Tillman, F.: The multiple terminal delivery problem with probabilistic demands. Transportation Science 3, 192–204 (1969)Google Scholar
  8. 8.
    Stewart, W., Golden, B.: Stochastic vehicle routing: a comprehensive approach. European Journal of Operational Research 14, 371–385 (1983)zbMATHCrossRefGoogle Scholar
  9. 9.
    Dror, M., Trudeau, P.: Stochastic vehicle routing with modified saving algorithm. European Journal of Operational Research 23, 228–235 (1986)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Laporte, G., Louveau, F., Mercure, H.: Models and exact solutions for a class of stochastic location-routing problems. Technical Report G-87-14, Ecole des Hautes Etudes Commerciale, University of Montreal, Montreal, Canada (1987)Google Scholar
  11. 11.
    Bertsimas, D.: A vehicle routing problem with stochastic demand. Operations Research 40(3), 574–585 (1992)zbMATHMathSciNetGoogle Scholar
  12. 12.
    Bertsimas, D., Simchi-Levi, D.: A new generation of vehicle routing research: robust algorithms, addressing uncertainty. Operations Research 44(3), 286–304 (1996)zbMATHCrossRefGoogle Scholar
  13. 13.
    Yang, W., Mathur, K., Ballou, R.: Stochastic vehicle routing problem with restocking. Transportation Science 34(1), 99–112 (2000)zbMATHCrossRefGoogle Scholar
  14. 14.
    Secomandi, N.: A rollout policy for the vehicle routing problem with stochastic demands. Operations Research 49, 796–802 (2001)CrossRefzbMATHGoogle Scholar
  15. 15.
    Secomandi, N.: Analysis of a rollout approach to sequencing problems with stochastic routing applications. Journal of Heuristics 9, 321–352 (2003)zbMATHCrossRefGoogle Scholar
  16. 16.
    Teodorović, D., Pavković, G.: A simulated annealing technique approach to the VRP in the case of stochastic demand. Transportation Planning and Technology 16, 261–273 (1992)CrossRefGoogle Scholar
  17. 17.
    Gendreau, M., Laporte, G., Séguin, R.: A tabu search heuristic for the vehicle routing problem with stochastic demands and customers. Working paper, CRT, University of Montreal, Montreal, Canada (1994)Google Scholar
  18. 18.
    Glover, F., Laguna, M.: Tabu Search. Kluwer Academic Publishers, Norwell (1997)zbMATHGoogle Scholar
  19. 19.
    Ingber, L.: Adaptive simulated annealing (ASA): lessons learned. Control and Cybernetics 26(1), 33–54 (1996)Google Scholar
  20. 20.
    Bäck, T., Fogel, D., Michalewicz, Z. (eds.): Handbook of Evolutionary Computation. IOP Publishing Ltd. Bristol, UK (1997)zbMATHGoogle Scholar
  21. 21.
    Laurenço, H., Martin, O., Stützle, T.: Iterated local search. In: Glover, F., Kochenberger, G. (eds.) Handbook of Metaheuristics, pp. 321–353. Kluwer Academic Publishers, Norwell (2002)Google Scholar
  22. 22.
    Zlochin, M., Birattari, M., Meuleau, N., Dorigo, M.: Model-based search for combinatorial optimization: A critical survey. Annals of Operations Research 131(1–4), 373–395 (2004)zbMATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Adenso-Díaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54(1), 99–114 (2006)CrossRefGoogle Scholar
  24. 24.
    Barr, R., Kelly, J., Resende, M., Stewart, W.: Designing and reporting computational experiments with heuristic methods. Journal of Heuristics 1(1), 9–32 (1995)zbMATHCrossRefGoogle Scholar
  25. 25.
    Bartz-Beielstein, T., Markon, S.: Tuning search algorithms for real-world applications: A regression tree based approach. In: Greenwood, G. (ed.) CEC 2004. Proc. 2004 Congress on Evolutionary Computation, Piscataway, NJ, USA, pp. 1111–1118. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  26. 26.
    Coy, S., Golden, B., Runger, G., Wasil, E.: Using experimental design to find effective parameter settings for heuristics. Journal of Heuristics 7(1), 77–97 (2001)zbMATHCrossRefGoogle Scholar
  27. 27.
    Xu, J., Kelly, J.: A network flow-based tabu search heuristic for the vehicle routing problem. Transportation Science 30, 379–393 (1996)zbMATHGoogle Scholar
  28. 28.
    Parson, R., Johnson, M.: A case study in experimental design applied to genetic algorithms with applications to dna sequence assembly. American Journal of Mathematical and Management Sciences 17, 369–396 (1997)Google Scholar
  29. 29.
    Breedam, A.V.: An analysis od the effect of local improvement operators in genetic algorithms and simulated annealing for the vehicle routing problem. Technical Report TR 96/14, Faculty of Applied Economics, University of Antwerp, Antwerp, Belgium (1996)Google Scholar
  30. 30.
    Xu, J., Chiu, S., Glover, F.: Fine-tuning a tabu search algorithm with statistical tests. International Transactions on Operational Research 5(3), 233–244 (1998)CrossRefGoogle Scholar
  31. 31.
    Pellegrini, P., Birattari, M.: Instances generator for the vehicle routing problem with stochastic demand. Technical Report TR/IRIDIA/2005-10, IRIDIA, Université Libre de Bruxelles, Brussels, Belgium (2005)Google Scholar
  32. 32.
    Pellegrini, P., Birattari, M.: The relevance of tuning the parameters of metaheuristics. A case study: The vehicle routing problem with stochastic demand. Technical Report TR/IRIDIA/2006-008, IRIDIA, Université Libre de Bruxelles, Brussels, Belgium (submitted for journal publication, 2006)Google Scholar
  33. 33.
    Aarts, E., Korst, J., van Laarhoven, P.: Simulated annealing. In: Aarts, E., Lenstra, J. (eds.) Local Search in Combinatorial Optimization, pp. 91–120. John Wiley & Sons, Inc. New York, USA (1997)Google Scholar
  34. 34.
    Whitley, D., Starkweather, T., Shaner, D.: The traveling salesman problem and sequence scheduling: quality solutions using genetic edge recombination. In: Davis, L. (ed.) Handbook of Genetic Algorithms, pp. 350–372. Van Nostrand Reinhold, New York, USA (1991)Google Scholar
  35. 35.
    Friedman, J.: Multivariate adaptive regression splines. The Annals of Statistics 19, 1–141 (1991)zbMATHMathSciNetGoogle Scholar
  36. 36.
    Birattari, M., Zlochin, M., Dorigo, M.: Towards a theory of practice in metaheuristics design: A machine learning perspective. Theoretical Informatics and Applications, Accepted for publication (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Paola Pellegrini
    • 1
  • Mauro Birattari
    • 2
  1. 1.Department of Applied Mathematics, Università Ca’ Foscari, VeniceItaly
  2. 2.IRIDIA, CoDE, Université Libre de Bruxelles, BrusselsBelgium

Personalised recommendations