Journal of Heuristics

, Volume 25, Issue 4–5, pp 591–628 | Cite as

A combined approach for analysing heuristic algorithms

  • Jeroen CorstjensEmail author
  • Nguyen Dang
  • Benoît Depaire
  • An Caris
  • Patrick De Causmaecker


When developing optimisation algorithms, the focus often lies on obtaining an algorithm that is able to outperform other existing algorithms for some performance measure. It is not common practice to question the reasons for possible performance differences observed. These types of questions relate to evaluating the impact of the various heuristic parameters and often remain unanswered. In this paper, the focus is on gaining insight in the behaviour of a heuristic algorithm by investigating how the various elements operating within the algorithm correlate with performance, obtaining indications of which combinations work well and which do not, and how all these effects are influenced by the specific problem instance the algorithm is solving. We consider two approaches for analysing algorithm parameters and components—functional analysis of variance and multilevel regression analysis—and study the benefits of using both approaches jointly. We present the results of a combined methodology that is able to provide more insights than when the two approaches are used separately. The illustrative case studies in this paper analyse a large neighbourhood search algorithm applied to the vehicle routing problem with time windows and an iterated local search algorithm for the unrelated parallel machine scheduling problem with sequence-dependent setup times.


Functional analysis of variance fANOVA Multilevel regression Algorithm performance Vehicle routing problem with time windows Large neighbourhood search Iterated local search Unrelated parallel machine scheduling problem 



This work is funded by COMEX (Project P7/36), a BELSPO/IAP Programme. The computational resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation—Flanders (FWO) and the Flemish Government department EWI. The authors would like to thank Túlio Toffolo for providing us the data for the second case study.


  1. Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: International Conference on Principles and Practice of Constraint Programming, pp. 142–157. Springer (2009)Google Scholar
  2. Bartz-Beielstein, T., Parsopoulos, K.E., Vrahatis, M.N.: Design and analysis of optimization algorithms using computational statistics. Appl. Numer. Anal. Comput. Math. 1(2), 413–433 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  3. Bertsimas, D.J., Simchi-Levi, D.: A new generation of vehicle routing research: robust algorithms, addressing uncertainty. Oper. Res. 44(2), 286 (1996)CrossRefzbMATHGoogle Scholar
  4. Birattari, M.: Tuning Metaheuristics, Studies in Computational Intelligence, vol. 197. Springer, Berlin (2009)CrossRefzbMATHGoogle Scholar
  5. Burke, E.K., Bykov, Y.: A late acceptance strategy in hill-climbing for exam timetabling problems. In: PATAT 2008 Conference, Montreal, Canada (2008)Google Scholar
  6. Bykov, Y., Petrovic, S.: An initial study of a novel step counting hill climbing heuristic applied to timetabling problems. In: Proceedings of 6th Multidisciplinary International Scheduling Conference (MISTA 2013) (2013)Google Scholar
  7. Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Exp. Methods Anal. Optim. Algorithms 1, 225 (2010)CrossRefzbMATHGoogle Scholar
  8. Corstjens, J., Caris, A., Depaire, B.: Explaining heuristic performance differences for vehicle routing problems with time windows. In: Kotsireas, S., Pardalos, P.M. (eds.) Learning and Intelligent Optimization. Lecture Notes in Computer Science. Springer, Berlin (2018). (in press)Google Scholar
  9. Corstjens, J., Depaire, B., Caris, A., Sörensen, K.: A multilevel evaluation method for heuristics with an application to the VRPTW. Manuscript submitted for publication (2017)Google Scholar
  10. Coy, S.P., Golden, B.L., Runger, G.C., Wasil, E.A.: Using experimental design to find effective parameter settings for heuristics. J. Heuristics 7(1), 77–97 (2001)CrossRefzbMATHGoogle Scholar
  11. De Leeuw, J., Meijer, E., Goldstein, H.: Handbook of Multilevel Analysis. Springer, Berlin (2008)CrossRefzbMATHGoogle Scholar
  12. Dietterich, T.: Overfitting and undercomputing in machine learning. ACM Comput. Surv. CSUR 27(3), 326–327 (1995)CrossRefGoogle Scholar
  13. Fawcett, C., Hoos, H.H.: Analysing differences between algorithm configurations through ablation. J. Heuristics 22, 1–28 (2015)Google Scholar
  14. Gelman, A., Hill, J.: Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, Cambridge (2006)CrossRefGoogle Scholar
  15. Hair, J.F., Anderson, R.E., Babin, B.J., Black, W.C.: Multivariate Data Analysis: A Global Perspective, vol. 7. Pearson, Upper Saddle River, NJ (2010)Google Scholar
  16. Hooker, G.: Generalized functional anova diagnostics for high-dimensional functions of dependent variables. J. Comput. Graph. Stat. 16, 709–732 (2012)MathSciNetCrossRefGoogle Scholar
  17. Hooker, J.N.: Testing heuristics: we have it all wrong. J. Heuristics 1(1), 33–42 (1995)CrossRefzbMATHGoogle Scholar
  18. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) Learning and Intelligent Optimization, Lecture Notes in Computer Science, vol. 6683, pp. 507–523. Springer, Berlin (2011)CrossRefGoogle Scholar
  19. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: International Conference on Learning and Intelligent Optimization, pp. 364–381. Springer (2013)Google Scholar
  20. Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: International Conference on Machine Learning, pp. 754–762 (2014)Google Scholar
  21. Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: Paramils: an automatic algorithm configuration framework. J. Artif. Intell. Res. 36(1), 267–306 (2009)CrossRefzbMATHGoogle Scholar
  22. Jones, Z., Linder, F.: Exploratory data analysis using random forests. In: Prepared for the 73rd Annual MPSA Conference (2015)Google Scholar
  23. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. science 220(4598), 671–680 (1983)Google Scholar
  24. Lawson, J., Erjavec, J.: Basic Experimental Strategies and Data Analysis for Science and Engineering. CRC Press, Boca Raton (2016)CrossRefzbMATHGoogle Scholar
  25. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Empirical hardness models: methodology and a case study on combinatorial auctions. J. ACM 56(4), 22 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  26. López-Ibáñez, M., Dubois-Lacoste, J., Cáceres, L.P., Birattari, M., Stützle, T.: The IRACE package: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)MathSciNetCrossRefGoogle Scholar
  27. Lourenço, H.R., Martin, O.C., Stützle, T.: Iterated local search. In: Glover, F.W., Kochenberger, G.A. (eds.) Handbook of Metaheuristics, pp. 320–353. Springer, Berlin (2003)CrossRefGoogle Scholar
  28. Montgomery, D.: Design and Analysis of Experiments, 8th edn. Wiley, New York (2012)Google Scholar
  29. Moore, D.S., McCabe, G.P., Craig, B.A.: Introduction to the Practice of Statistics, 6th edn. W. H. Freeman, New York (2007)Google Scholar
  30. Nannen, V., Eiben, A.E.: Relevance estimation and value calibration of evolutionary algorithm parameters. In: International Joint Conference on Artificial Intelligence, vol. 7, pp. 975–980 (2007)Google Scholar
  31. PassMark Software: CPU benchmarks. (2018). Accessed 26 Mar 2018
  32. Pellegrini, P., Birattari, M.: The relevance of tuning the parameters of metaheuristics. A case study: the vehicle routing problem with stochastic demand. Technical report TR/IRIDIA/2006-008, IRIDIA, Universit Libre de Bruxelles, Brussels, Belgium (2006)Google Scholar
  33. Pisinger, D., Ropke, S.: A general heuristic for vehicle routing problems. Comput. Oper. Res. 34(8), 2403–2435 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  34. Rardin, R.L., Uzsoy, R.: Experimental evaluation of heuristic optimization algorithms: a tutorial. J. Heuristics 7(3), 261–304 (2001)CrossRefzbMATHGoogle Scholar
  35. Rasku, J., Musliu, N., Kärkkäinen, T.: Automating the parameter selection in VRP: an off-line parameter tuning tool comparison. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology, pp. 191–209. Springer, Berlin (2014)Google Scholar
  36. Santos, H.G., Toffolo, T.A., Silva, C.L., Vanden Berghe, G.: Analysis of stochastic local search methods for the unrelated parallel machine scheduling problem. Int. Trans. Oper. Res. (2016).
  37. Simmons, J.P., Nelson, L.D., Simonsohn, U.: False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22(11), 1359–1366 (2011)CrossRefGoogle Scholar
  38. Smith-Miles, K., Bowly, S.: Generating new test instances by evolving in instance space. Comput. Oper. Res. 63, 102–113 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  39. Solomon, M.M.: Algorithms for the vehicle routing and scheduling problems with time window constraints. Oper. Res. 35(2), 254–265 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  40. Stock, J., Watson, M.W.: Introduction to Econometrics. Prentice Hall, New York (2011)Google Scholar
  41. Sullivan, G.M., Feinn, R.: Using effect size—or why the p value is not enough. J. Grad. Med. Educ. 4(3), 279–282 (2012)CrossRefGoogle Scholar
  42. Vallada, E., Ruiz, R.: A genetic algorithm for the unrelated parallel machine scheduling problem with sequence dependent setup times. Eur. J. Oper. Res. 211(3), 612–622 (2011)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.UHasselt, Research Group LogisticsDiepenbeekBelgium
  2. 2.KU Leuven, CODeS, imec-ITECKortrijkBelgium

Personalised recommendations