Experiments and Applications

Abstract

In this chapter we provide some composite evidence of the effectiveness of the racing approach, and in particular of the F-Race algorithm, for tuning metaheuristics and more in general for tuning stochastic algorithms.

In particular, Section 5.1 proposes a formal empirical analysis of F-Race on the problem of tuning metaheuristics. In this analysis, the algorithm is compared against other racing algorithms and against the brute-force approach. The latter, being the most straightforward approach for tackling the tuning problem defined in Chapter 3, can be considered as the natural choice of a baseline algorithm for assessing the performance of tuning methods. In order to assess F-Race, two tuning problems are considered. In the first, proposed in Section 5.1.1, the metaheuristic to be tuned is iterated local search and the optimization problem considered is the quadratic assignment problem. In the second, proposed in Section 5.1.2, the algorithm is ant colony optimization and the problem is the traveling salesman problem. On these problems, a formal empirical analysis is carried out: a number of algorithms, including indeed F-Race, are compared under controlled conditions, the differences in their performance are assessed through appropriate statistical tests of significance, and a number of graphs are proposed that help visualize and understand the characteristics of the algorithms under study. The experimental methodology followed in the analysis is heavily influenced by the one commonly adopted within the machine learning community. A discussion on the implications and on the opportunity of adopting this methodology is given in Chapter 6. A particularly valuable tool that has been employed is a so-called re-sampling method. This method, originally presented in the literature on nonparametric statistics, is rather well known by machine learning practitioners but, to the best of our knowledge, it has never been adopted before for what concerns the empirical analysis of metaheuristics: Its first application in the field is described in Birattari et al. (2002).