Abstract
Modern high-performing algorithms are usually highly parameterised, and can be configured either manually or by an automatic algorithm configurator. The algorithm performance dataset obtained after the configuration step can be used to gain insights into how different algorithm parameters influence algorithm performance. This can be done by a number of analysis methods that exploit the idea of learning prediction models from an algorithm performance dataset and then using them for the data analysis on the importance of variables. In this paper, we demonstrate the complementary usage of three methods along this line, namely forward selection, fANOVA and ablation analysis with surrogates on three case studies, each of which represents some special situations that the analyses can fall into. By these examples, we illustrate how to interpret analysis results and discuss the advantage of combining different analysis methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
This amount of running time is reported on the Python-based fANOVA package linked by PIMP. The previous fANOVA version, which is Java-based, is faster, although it still needs several hours to finish this analysis.
References
Bartz-Beielstein, T., Lasarczyk, C., Preuss, M.: The sequential parameter optimization toolbox. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 337–362. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-02538-9_14
Biedenkapp, A., Lindauer, M., Eggensperger, K., Hutter, F., Fawcett, C., Hoos, H.H.: Efficient parameter importance analysis via ablation with surrogates. In: Singh, S.P., Markovitch, A. (eds.) AAAI Conference on Artificial Intelligence. AAAI Press (2017)
Chiarandini, M., Goegebeur, Y.: Mixed models for the analysis of optimization algorithms. Exp. Methods Anal. Optim. Algorithms 1, 225 (2010)
Corstjens, J., Caris, A., Depaire, B., Sörensen, K.: A multilevel methodology for analysing metaheuristic algorithms for the VRPTW
Dang, N., Pérez Cáceres, L., De Causmaecker, P., Stützle, T.: Configuring irace using surrogate configuration benchmarks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 243–250. ACM (2017)
Fawcett, C., Hoos, H.H.: Analysing differences between algorithm configurations through ablation. J. Heuristics 22(4), 431–458 (2016)
Hooker, G.: Generalized functional ANOVA diagnostics for high-dimensional functions of dependent variables. J. Comput. Graph. Stat 16(3), 709–732 (2012)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Identifying key algorithm parameters and instance features using forward selection. In: Nicosia, G., Pardalos, P. (eds.) LION 2013. LNCS, vol. 7997, pp. 364–381. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-44973-4_40
Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31th International Conference on Machine Learning, vol. 32, pp. 754–762 (2014)
Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res 36, 267–306 (2009). Oct
Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: methods and evaluation. Artif. Intell. 206, 79–111 (2014)
IBM. ILOG CPLEX optimizer (2017). http://www.ibm.com/software/integration/optimization/cplex-optimizer/
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Stützle, T., Birattari, M.: The irace package: Iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)
Stützle, T.: ACOTSP: a software package of various ant colony optimization algorithms applied to the symmetric traveling salesman problem (2002)
Acknowledgement
This work is funded by COMEX (Project P7/36), a BELSPO/IAP Programme. The computational resources and services were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government - department EWI. The authors are grateful to Thomas Stützle and the anonymous reviewers for their valuable comments, which help to improve the quality of the paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Dang, N., De Causmaecker, P. (2019). Analysis of Algorithm Components and Parameters: Some Case Studies. In: Battiti, R., Brunato, M., Kotsireas, I., Pardalos, P. (eds) Learning and Intelligent Optimization. LION 12 2018. Lecture Notes in Computer Science(), vol 11353. Springer, Cham. https://doi.org/10.1007/978-3-030-05348-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-05348-2_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-05347-5
Online ISBN: 978-3-030-05348-2
eBook Packages: Computer ScienceComputer Science (R0)