Designing and Comparing Multiple Portfolios of Parameter Configurations for Online Algorithm Selection

  • Aldy GunawanEmail author
  • Hoong Chuin Lau
  • Mustafa Mısır
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10079)


Algorithm portfolios seek to determine an effective set of algorithms that can be used within an algorithm selection framework to solve problems. A limited number of these portfolio studies focus on generating different versions of a target algorithm using different parameter configurations. In this paper, we employ a Design of Experiments (DOE) approach to determine a promising range of values for each parameter of an algorithm. These ranges are further processed to determine a portfolio of parameter configurations, which would be used within two online Algorithm Selection approaches for solving different instances of a given combinatorial optimization problem effectively. We apply our approach on a Simulated Annealing-Tabu Search (SA-TS) hybrid algorithm for solving the Quadratic Assignment Problem (QAP) as well as an Iterated Local Search (ILS) on the Travelling Salesman Problem (TSP). We also generate a portfolio of parameter configurations using best-of-breed parameter tuning approaches directly for the comparison purpose. Experimental results show that our approach lead to improvements over best-of-breed parameter tuning approaches.


Algorithm Selection Travelling Salesman Problem Travel Salesman Problem Testing Instance Parameter Configuration 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centres in Singapore Funding Initiative.


  1. 1.
    Rice, J.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  2. 2.
    Gomes, C., Selman, B.: Algorithm portfolios. AI 126(1), 43–62 (2001)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Huberman, B., Lukose, R., Hogg, T.: An economics approach to hard computational problems. Science 275(5296), 51 (1997)CrossRefGoogle Scholar
  4. 4.
    Smith-Miles, K.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41(1), 1–25 (2008)CrossRefGoogle Scholar
  5. 5.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. JAIR 36(1), 267–306 (2009)zbMATHGoogle Scholar
  6. 6.
    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. JMLR 13, 281–305 (2012)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Eiben, A., Michalewicz, Z., Schoenauer, M., Smith, J.: Parameter control in evolutionary algorithms. In: Lobo, F.G., Lima, C.F., Michalewicz, Z. (eds.) Parameter Setting in Evolutionary Algorithms. SCI, vol. 54, pp. 19–46. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-69432-8_2 CrossRefGoogle Scholar
  8. 8.
    Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. JAIR 32(1), 565–606 (2008)zbMATHGoogle Scholar
  9. 9.
    Xu, L., Hoos, H.H., Leyton-Brown, K.: Hydra: automatically configuring algorithms for portfolio-based selection. In: AAAI 2010, pp. 210–216 (2010)Google Scholar
  10. 10.
    Adenso-Diaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental designs and local search. OR 54(1), 99–114 (2006)CrossRefzbMATHGoogle Scholar
  11. 11.
    Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: GECCO 2002, pp. 11–18 (2002)Google Scholar
  12. 12.
    Kadioglu, S., Malitsky, Y., Sellmann, M., Tierney, K.: ISAC–instance-specific algorithm configuration. In: ECAI 2010, pp. 751–756 (2010)Google Scholar
  13. 13.
    Lau, H.C., Xiao, F., Halim, S.: A framework for automated parameter tuning in heuristic design. In: MIC 2009, Hamburg, Germany, 13–16 June 2009Google Scholar
  14. 14.
    Seipp, J., Braun, M., Garimort, J., Helmert, M.: Learning portfolios of automatically tuned planners. In: ICAPS 2012, Atibaia/Sao Paulo, Brazil (2012)Google Scholar
  15. 15.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-25566-3_40 CrossRefGoogle Scholar
  16. 16.
    Lindauer, M., Hoos, H.H., Hutter, F., Schaub, T.: AutoFolio: an automatically configured algorithm selector. J. Artif. Intell. Res. 53, 745–778 (2015)MathSciNetGoogle Scholar
  17. 17.
    Mısır, M., Handoko, S.D., Lau, H.C.: ADVISER: a web-based algorithm portfolio deviser. In: Dhaenens, C., Jourdan, L., Marmion, M.-E. (eds.) LION 2015. LNCS, vol. 8994, pp. 23–28. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-19084-6_3 CrossRefGoogle Scholar
  18. 18.
    Kadioglu, S., Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm selection and scheduling. In: Lee, J. (ed.) CP 2011. LNCS, vol. 6876, pp. 454–469. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-23786-7_35 CrossRefGoogle Scholar
  19. 19.
    Hutter, F., Hoos, H.H., Stutzle, T.: Automatic algorithm configuration based on local search. In: AAAI 2007, vol. 22, p. 1152 (2007)Google Scholar
  20. 20.
    Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 142–157. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-04244-7_14 CrossRefGoogle Scholar
  21. 21.
    KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search sat solvers from components. In: IJCAI 2009, pp. 517–524 (2009)Google Scholar
  22. 22.
    Montgomery, D.: Design and Analysis of Expeirments, 6th edn. Wiley, Hoboken (2005)Google Scholar
  23. 23.
    Gunawan, A., Lau, H.C., Lindawati, : Fine-tuning algorithm parameters using the design of experiments approach. In: Coello Coello, A. (ed.) LION 2011. LNCS, vol. 6683, pp. 278–292. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-25566-3 CrossRefGoogle Scholar
  24. 24.
    Mısır, M., Verbeeck, K., De Causmaecker, P., Vanden Berghe, G.: An investigation on the generality level of selection hyper-heuristics under different empirical conditions. Appl. Soft Comput. 13(7), 3335–3353 (2013)CrossRefGoogle Scholar
  25. 25.
    Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)CrossRefzbMATHGoogle Scholar
  26. 26.
    Mısır, M., Handoko, S.D., Lau, H.C.: OSCAR: online selection of algorithm portfolios with case study on memetic algorithms. In: Dhaenens, C., Jourdan, L., Marmion, M.-E. (eds.) LION 2015. LNCS, vol. 8994, pp. 59–73. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-19084-6_6 CrossRefGoogle Scholar
  27. 27.
    Pfahringer, B., Bensusan, H., Giraud-Carrier, C.: Tell me who can learn you and I can tell you who you are: landmarking various learning algorithms. In: Proceedings of the 17th International Conference on Machine Learning, pp. 743–750 (2000)Google Scholar
  28. 28.
    Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)CrossRefGoogle Scholar
  29. 29.
    He, J., He, F., Dong, H.: Pure strategy or mixed strategy? In: Hao, J.-K., Middendorf, M. (eds.) EvoCOP 2012. LNCS, vol. 7245, pp. 218–229. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-29124-1_19 CrossRefGoogle Scholar
  30. 30.
    Lehre, P.K., Özcan, E.: A runtime analysis of simple hyper-heuristics: to mix or not to mix operators. In: FOGA 2013, Adelaide, Australia (2013)Google Scholar
  31. 31.
    Cowling, P., Kendall, G., Soubeiga, E.: A hyperheuristic approach to scheduling a sales summit. In: Burke, E., Erben, W. (eds.) PATAT 2000. LNCS, vol. 2079, pp. 176–190. Springer, Heidelberg (2001). doi: 10.1007/3-540-44629-X_11 CrossRefGoogle Scholar
  32. 32.
    Mısır, M., Wauters, T., Verbeeck, K., Vanden Berghe, G.: A new learning hyper-heuristic for the traveling tournament problem. In: Caserta, M., Voß, S. (eds.) Metaheuristics: Intelligent Problem Solving - MIC 2009. Springer, Heidelberg (2010)Google Scholar
  33. 33.
    Thathachar, M., Sastry, P.: Networks of Learning Automata: Techniques for Online Stochastic Optimization. Kluwer Academic Publishers, Dordrecht (2004)CrossRefGoogle Scholar
  34. 34.
    Ng, K.M., Gunawan, A., Poh, K.L.: A hybrid algorithm for the quadratic assignment problem. In: CSC 2008, Nevada, USA, 14–17 July 2008Google Scholar
  35. 35.
    Burkard, R., Karisch, S., Rendl, F.: Qaplib-a quadratic assignment problem library. J. Global Optim. 10(4), 391–403 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Halim, S., Yap, R.H.C., Lau, H.C.: An integrated white+black box approach for designing and tuning stochastic local search. In: Bessière, C. (ed.) CP 2007. LNCS, vol. 4741, pp. 332–347. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-74970-7_25 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Aldy Gunawan
    • 1
    Email author
  • Hoong Chuin Lau
    • 1
  • Mustafa Mısır
    • 2
    • 3
  1. 1.School of Information SystemsSingapore Management UniversitySingaporeSingapore
  2. 2.Department of Computer ScienceUniversity of FreiburgFreiburg im BreisgauGermany
  3. 3.College of Computer Science and TechnologyNanjing University of Aeronautics and AstronauticsNanjingChina

Personalised recommendations