Why CP Portfolio Solvers Are (under)Utilized? Issues and Challenges

  • Roberto AmadiniEmail author
  • Maurizio Gabbrielli
  • Jacopo Mauro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9527)


It is well recognized that a single, arbitrarily efficient solver can be significantly outperformed by a portfolio solver exploiting a combination of possibly slower on-average different solvers. Despite the success of portfolio solvers within the context of solving competitions, they are rarely used in practice. In this paper we give an overview of the main limitations that hinder the practical adoption and development of portfolio solvers within the Constraint Programming (CP) paradigm, discussing also possible ways to overcome them and potential extensions outside the CP field.


Constraint Programming Constraint Logic Program Solver Selection Quantify Boolean Formula Portfolio Approach 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Abío, I., Stuckey, P.J.: Encoding linear constraints into SAT. In: O’Sullivan, B. (ed.) CP 2014. LNCS, vol. 8656, pp. 75–91. Springer, Heidelberg (2014) Google Scholar
  2. 2.
    Amadini, R., Gabbrielli, M., Mauro J.: An enhanced features extractor for a portfolio of constraint solvers. In: SAC, pp. 1357–1359. ACM (2014)Google Scholar
  3. 3.
    Amadini, R., Gabbrielli, M., Mauro, J.: SUNNY: a lazy portfolio approach for constraint solving. TPLP 14(4–5), 509–524 (2014)zbMATHGoogle Scholar
  4. 4.
    Amadini, R., Gabbrielli, M., Mauro J.: A multicore tool for constraint solving. In: IJCAI, pp. 232–238. AAAI Press (2015)Google Scholar
  5. 5.
    Amadini, R., Gabbrielli, M., Mauro, J.: Portfolio approaches for constraint optimization problems. In: AMAI, pp. 1–18 (2015)Google Scholar
  6. 6.
    Amadini, R., Gabbrielli, M., Mauro, J.: SUNNY-CP: a sequential CP portfolio solver. In: SAC, pp. 1861–1867. ACM (2015)Google Scholar
  7. 7.
    Amadini, R., Stuckey, P.J.: Sequential time splitting and bounds communication for a portfolio of optimization solvers. In: O’Sullivan, B. (ed.) CP 2014. LNCS, vol. 8656, pp. 108–124. Springer, Heidelberg (2014) Google Scholar
  8. 8.
    Arbelaez, A., Hamadi, Y., Sebag, M.: Online heuristic selection in constraint programming. In: SoCS (2009)Google Scholar
  9. 9.
    Arbelaez, A., Hamadi, Y., Sebag, M.: Continuous search in constraint programming. In: ICTAI, pp. 219–243. IEEE Computer Society (2010)Google Scholar
  10. 10.
    Audemard, G., Hoessen, B., Jabbour, S., Lagniez, J-M., Piette, C.: PeneLoPe, a parallel clause-freezer solver. In: SAT Challenge, pp. 43–44 (2012)Google Scholar
  11. 11.
    Barahona, P., Hölldobler, S., Nguyen, V.-H.: Representative encodings to translate finite CSPs into SAT. In: Simonis, H. (ed.) CPAIOR 2014. LNCS, vol. 8451, pp. 251–267. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  12. 12.
    Le Berre, D., Simon, L.: Preface to the special volume on the SAT 2005 competitions and evaluations. In: JSAT, 2(1–4), (2006)Google Scholar
  13. 13.
    Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M.T., Malitsky, Y., Fréchette, A., Hoos, H.H., Hutter, F., Leyton-Brown, K., Tierney, K., Vanschoren, J.: Aslib: A benchmark library for algorithm selection. CoRR, abs/1506.02465 (2015)
  14. 14.
    Cenamor, I., de la Rosa, T., Fernández, F.: IBACOP and IBACOP2 Planner (2014).
  15. 15.
    Chu, G., de la Banda, M.G., Stuckey, P.J.: Automatically exploiting subproblem equivalence in constraint programming. In: Lodi, A., Milano, M., Toth, P. (eds.) CPAIOR 2010. LNCS, vol. 6140, pp. 71–86. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  16. 16.
    Cipriano, R., Dovier, A., Mauro, J.: Compiling and executing declarative modeling languages to gecode. In: Garcia de la Banda, M., Pontelli, E. (eds.) ICLP 2008. LNCS, vol. 5366, pp. 744–748. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  17. 17.
  18. 18.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley-Interscience, New York (2000)zbMATHGoogle Scholar
  19. 19.
    Fawcett, C., Vallati, M., Hutter, F., Hoffmann, J., Hoos, H.H., Leyton-Brown, K.: Improved features for runtime prediction of domain-independent planners. In: ICAPS. AAAI (2014)Google Scholar
  20. 20.
    Frisch, A.M., Harvey, W., Jefferson, C., Martínez-Hernández, B., Miguel, I.: Essence: a constraint language for specifying combinatorial problems. Constraints 13(3), 268–306 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    FullContact. How We Used Immutable Servers to Simplify Our Cloud Infrastructure. (2014)
  22. 22.
    Van Gelder, A.: Careful ranking of multiple solvers with timeouts and ties. In: Sakallah, K.A., Simon, L. (eds.) SAT 2011. LNCS, vol. 6695, pp. 317–328. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  23. 23.
    Gent, I.P., Walsh, T.: CSPlib: a benchmark library for constraints. In: Jaffar, J. (ed.) CP 1999. LNCS, vol. 1713, pp. 480–481. Springer, Heidelberg (1999) Google Scholar
  24. 24.
    Geschwender, D., Hutter, F., Kotthoff, L., Malitsky, Y., Hoos, H.H., Leyton-Brown, K.: Algorithm configuration in the cloud: a feasibility study. In: Pardalos, P.M., Resende, M.G.C., Vogiatzis, C., Walteros, J.L. (eds.) Learning and Intelligent Optimization. Lecture Notes in Computer Science, pp. 41–46. Springer, Heidelberg (2014) Google Scholar
  25. 25.
    Gomes, C.P., Selman, B.: Algorithm portfolios. Artif. Intell. 126(1–2), 43–62 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Guo, H., Hsu, W.H.: A machine learning approach to algorithm selection for NP-hard optimization problems: a case study on the MPE problem. Ann. OR 156(1), 61–82 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach Learn. Res. 3, 1157–1182 (2003)zbMATHGoogle Scholar
  28. 28.
    Hamadi, Y., Jabbour, S., Sais, L.: ManySAT: a parallel SAT solver. JSAT 6(4), 245–262 (2009)zbMATHGoogle Scholar
  29. 29.
    Hebrard, E., O’Mahony, E., O’Sullivan, B.: Constraint programming and combinatorial optimisation in numberjack. In: Lodi, A., Milano, M., Toth, P. (eds.) CPAIOR 2010. LNCS, vol. 6140, pp. 181–185. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  30. 30.
    Hoos, H.H., Kaminski, R., Lindauer, M.T., Schaub, T.: aspeed: solver scheduling via answer set programming. In: TPLP (2015)Google Scholar
  31. 31.
    Hoos, H.H., Kaufmann, B., Schaub, T., Schneider, M.: Robust benchmark set selection for boolean constraint solvers. In: Nicosia, G., Pardalos, P. (eds.) LION 7. LNCS, vol. 7997, pp. 138–152. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  32. 32.
    Lindauer, M., Hoos, H., Hutter, F.: From sequential algorithm selection to parallel portfolio selection. In: Jourdan, L., Dhaenens, C., Marmion, M.-E. (eds.) LION 9 2015. LNCS, vol. 8994, pp. 1–16. Springer, Heidelberg (2015) CrossRefGoogle Scholar
  33. 33.
    Hurley, B., Kotthoff, L., Malitsky, Y., O’Sullivan, B.: Proteus: a hierarchical portfolio of solvers and transformations. In: Simonis, H. (ed.) CPAIOR 2014. LNCS, vol. 8451, pp. 301–317. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  34. 34.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  35. 35.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res. (JAIR) 36, 267–306 (2009)zbMATHGoogle Scholar
  36. 36.
    Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: the state of the art. CoRR, abs/1211.0906 (2012)
  37. 37.
    International planning competition:
  38. 38.
    Kadioglu, S., Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm selection and scheduling. In: Lee, J. (ed.) CP 2011. LNCS, vol. 6876, pp. 454–469. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  39. 39.
    Kadioglu, S., Malitsky, Y., Sellmann, M., Tierney, K.: ISAC - instance-specific algorithm configuration. In: Coelho, H., Studer, R., Wooldridge, M. (eds.) ECAI, Frontiers in Artificial Intelligence and Applications, vol. 215 pp. 751–756. IOS Press (2010)Google Scholar
  40. 40.
    Kiziltan, Z., Mauro, J.: Service-oriented volunteer computing for massively parallel constraint solving using portfolios. In: Lodi, A., Milano, M., Toth, P. (eds.) CPAIOR 2010. LNCS, vol. 6140, pp. 246–251. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  41. 41.
    Kotthoff, L.: LLAMA: leveraging learning to automatically manage algorithms. CoRR, abs/1306.1031 (2013)
  42. 42.
    Kotthoff, L.: Algorithm selection for combinatorial search problems: a survey. AI Mag. 35(3), 48–60 (2014)Google Scholar
  43. 43.
    Kotthoff, L.: Reliability of computational experiments on virtualised hardware. J. Exp. Theor. Artif. Intell. 26(1), 33–49 (2014)CrossRefGoogle Scholar
  44. 44.
    Kroer, C., Malitsky, Y.: Feature filtering for instance-specific algorithm configuration. In: ICTAI, pp. 849–855. IEEE (2011)Google Scholar
  45. 45.
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Parallel SAT solver selection and scheduling. In: Milano, M. (ed.) CP 2012. LNCS, vol. 7514, pp. 512–526. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  46. 46.
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm portfolios based on cost-sensitive hierarchical clustering. In: IJCAI. AAAI (2013)Google Scholar
  47. 47.
    Maratea, M., Pulina, L., Ricca, F.: Multi-engine ASP solving with policy adaptation. J. Logic Comput. 1–22 (2013)Google Scholar
  48. 48.
    Maratea, M., Pulina, L., Ricca, F.: A multi-engine approach to answer-set programming. TPLP 14(6), 841–868 (2014)MathSciNetGoogle Scholar
  49. 49.
  50. 50.
    Morris, K.: Immutable server web page (2013).
  51. 51.
  52. 52.
    Nethercote, N., Stuckey, P.J., Becket, R., Brand, S., Duck, G.J., Tack, G.: MiniZinc: towards a standard CP modelling language. In: Bessière, C. (ed.) CP 2007. LNCS, vol. 4741, pp. 529–543. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  53. 53.
    Ohrimenko, O., Stuckey, P.J., Codish, M.: Propagation via lazy clause gener. Constraints 14(3), 357–391 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  54. 54.
    O’Mahony, E., Hebrard, E., Holland, A., Nugent C., O’Sullivan, B.: Using case-based reasoning in an algorithm portfolio for constraint solving. In: AICS 2008 (2009)Google Scholar
  55. 55.
    Pulina, L., Tacchella, A.: A multi-engine solver for quantified boolean formulas. In: Bessière, C. (ed.) CP 2007. LNCS, vol. 4741, pp. 574–589. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  56. 56.
    Pulina, L., Tacchella, A.: A self-adaptive multi-engine solver for quantified boolean formulas. Constraints 14(1), 80–116 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  58. 58.
  59. 59.
    Roussel O., Lecoutre, C.: XML representation of constraint networks: format XCSP 2.1. CoRR, abs/0902.2362 (2009)
  60. 60.
    Sabharwal, A., Samulowitz, H.: Insights into parallelism with intensive knowledge sharing. In: O’Sullivan, B. (ed.) CP 2014. LNCS, vol. 8656, pp. 655–671. Springer, Heidelberg (2014) Google Scholar
  61. 61.
    Samulowitz, H., Reddy, C., Sabharwal, A., Sellmann, M.: Snappy: a simple algorithm portfolio. In: Järvisalo, M., Van Gelder, A. (eds.) SAT 2013. LNCS, vol. 7962, pp. 422–428. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  62. 62.
  63. 63.
  64. 64.
    Smith-Miles, K.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41(1) (2009)Google Scholar
  65. 65.
    Stojadinovic M., Maric, F.: Instance-based selection of CSP solvers using short training. In: Pragmatics of SAT (2014)Google Scholar
  66. 66.
    Stojadinovic, M., Maric, F.: meSAT: multiple encodings of CSP to SAT. Constraints 19(4), 380–403 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  67. 67.
    Stuckey, P.J., Becket, R., Fischer, J.: Philosophy of the miniZinc challenge. Constraints 15(3), 307–316 (2010)CrossRefzbMATHGoogle Scholar
  68. 68.
    Tamura, N., Taga, A., Kitagawa, S., Banbara, M.: Compiling finite linear CSP into SAT. Constraints 14(2), 254–272 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  69. 69.
    Telelis, O., Stamatopoulos, P.: Combinatorial optimization through statistical instance-based learning. In: ICTAI, pp. 203–209 (2001)Google Scholar
  70. 70.
    Valenzano, R.A., Nakhost, H., Müller, M., Schaeffer, J., Sturtevant, N.R.: ArvandHerd: parallel planning with a portfolio. In: ECAI, pp. 786–791 (2012)Google Scholar
  71. 71.
    Xu, L., Hutter, F., Shen, J., Hoos, H., Leyton-Brown, K.: SATzilla2012: improved algorithm selection based on cost-sensitive classification models. In: SAT Challenge (2012)Google Scholar
  72. 72.
    Lin, X., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. JAIR 32, 565–606 (2008)zbMATHGoogle Scholar
  73. 73.
    Yun, X., Epstein, S.L.: Learning algorithm portfolios for parallel execution. In: Hamadi, Y., Schoenauer, M. (eds.) LION 2012. LNCS, vol. 7219, pp. 323–338. Springer, Heidelberg (2012) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Roberto Amadini
    • 1
    Email author
  • Maurizio Gabbrielli
    • 1
  • Jacopo Mauro
    • 2
  1. 1.Department of Computer Science and Engineering/Laboratory Focus INRIAUniversity of BolognaBolognaItaly
  2. 2.Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations