Advertisement

Mathematical Programming Computation

, Volume 10, Issue 4, pp 597–629 | Cite as

RBFOpt: an open-source library for black-box optimization with costly function evaluations

  • Alberto Costa
  • Giacomo Nannicini
Full Length Paper

Abstract

We consider the problem of optimizing an unknown function given as an oracle over a mixed-integer box-constrained set. We assume that the oracle is expensive to evaluate, so that estimating partial derivatives by finite differences is impractical. In the literature, this is typically called a black-box optimization problem with costly evaluation. This paper describes the solution methodology implemented in the open-source library RBFOpt, available on COIN-OR. The algorithm is based on the Radial Basis Function method originally proposed by Gutmann (J Glob Optim 19:201–227, 2001.  https://doi.org/10.1023/A:1011255519438), which builds and iteratively refines a surrogate model of the unknown objective function. The two main methodological contributions of this paper are an approach to exploit a noisy but less expensive oracle to accelerate convergence to the optimum of the exact oracle, and the introduction of an automatic model selection phase during the optimization process. Numerical experiments show that RBFOpt is highly competitive on a test set of continuous and mixed-integer nonlinear unconstrained problems taken from the literature: it outperforms the open-source solvers included in our comparison by a large amount, and performs slightly better than a commercial solver. Our empirical evaluation provides insight on which parameterizations of the algorithm are the most effective in practice. The software reviewed as part of this submission was given the Digital Object Identifier (DOI)  https://doi.org/10.5281/zenodo.597767.

Keywords

Black-box optimization Derivative-free optimization Global optimization Radial basis function Open-source software Mixed-integer nonlinear programming 

Mathematics Subject Classification

90C56 90C30 65K05 97N80 

Notes

Acknowledgements

The authors are grateful for partial support by the SUTD-MIT International Design Center under grant IDG21300102. The research of A. C. was partially conducted at the Future Resilient Systems and the Future Cities Laboratory at the Singapore-ETH Centre (SEC). The SEC was established as a collaboration between ETH Zurich and National Research Foundation (NRF) Singapore (FI 370074011, FI 370074016) under the auspices of the NRF’s Campus for Research Excellence and Technological Enterprise (CREATE) program.

References

  1. 1.
    Achterberg, T.: SCIP: solving constraint integer programs. Math. Program. Comput. 1(1), 1–41 (2009)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Audet, C., Dennis Jr., J.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2004)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Audet, C., Kokkolaras, M., Le Digabel, S., Talgorn, B.: Order-based error for managing ensembles of surrogates in mesh adaptive direct search. J. Glob. Optim. 70(3), 645–675 (2018)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Baudoui, V.: Optimisation robuste multiobjectifs par modèles de substitution. Ph.D. thesis, University of Toulouse Paul Sabatier (2012)Google Scholar
  5. 5.
    Björkman, M., Holmström, K.: Global optimization of costly nonconvex functions using radial basis functions. Optim. Eng. 1(4), 373–397 (2000)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Bonami, P., Biegler, L., Conn, A., Cornuéjols, G., Grossmann, I., Laird, C., Lee, J., Lodi, A., Margot, F., Sawaya, N., Wächter, A.: An algorithmic framework for convex mixed integer nonlinear programs. Discrete Optim. 5, 186–204 (2008)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Bussieck, M.R., Drud, A.S., Meeraus, A.: MINLPLib—a collection of test models for mixed-integer nonlinear programming. INFORMS J. Comput. 15(1), 114–119 (2003)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Byrd, R.H., Nocedal, J., Waltz, R.A.: KNITRO: an integrated package for nonlinear optimization. In: Di Pillo, G., Roma, M. (eds.) Large-Scale Nonlinear Optimization, pp. 35–59. Springer, New York (2006)CrossRefGoogle Scholar
  9. 9.
    Conn, A.R., Scheinberg, K., Toint, P.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79(1–3), 397–414 (1997).  https://doi.org/10.1007/BF02614326 MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics, Philadelphia (2009)CrossRefGoogle Scholar
  11. 11.
    Costa, A., Di Buccio, E., Melucci, M., Nannicini, G.: Efficient parameter estimation for information retrieval using black-box optimization. IEEE Trans. Knowl. Data Eng. 30, 1240–1253 (2017)CrossRefGoogle Scholar
  12. 12.
    Costa, A., Nannicini, G., Schroepfer, T., Wortmann, T.: Black-box optimization of lighting simulation in architectural design. In: Cardin, M.A., Krob, D., Chuen, L., Tan, Y., Wood, K. (eds.) Designing Smart Cities: Proceedings of the First Asia-Pacific Conference on Complex Systems Design & Management, CSD&M Asia 2014, pp. 27–39. Springer (2015)Google Scholar
  13. 13.
    D’Ambrosio, C., Nannicini, G., Sartor, G.: MILP models for the selection of a small set of well-distributed points. Oper. Res. Lett. 45(1), 46–52 (2017)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Diaz, G.I., Fokour, A., Nannicini, G., Samulowitz, H.: An effective algorithm for hyperparameter optimization of neural networks. IBM J. Res. Dev. 61(4/5), 9-1 (2017)CrossRefGoogle Scholar
  15. 15.
    Dixon, L., Szego, G.: The global optimization problem: an introduction. In: Dixon, L., Szego, G. (eds.) Towards Global Optimization, pp. 1–15. North Holland, Amsterdam (1975)Google Scholar
  16. 16.
    Eriksson, D., Bindel, D., Shoemaker, C.: Surrogate optimization toolbox (pySOT) (2015). http://github.com/dme65/pySOT
  17. 17.
    Fuerle, F., Sienz, J.: Formulation of the Audze–Eglais uniform latin hypercube design of experiments for constrained design spaces. Adv. Eng. Softw. 42(9), 680–689 (2011)CrossRefGoogle Scholar
  18. 18.
    Gablonsky, J., Kelley, C.: A locally-biased form of the DIRECT algorithm. J. Glob. Optim. 21(1), 27–37 (2001)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Gendreau, M., Potvin, J.Y. (eds.): Handbook of Metaheuristics, 2nd edn. Kluwer, Dordrecht (2010)zbMATHGoogle Scholar
  20. 20.
    Glover, F., Kochenberger, G. (eds.): Handbook of Metaheuristics. Kluwer, Dordrecht (2003)zbMATHGoogle Scholar
  21. 21.
    Gomes, C.P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Autom. Reason. 24(1–2), 67–100 (2000).  https://doi.org/10.1023/A:1006314320276 MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Gutmann, H.M.: A radial basis function method for global optimization. J. Glob. Optim. 19, 201–227 (2001).  https://doi.org/10.1023/A:1011255519438 MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Hart, W.E., Laird, C., Watson, J.P., Woodruff, D.L.: Pyomo—optimization Modeling in Python. Springer Optimization and Its Applications, vol. 67. Springer, Berlin (2012)CrossRefGoogle Scholar
  24. 24.
    Hart, W.E., Watson, J.P., Woodruff, D.L.: Pyomo: modeling and solving mathematical programs in Python. Math. Program. Comput. 3(3), 219–260 (2011).  https://doi.org/10.1007/s12532-011-0026-8 MathSciNetCrossRefGoogle Scholar
  25. 25.
    Hemker, T.: Derivative free surrogate optimization for mixed-integer nonlinear black-box problems in engineering. Master’s thesis, Technischen Universität Darmstadt (2008)Google Scholar
  26. 26.
    Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization. J. Glob. Optim. 41(3), 447–464 (2008)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Holmström, K., Quttineh, N.H., Edvall, M.M.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer constrained global optimization. Optim. Eng. 9(4), 311–339 (2008)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Huyer, W., Neumaier, A.: SNOBFIT—stable noisy optimization by branch and fit. ACM Trans. Math. Softw. 35(2), 1–25 (2008)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.A.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)Google Scholar
  30. 30.
    Jakobsson, S., Patriksson, M., Rudholm, J., Wojciechowski, A.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Johnson, S.G.: The NLopt nonlinear-optimization package. http://ab-initio.mit.edu/nlopt
  32. 32.
    Jones, D., Perttunen, C., Stuckman, B.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl. 79(1), 157–181 (1993)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Kolda, T.G., Lewis, R.M., Torczon, V.J.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Le Digabel, S.: Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37(4), 44:1–44:15 (2011).  https://doi.org/10.1145/1916461.1916468 MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
  37. 37.
    Moré, J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Müller, J.: MISO: mixed-integer surrogate optimization framework. Optim. Eng. 1–27 (2015).  https://doi.org/10.1007/s11081-015-9281-2 MathSciNetCrossRefGoogle Scholar
  39. 39.
    Müller, J., Paudel, R., Shoemaker, C.A., Woodbury, J., Wang, Y., Mahowald, N.: \(\text{ CH }_{4}\) parameter estimation in CLM4.5bgc using surrogate global optimization. Geosci. Model Dev. 8(10), 3285–3310 (2015).  https://doi.org/10.5194/gmd-8-3285-2015 CrossRefGoogle Scholar
  40. 40.
    Müller, J., Shoemaker, C.A.: Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms forcomputationally expensive black-box global optimization problems. J. Glob. Optim. 60(2), 123–144 (2014).  https://doi.org/10.1007/s10898-014-0184-0 CrossRefzbMATHGoogle Scholar
  41. 41.
    Müller, J., Shoemaker, C.A., Piché, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013).  https://doi.org/10.1016/j.cor.2012.08.022 MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Nelder, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7(4), 308–313 (1965)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Neumaier, A.: Neumaier’s collection of test problems for global optimization. http://www.mat.univie.ac.at/~neum/glopt/my_problems.html. Retrieved in May 2014
  44. 44.
    Powell, M.: Recent research at Cambridge on radial basis functions. In: Müller, M.W., Buhmann, M.D., Mache, D.H., Felten, M. (eds.) New Developments in Approximation Theory. International Series of Numerical Mathematics, vol. 132, pp. 215–232. Birkhauser Verlag, Basel (1999)CrossRefGoogle Scholar
  45. 45.
    Powell, M.J.: The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report, Cambridge NA Report NA2009/06, University of Cambridge (2009)Google Scholar
  46. 46.
    Regis, R., Shoemaker, C.: Improved strategies for radial basis function methods for global optimization. J. Glob. Optim. 37, 113–135 (2007).  https://doi.org/10.1007/s10898-006-9040-1 MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19(4), 497–509 (2007).  https://doi.org/10.1287/ijoc.1060.0182 MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Schoen, F.: A wide class of test functions for global optimization. J. Glob. Optim. 3(2), 133–137 (1993)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Törn, A., Žilinskas, A.: Global Optimization. Springer, Berlin (1987)zbMATHGoogle Scholar
  52. 52.
    Wächter, A., Biegler, L.T.: On the implementation of a primal-dual interior point filter line search algorithm for large-scale nonlinear programming. Math. Program. 106(1), 25–57 (2006)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Wortmann, T., Costa, A., Nannicini, G., Schroepfer, T.: Advantages of surrogate models for architectural design optimization. Artif. Intell. Eng. Des. Anal. Manuf. 29(4), 471–481 (2015)CrossRefGoogle Scholar
  54. 54.
    Wortmann, T., Waibel, C., Nannicini, G., Evins, R., Schroepfer, T., Carmeliet, J.: Are genetic algorithms really the best choice for building energy optimization? In: Proceedings of the Symposium on Simulation for Architecture & Urban Design (SimAUD), pp. 51–58. SCS, Toronto, Canada (2017)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and The Mathematical Programming Society 2018

Authors and Affiliations

  1. 1.Future Cities Laboratory ProgramETH ZurichSingaporeSingapore
  2. 2.IBM T.J. Watson Research CenterYorktown HeightsUSA

Personalised recommendations