Optimization of black-box problems using Smolyak grids and polynomial approximations
- 377 Downloads
Abstract
A surrogate-based optimization method is presented, which aims to locate the global optimum of box-constrained problems using input–output data. The method starts with a global search of the n-dimensional space, using a Smolyak (Sparse) grid which is constructed using Chebyshev extrema in the one-dimensional space. The collected samples are used to fit polynomial interpolants, which are used as surrogates towards the search for the global optimum. The proposed algorithm adaptively refines the grid by collecting new points in promising regions, and iteratively refines the search space around the incumbent sample until the search domain reaches a minimum hyper-volume and convergence has been attained. The algorithm is tested on a large set of benchmark problems with up to thirty dimensions and its performance is compared to a recent algorithm for global optimization of grey-box problems using quadratic, kriging and radial basis functions. It is shown that the proposed algorithm has a consistently reliable performance for the vast majority of test problems, and this is attributed to the use of Chebyshev-based Sparse Grids and polynomial interpolants, which have not gained significant attention in surrogate-based optimization thus far.
Keywords
Surrogate-based optimization Black-box optimization Smolyak grids Sparse-grids Chebyshev Polynomial interpolationNotes
Acknowledgements
The authors would like to acknowledge the contribution of the late Professor Christodoulos A Floudas who introduced the authors to this area. CAF was a visionary and pioneer in the areas of global optimization, computational biology, energy systems, grey-box optimization and machine learning. This work was initiated when CAK and FB were postdoctoral associates in the laboratory of CAF, but was completed after his passing when they joined Georgia Institute of Technology.
References
- 1.Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
- 2.Conn, A.R., Le Digabel, S.: Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
- 3.Davis, E., Ierapetritou, M.: A kriging-based approach to MINLP containing black-box models and noise. Ind. Eng. Chem. Res. 47(16), 6101–6125 (2008)CrossRefGoogle Scholar
- 4.Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
- 5.Regis, R.G.: Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points. Eng. Optim. 46(2), 218–243 (2014)MathSciNetCrossRefGoogle Scholar
- 6.Audet, C., Dennis Jr., J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
- 7.Boukouvala, F., Ierapetritou, M.G.: Derivative-free optimization for expensive constrained problems using a novel expected improvement objective function. AIChE J. 60(7), 2462–2474 (2014)CrossRefGoogle Scholar
- 8.Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)zbMATHCrossRefGoogle Scholar
- 9.Echebest, N., Schuverdt, M.L., Vignau, R.P.: A derivative-free method for solving box-constrained underdetermined nonlinear systems of equations. Appl. Math. Comput. 219(6), 3198–3208 (2012)MathSciNetzbMATHGoogle Scholar
- 10.Le Thi, H.A., Vaz, A.I.F., Vicente, L.N.: Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization. Top 20(1), 190–214 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
- 11.Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
- 12.Boukouvala, F., Floudas, C.A.: ARGONAUT: AlgoRithms for global optimization of coNstrAined grey-box compUTational problems. Optim. Lett. 11(5), 895–913 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
- 13.Eason, J.P., Biegler, L.T.: A trust region filter method for glass box/black box optimization. AIChE J. 62(9), 3124–3136 (2016)CrossRefGoogle Scholar
- 14.Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. 4OR 12(4), 301–333 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
- 15.Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. Ann. Oper. Res. 240(1), 351–380 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
- 16.Cozad, A., Sahinidis, N.V., Miller, D.C.: Learning surrogate models for simulation-based optimization. AIChE J. 60(6), 2211–2227 (2014)CrossRefGoogle Scholar
- 17.Jakobsson, S., et al.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
- 18.Quan, N., et al.: Simulation optimization via kriging: a sequential search using expected improvement with computing budget constraints. IIE Trans. 45(7), 763–780 (2013)CrossRefGoogle Scholar
- 19.Das, S., Suganthan, P.N.: Differential evolution: a survey of the state-of-the-art. IEEE Trans. Evol. Comput. 15(1), 4–31 (2011)CrossRefGoogle Scholar
- 20.Egea, J.A., Martí, R., Banga, J.R.: An evolutionary method for complex-process optimization. Comput. Oper. Res. 37(2), 315–324 (2010)zbMATHCrossRefGoogle Scholar
- 21.Ong, Y.S., Nair, P.B., Keane, A.J.: Evolutionary optimization of computationally expensive problems via surrogate modeling. Aiaa J. 41(4), 687–696 (2003)CrossRefGoogle Scholar
- 22.Boukouvala, F., Misener, R., Floudas, C.A.: Global optimization advances in mixed-integer nonlinear programming, MINLP, and constrained derivative-free optimization, CDFO. Eur. J. Oper. Res. 252(3), 701–727 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
- 23.Hooke, R., Jeeves, T.A.: Direct search solution of numerical and statistical problems. J. ACM 8(2), 212–229 (1961)zbMATHCrossRefGoogle Scholar
- 24.Nedler, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)MathSciNetzbMATHCrossRefGoogle Scholar
- 25.Booker, A.J., et al.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17(1), 1–13 (1999)MathSciNetCrossRefGoogle Scholar
- 26.Boukouvala, F., Ierapetritou, M.G.: Surrogate-based optimization of expensive flowsheet modeling for continuous pharmaceutical manufacturing. J. Pharm. Innov. 8(2), 131–145 (2013)CrossRefGoogle Scholar
- 27.Caballero, J.A., Grossmann, I.E.: An algorithm for the use of surrogate models in modular flowsheet optimization. AIChE J. 54(10), 2633–2650 (2008)CrossRefGoogle Scholar
- 28.Ciaurri, D.E., Mukerji, T., Durlofsky, L.J.: Derivative-free optimization for oil field operations. In: Yang, X.-S., Koziel, S. (eds.) Computational Optimization and Applications in Engineering and Industry, pp. 19–55. Springer, Berlin (2011)CrossRefGoogle Scholar
- 29.Egea, J.A., et al.: Scatter search for chemical and bio-process optimization. J. Glob. Optim. 37(3), 481–503 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
- 30.Torn, A., Zilinskas, A.: Global Optimization. Springer, New York (1989)zbMATHCrossRefGoogle Scholar
- 31.Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21(4), 345–383 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
- 32.Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J.-P. (eds.) Advances in Optimization and Numerical Analysis, p. 51–67. Springer, Dordrecht (1994)Google Scholar
- 33.Powell, M.J.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Cambridge NA Report NA2009/06. University of Cambridge, Cambridge (2009)Google Scholar
- 34.Wilson, Z.T., Sahinidis, N.V.: The ALAMO approach to machine learning. Comput. Chem. Eng. 106, 785–795 (2017)CrossRefGoogle Scholar
- 35.Davis, E., Ierapetritou, M.: A kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions. J. Glob. Optim. 43(2–3), 191–205 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
- 36.Palmer, K., Realff, M.: Optimization and validation of steady-state flowsheet simulation metamodels. Chem. Eng. Res. Des. 80(7), 773–782 (2002)CrossRefGoogle Scholar
- 37.Boukouvala, F., Hasan, M.F., Floudas, C.A.: Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption. J. Global Optim. 67(1–2), 3–42 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
- 38.Gutmann, H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19(3), 201–227 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
- 39.Muller, J., Shoemaker, C.A., Piche, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
- 40.Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
- 41.Eason, J., Cremaschi, S.: Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220–232 (2014)CrossRefGoogle Scholar
- 42.Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012)CrossRefGoogle Scholar
- 43.Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. Comput. Aided Chem. Eng. 28, 1129–1134 (2010)CrossRefGoogle Scholar
- 44.Martelli, E., Amaldi, E.: PGS-COM: a hybrid method for constrained non-smooth black-box optimization problems: Brief review, novel algorithm and comparative evaluation. Comput. Chem. Eng. 63, 108–139 (2014)CrossRefGoogle Scholar
- 45.Novak, E., et al.: Smolyak/sparse grid algorithms. Tractability Multivar. Probl. Std. Inf. Funct. 12, 320–397 (2010)Google Scholar
- 46.Plaskota, L., Wasilkowski, G.W.: Smolyak’s algorithm for integration and L-1-approximation of multivariate functions with bounded mixed derivatives of second order. Numer. Algorithms 36(3), 229–246 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
- 47.Bungartz, H.J., Dirnstorfer, S.: Multivariate quadrature on adaptive sparse grids. Computing 71(1), 89–114 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
- 48.Bungartz, H.J., Dirnstorfer, S.: Higher order quadrature on sparse grids. In: Computational Science—Iccs 2004. In: Bubak, M., et al. (Ed.) Proceedings, p. 394–401 (2004)Google Scholar
- 49.Bungartz, H.-J., Pfluger, D., Zimmer, S.: Adaptive sparse grid techniques for data mining. Springer, Berlin (2008)CrossRefGoogle Scholar
- 50.Harding, B.: Adaptive Sparse Grids and Extrapolation Techniques. In: Garcke, J., Pfluger, D. (eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Berlin (2016)Google Scholar
- 51.Jiang, Y., Xu, Y.S.: B-spline quasi-interpolation on sparse grids. J. Complex. 27(5), 466–488 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
- 52.Pfluger, D., Peherstorfer, B., Bungartz, H.J.: Spatially adaptive sparse grids for high-dimensional data-driven problems. J. Complex. 26(5), 508–522 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
- 53.Sickel, W., Ullrich, T.: Spline interpolation on sparse grids. Appl. Anal. 90(3–4), 337–383 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
- 54.Zenger, C.: Sparse grids. In: Hackbusch, W. (ed.) Parallel algorithms for partial differential equations. Proceedings of the Sixth GAMM Seminar. Notes on numerical fluid mechanics. Braunschweig, Verlag Vieweg, vol 31 (1991)Google Scholar
- 55.Novak, E., Ritter, K.: Global optimization using hyperbolic cross points. In: Floudas, C.A., Pardalos, P.M. (eds.) State of the Art in Global Optimization: Computational Methods and Applications. Springer, Boston (1996)Google Scholar
- 56.Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. In: Dokl. Akad. Nauk SSSR (1963)Google Scholar
- 57.Gerstner, T., Griebel, M.: Numerical integration using sparse grids. Numer. Algorithms 18(3–4), 209–232 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
- 58.Dung, D.: Sampling and cubature on sparse grids based on a B-spline quasi-interpolation. Found. Comput. Math. 16(5), 1193–1240 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
- 59.Tang, J.J., et al.: Dimension-adaptive sparse grid interpolation for uncertainty quantification in modern power systems: probabilistic power flow. IEEE Trans. Power Syst. 31(2), 907–919 (2016)CrossRefGoogle Scholar
- 60.Peherstorfer, B., et al.: Selected recent applications of sparse grids. Numer. Math. Theory Methods Appl. 8(1), 47–77 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
- 61.Gajda, P.: Smolyak’s algorithm for weighted L-1-approximation of multivariate functions with bounded rth mixed derivatives over R-d. Numer. Algorithms 40(4), 401–414 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
- 62.Xu, G.Q.: On weak tractability of the Smolyak algorithm for approximation problems. J. Approx. Theory 192, 347–361 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
- 63.Wasilkowski, G.W., Wozniakowski, H.: Explicit cost bounds of algorithms for multivariate tensor product problems. J. Complex. 11(1), 1–56 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
- 64.Valentin, J., Pfluger, D.: Hierarchical gradient-based optimization with b-splines on sparse grids. In: Garcke, J., Pfluger, D. (Eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Cham (2016)Google Scholar
- 65.Grimstad, B., Sandnes, A.: Global optimization with spline constraints: a new branch-and-bound method based on B-splines. J. Glob. Optim. 65(3), 401–439 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
- 66.Hulsmann, M., Reith, D.: SpaGrOW-A derivative-free optimization scheme for intermolecular force field parameters based on sparse grid methods. Entropy 15(9), 3640–3687 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
- 67.Sankaran, S.: Stochastic optimization using a sparse grid collocation scheme. Probab. Eng. Mech. 24(3), 382–396 (2009)CrossRefGoogle Scholar
- 68.Chen, P., Quarteroni, A.: A new algorithm for high-dimensional uncertainty quantification based on dimension-adaptive sparse grid approximation and reduced basis methods. J. Comput. Phys. 298, 176–193 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
- 69.Pronzato, L., Müller, W.G.: Design of computer experiments: space filling and beyond. Stat. Comput. 22(3), 681–701 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
- 70.McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)MathSciNetzbMATHGoogle Scholar
- 71.Davis, E., Ierapetritou, M.: A centroid-based sampling strategy for kriging global modeling and optimization. AIChE J. 56(1), 220–240 (2010)Google Scholar
- 72.Owen, A.B.: Orthogonal arrays for computer experiments, integration and visualization. Stat. Sin. 2(2), 439–452 (1992)MathSciNetzbMATHGoogle Scholar
- 73.Garud, S.S., Karimi, I.A., Kraft, M.: Smart sampling algorithm for surrogate model development. Comput. Chem. Eng. 96, 103–114 (2017)CrossRefGoogle Scholar
- 74.Fornberg, B., Zuev, J.: The Runge phenomenon and spatially variable shape parameters in RBF interpolation. Comput. Math Appl. 54(3), 379–398 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
- 75.Runge, C.: Über empirische Funktionen und die Interpolation zwischen äquidistanten Ordinaten. Zeitschrift für Mathematik und Physik 46(224–243), 20 (1901)zbMATHGoogle Scholar
- 76.Xiang, S., Chen, X., Wang, H.: Error bounds for approximation in Chebyshev points. Numer. Math. 116(3), 463–491 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
- 77.Barthelmann, V., Novak, E., Ritter, K.: High dimensional polynomial interpolation on sparse grids. Adv. Comput. Math. 12(4), 273–288 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
- 78.Davis, P.J., Rabinowitz, P.: Methods of Numerical Integration. Dover Publications (2007)Google Scholar
- 79.Judd, K.L., et al.: Smolyak method for solving dynamic economic models: lagrange interpolation, anisotropic grid and adaptive domain. J. Econ. Dyn. Control 44, 92–123 (2014)MathSciNetCrossRefGoogle Scholar
- 80.Trefethen, N.: Six myths of polynomial interpolation and quadrature. Math. Today 47, 184–188 (2011)MathSciNetGoogle Scholar