Advertisement

Journal of Global Optimization

, Volume 71, Issue 4, pp 845–869 | Cite as

Optimization of black-box problems using Smolyak grids and polynomial approximations

  • Chris A. Kieslich
  • Fani Boukouvala
  • Christodoulos A. Floudas
Article
First Online:
  • 318 Downloads

Abstract

A surrogate-based optimization method is presented, which aims to locate the global optimum of box-constrained problems using input–output data. The method starts with a global search of the n-dimensional space, using a Smolyak (Sparse) grid which is constructed using Chebyshev extrema in the one-dimensional space. The collected samples are used to fit polynomial interpolants, which are used as surrogates towards the search for the global optimum. The proposed algorithm adaptively refines the grid by collecting new points in promising regions, and iteratively refines the search space around the incumbent sample until the search domain reaches a minimum hyper-volume and convergence has been attained. The algorithm is tested on a large set of benchmark problems with up to thirty dimensions and its performance is compared to a recent algorithm for global optimization of grey-box problems using quadratic, kriging and radial basis functions. It is shown that the proposed algorithm has a consistently reliable performance for the vast majority of test problems, and this is attributed to the use of Chebyshev-based Sparse Grids and polynomial interpolants, which have not gained significant attention in surrogate-based optimization thus far.

Keywords

Surrogate-based optimization Black-box optimization Smolyak grids Sparse-grids Chebyshev Polynomial interpolation 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

The authors would like to acknowledge the contribution of the late Professor Christodoulos A Floudas who introduced the authors to this area. CAF was a visionary and pioneer in the areas of global optimization, computational biology, energy systems, grey-box optimization and machine learning. This work was initiated when CAK and FB were postdoctoral associates in the laboratory of CAF, but was completed after his passing when they joined Georgia Institute of Technology.

References

  1. 1.
    Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Conn, A.R., Le Digabel, S.: Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim. Methods Softw. 28(1), 139–158 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Davis, E., Ierapetritou, M.: A kriging-based approach to MINLP containing black-box models and noise. Ind. Eng. Chem. Res. 47(16), 6101–6125 (2008)CrossRefGoogle Scholar
  4. 4.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  5. 5.
    Regis, R.G.: Constrained optimization by radial basis function interpolation for high-dimensional expensive black-box problems with infeasible initial points. Eng. Optim. 46(2), 218–243 (2014)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Audet, C., Dennis Jr., J.E.: A progressive barrier for derivative-free nonlinear programming. SIAM J. Optim. 20(1), 445–472 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Boukouvala, F., Ierapetritou, M.G.: Derivative-free optimization for expensive constrained problems using a novel expected improvement objective function. AIChE J. 60(7), 2462–2474 (2014)CrossRefGoogle Scholar
  8. 8.
    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)zbMATHCrossRefGoogle Scholar
  9. 9.
    Echebest, N., Schuverdt, M.L., Vignau, R.P.: A derivative-free method for solving box-constrained underdetermined nonlinear systems of equations. Appl. Math. Comput. 219(6), 3198–3208 (2012)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Le Thi, H.A., Vaz, A.I.F., Vicente, L.N.: Optimizing radial basis functions by d.c. programming and its use in direct search for global derivative-free optimization. Top 20(1), 190–214 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Boukouvala, F., Floudas, C.A.: ARGONAUT: AlgoRithms for global optimization of coNstrAined grey-box compUTational problems. Optim. Lett. 11(5), 895–913 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Eason, J.P., Biegler, L.T.: A trust region filter method for glass box/black box optimization. AIChE J. 62(9), 3124–3136 (2016)CrossRefGoogle Scholar
  14. 14.
    Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. 4OR 12(4), 301–333 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  15. 15.
    Amaran, S., et al.: Simulation optimization: a review of algorithms and applications. Ann. Oper. Res. 240(1), 351–380 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Cozad, A., Sahinidis, N.V., Miller, D.C.: Learning surrogate models for simulation-based optimization. AIChE J. 60(6), 2211–2227 (2014)CrossRefGoogle Scholar
  17. 17.
    Jakobsson, S., et al.: A method for simulation based optimization using radial basis functions. Optim. Eng. 11(4), 501–532 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Quan, N., et al.: Simulation optimization via kriging: a sequential search using expected improvement with computing budget constraints. IIE Trans. 45(7), 763–780 (2013)CrossRefGoogle Scholar
  19. 19.
    Das, S., Suganthan, P.N.: Differential evolution: a survey of the state-of-the-art. IEEE Trans. Evol. Comput. 15(1), 4–31 (2011)CrossRefGoogle Scholar
  20. 20.
    Egea, J.A., Martí, R., Banga, J.R.: An evolutionary method for complex-process optimization. Comput. Oper. Res. 37(2), 315–324 (2010)zbMATHCrossRefGoogle Scholar
  21. 21.
    Ong, Y.S., Nair, P.B., Keane, A.J.: Evolutionary optimization of computationally expensive problems via surrogate modeling. Aiaa J. 41(4), 687–696 (2003)CrossRefGoogle Scholar
  22. 22.
    Boukouvala, F., Misener, R., Floudas, C.A.: Global optimization advances in mixed-integer nonlinear programming, MINLP, and constrained derivative-free optimization, CDFO. Eur. J. Oper. Res. 252(3), 701–727 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Hooke, R., Jeeves, T.A.: Direct search solution of numerical and statistical problems. J. ACM 8(2), 212–229 (1961)zbMATHCrossRefGoogle Scholar
  24. 24.
    Nedler, J.A., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Booker, A.J., et al.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17(1), 1–13 (1999)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Boukouvala, F., Ierapetritou, M.G.: Surrogate-based optimization of expensive flowsheet modeling for continuous pharmaceutical manufacturing. J. Pharm. Innov. 8(2), 131–145 (2013)CrossRefGoogle Scholar
  27. 27.
    Caballero, J.A., Grossmann, I.E.: An algorithm for the use of surrogate models in modular flowsheet optimization. AIChE J. 54(10), 2633–2650 (2008)CrossRefGoogle Scholar
  28. 28.
    Ciaurri, D.E., Mukerji, T., Durlofsky, L.J.: Derivative-free optimization for oil field operations. In: Yang, X.-S., Koziel, S. (eds.) Computational Optimization and Applications in Engineering and Industry, pp. 19–55. Springer, Berlin (2011)CrossRefGoogle Scholar
  29. 29.
    Egea, J.A., et al.: Scatter search for chemical and bio-process optimization. J. Glob. Optim. 37(3), 481–503 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Torn, A., Zilinskas, A.: Global Optimization. Springer, New York (1989)zbMATHCrossRefGoogle Scholar
  31. 31.
    Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21(4), 345–383 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  32. 32.
    Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J.-P. (eds.) Advances in Optimization and Numerical Analysis, p. 51–67. Springer, Dordrecht (1994)Google Scholar
  33. 33.
    Powell, M.J.: The BOBYQA Algorithm for Bound Constrained Optimization Without Derivatives. Cambridge NA Report NA2009/06. University of Cambridge, Cambridge (2009)Google Scholar
  34. 34.
    Wilson, Z.T., Sahinidis, N.V.: The ALAMO approach to machine learning. Comput. Chem. Eng. 106, 785–795 (2017)CrossRefGoogle Scholar
  35. 35.
    Davis, E., Ierapetritou, M.: A kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions. J. Glob. Optim. 43(2–3), 191–205 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Palmer, K., Realff, M.: Optimization and validation of steady-state flowsheet simulation metamodels. Chem. Eng. Res. Des. 80(7), 773–782 (2002)CrossRefGoogle Scholar
  37. 37.
    Boukouvala, F., Hasan, M.F., Floudas, C.A.: Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption. J. Global Optim. 67(1–2), 3–42 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Gutmann, H.-M.: A radial basis function method for global optimization. J. Glob. Optim. 19(3), 201–227 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Muller, J., Shoemaker, C.A., Piche, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40(5), 1383–1400 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Regis, R.G., Shoemaker, C.A.: A quasi-multistart framework for global optimization of expensive functions using response surface models. J. Glob. Optim. 56(4), 1719–1753 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Eason, J., Cremaschi, S.: Adaptive sequential sampling for surrogate model generation with artificial neural networks. Comput. Chem. Eng. 68, 220–232 (2014)CrossRefGoogle Scholar
  42. 42.
    Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012)CrossRefGoogle Scholar
  43. 43.
    Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. Comput. Aided Chem. Eng. 28, 1129–1134 (2010)CrossRefGoogle Scholar
  44. 44.
    Martelli, E., Amaldi, E.: PGS-COM: a hybrid method for constrained non-smooth black-box optimization problems: Brief review, novel algorithm and comparative evaluation. Comput. Chem. Eng. 63, 108–139 (2014)CrossRefGoogle Scholar
  45. 45.
    Novak, E., et al.: Smolyak/sparse grid algorithms. Tractability Multivar. Probl. Std. Inf. Funct. 12, 320–397 (2010)Google Scholar
  46. 46.
    Plaskota, L., Wasilkowski, G.W.: Smolyak’s algorithm for integration and L-1-approximation of multivariate functions with bounded mixed derivatives of second order. Numer. Algorithms 36(3), 229–246 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Bungartz, H.J., Dirnstorfer, S.: Multivariate quadrature on adaptive sparse grids. Computing 71(1), 89–114 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Bungartz, H.J., Dirnstorfer, S.: Higher order quadrature on sparse grids. In: Computational Science—Iccs 2004. In: Bubak, M., et al. (Ed.) Proceedings, p. 394–401 (2004)Google Scholar
  49. 49.
    Bungartz, H.-J., Pfluger, D., Zimmer, S.: Adaptive sparse grid techniques for data mining. Springer, Berlin (2008)CrossRefGoogle Scholar
  50. 50.
    Harding, B.: Adaptive Sparse Grids and Extrapolation Techniques. In: Garcke, J., Pfluger, D. (eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Berlin (2016)Google Scholar
  51. 51.
    Jiang, Y., Xu, Y.S.: B-spline quasi-interpolation on sparse grids. J. Complex. 27(5), 466–488 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  52. 52.
    Pfluger, D., Peherstorfer, B., Bungartz, H.J.: Spatially adaptive sparse grids for high-dimensional data-driven problems. J. Complex. 26(5), 508–522 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  53. 53.
    Sickel, W., Ullrich, T.: Spline interpolation on sparse grids. Appl. Anal. 90(3–4), 337–383 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  54. 54.
    Zenger, C.: Sparse grids. In: Hackbusch, W. (ed.) Parallel algorithms for partial differential equations. Proceedings of the Sixth GAMM Seminar. Notes on numerical fluid mechanics. Braunschweig, Verlag Vieweg, vol 31 (1991)Google Scholar
  55. 55.
    Novak, E., Ritter, K.: Global optimization using hyperbolic cross points. In: Floudas, C.A., Pardalos, P.M. (eds.) State of the Art in Global Optimization: Computational Methods and Applications. Springer, Boston (1996)Google Scholar
  56. 56.
    Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. In: Dokl. Akad. Nauk SSSR (1963)Google Scholar
  57. 57.
    Gerstner, T., Griebel, M.: Numerical integration using sparse grids. Numer. Algorithms 18(3–4), 209–232 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Dung, D.: Sampling and cubature on sparse grids based on a B-spline quasi-interpolation. Found. Comput. Math. 16(5), 1193–1240 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    Tang, J.J., et al.: Dimension-adaptive sparse grid interpolation for uncertainty quantification in modern power systems: probabilistic power flow. IEEE Trans. Power Syst. 31(2), 907–919 (2016)CrossRefGoogle Scholar
  60. 60.
    Peherstorfer, B., et al.: Selected recent applications of sparse grids. Numer. Math. Theory Methods Appl. 8(1), 47–77 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  61. 61.
    Gajda, P.: Smolyak’s algorithm for weighted L-1-approximation of multivariate functions with bounded rth mixed derivatives over R-d. Numer. Algorithms 40(4), 401–414 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  62. 62.
    Xu, G.Q.: On weak tractability of the Smolyak algorithm for approximation problems. J. Approx. Theory 192, 347–361 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    Wasilkowski, G.W., Wozniakowski, H.: Explicit cost bounds of algorithms for multivariate tensor product problems. J. Complex. 11(1), 1–56 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  64. 64.
    Valentin, J., Pfluger, D.: Hierarchical gradient-based optimization with b-splines on sparse grids. In: Garcke, J., Pfluger, D. (Eds.) Sparse Grids and Applications—Stuttgart 2014. Lecture Notes in Computational Science and Engineering, vol 109. Springer, Cham (2016)Google Scholar
  65. 65.
    Grimstad, B., Sandnes, A.: Global optimization with spline constraints: a new branch-and-bound method based on B-splines. J. Glob. Optim. 65(3), 401–439 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  66. 66.
    Hulsmann, M., Reith, D.: SpaGrOW-A derivative-free optimization scheme for intermolecular force field parameters based on sparse grid methods. Entropy 15(9), 3640–3687 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  67. 67.
    Sankaran, S.: Stochastic optimization using a sparse grid collocation scheme. Probab. Eng. Mech. 24(3), 382–396 (2009)CrossRefGoogle Scholar
  68. 68.
    Chen, P., Quarteroni, A.: A new algorithm for high-dimensional uncertainty quantification based on dimension-adaptive sparse grid approximation and reduced basis methods. J. Comput. Phys. 298, 176–193 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  69. 69.
    Pronzato, L., Müller, W.G.: Design of computer experiments: space filling and beyond. Stat. Comput. 22(3), 681–701 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  70. 70.
    McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)MathSciNetzbMATHGoogle Scholar
  71. 71.
    Davis, E., Ierapetritou, M.: A centroid-based sampling strategy for kriging global modeling and optimization. AIChE J. 56(1), 220–240 (2010)Google Scholar
  72. 72.
    Owen, A.B.: Orthogonal arrays for computer experiments, integration and visualization. Stat. Sin. 2(2), 439–452 (1992)MathSciNetzbMATHGoogle Scholar
  73. 73.
    Garud, S.S., Karimi, I.A., Kraft, M.: Smart sampling algorithm for surrogate model development. Comput. Chem. Eng. 96, 103–114 (2017)CrossRefGoogle Scholar
  74. 74.
    Fornberg, B., Zuev, J.: The Runge phenomenon and spatially variable shape parameters in RBF interpolation. Comput. Math Appl. 54(3), 379–398 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  75. 75.
    Runge, C.: Über empirische Funktionen und die Interpolation zwischen äquidistanten Ordinaten. Zeitschrift für Mathematik und Physik 46(224–243), 20 (1901)zbMATHGoogle Scholar
  76. 76.
    Xiang, S., Chen, X., Wang, H.: Error bounds for approximation in Chebyshev points. Numer. Math. 116(3), 463–491 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  77. 77.
    Barthelmann, V., Novak, E., Ritter, K.: High dimensional polynomial interpolation on sparse grids. Adv. Comput. Math. 12(4), 273–288 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  78. 78.
    Davis, P.J., Rabinowitz, P.: Methods of Numerical Integration. Dover Publications (2007)Google Scholar
  79. 79.
    Judd, K.L., et al.: Smolyak method for solving dynamic economic models: lagrange interpolation, anisotropic grid and adaptive domain. J. Econ. Dyn. Control 44, 92–123 (2014)MathSciNetCrossRefGoogle Scholar
  80. 80.
    Trefethen, N.: Six myths of polynomial interpolation and quadrature. Math. Today 47, 184–188 (2011)MathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Wallace H. Coulter Department of Biomedical EngineeringGeorgia Institute of TechnologyAtlantaUSA
  2. 2.School of Chemical and Biomolecular EngineeringGeorgia Institute of TechnologyAtlantaUSA
  3. 3.Texas A&M Energy InstituteTexas A&M UniversityCollege StationUSA
  4. 4.Department of Chemical EngineeringTexas A&M UniversityCollege StationUSA

Personalised recommendations

Cite article

Buy options