Abstract
We consider engineering design optimization problems where the objective and/or constraint functions are evaluated by means of computationally expensive blackboxes. Our practical optimization strategy consists of solving surrogate optimization problems in the search step of the mesh adaptive direct search algorithm. In this paper, we consider locally weighted regression models to build the necessary surrogates, and present three ideas for appropriate and effective use of locally weighted scatterplot smoothing (LOWESS) models for surrogate optimization. First, a method is proposed to reduce the computational cost of LOWESS models. Second, a local scaling coefficient is introduced to adapt LOWESS models to the density of neighboring points while retaining smoothness. Finally, an appropriate order error metric is used to select the optimal shape coefficient of the LOWESS model. Our surrogate-assisted optimization approach utilizes LOWESS models to both generate and rank promising candidates found in the search and poll steps. The “real” blackbox functions that govern the original optimization problem are then evaluated at these ranked candidates with an opportunistic strategy, reducing CPU time significantly. Computational results are reported for four engineering design problems with up to six variables and six constraints. The results demonstrate the effectiveness of the LOWESS models as well as the order error metric for surrogate optimization.
Similar content being viewed by others
Notes
A nychthemeron is a time period of 24 consecutive hours.
References
Abramson MA, Audet C, Couture G, Dennis JE Jr, Le Digabel S, Tribes C (2015) The NOMAD project. Software available at https://www.gerad.ca/nomad
Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37(3):279–294
Adams BM, Ebeida MS, Eldred MS, Geraci G, Jakeman JD, Maupin KA, Monschke JA, Swiler LP, Stephens JA, Vigil DM, Wildey TM, Bohnhoff WJ, Dalbey KR, Eddy JP, Frye JR, Hooper RW, Hu KT, Hough PD, Khalil M, Ridgway EM, Rushdi A (2014) Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: version 6.6 users manual. Technical report SAND2014-4633, Sandia National Laboratories
Arora J (2004) Introduction to optimum design. Academic Press, London
Atkeson CG, Moore AW, Schaal S (1997) Locally weighted learning for control. Artif Intell Rev 11(1):11–73
Audet C, Dennis JE Jr (2006) Mesh adaptive direct search algorithms for constrained optimization. SIAM J Optim 17(1):188–217
Audet C, Dennis JE Jr (2009) A progressive barrier for derivative-free nonlinear programming. SIAM J Optim 20(1):445–472
Audet C, Kokkolaras M, Le Digabel S, Talgorn B (2017) Order-based error for managing ensembles of surrogates in mesh adaptive direct search. J Glob Optim. ISSN: 1573-2916. https://doi.org/10.1007/s10898-017-0574-1
Belegundu AD (1982) A study of mathematical programming methods for structural optimization. University of Iowa, Iowa City
Booker AJ, Dennis JE Jr, Frank PD, Serafini DB, Torczon V, Trosset MW (1999) A rigorous framework for optimization of expensive functions by surrogates. Struct Multidiscip Optim 17(1):1–13
Chen X, Kelley CT (2016) Optimization with hidden constraints and embedded Monte Carlo computations. Optim Eng 17(1):157–175
Choi SC, Wette R (1969) Maximum likelihood estimation of the parameters of the Gamma distribution and their bias. Technometrics 11(4):683–690
Cleveland WS (1979) Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc 74:829–836
Cleveland WS (1981) LOWESS: a program for smoothing scatterplots by robust locally weighted regression. Am Stat 35(1):54
Cleveland WS, Devlin SJ (1988) Locally weighted regression: an approach to regression analysis by local fitting. J Am Stat Assoc 83:596–610
Cleveland WS, Devlin SJ, Grosse E (1988) Regression by local fitting: methods, properties, and computational algorithms. J Econ 37(1):87–114
Cohen RA (1999) An introduction to PROC LOESS for local regression. In: Proceedings of the 24th SAS users group international conference. http://www.ats.ucla.edu/stat/SAS/library/
Conn AR, Le Digabel S (2013) Use of quadratic models with mesh-adaptive direct search for constrained black box optimization. Optim Methods Softw 28(1):139–158
Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78(382):316–331
Fan J (1993) Local linear regression smoothers and their minimax efficiencies. Ann Stat 21(1):196–216
Fan J, Gijbels I (1992) Variable bandwidth and local linear regression smoothers. Ann Stat 20(4):2008–2036
Fletcher R, Leyffer S (2002) Nonlinear programming without a penalty function. Math Program Ser A 91:239–269
Gablonsky J (2001) DIRECT version 2.0 userguide technical report. Technical report CRSC-TR01-08, North Carolina State University, Center for Research in Scientific Computation, Raleigh, NC
Garg H (2014) Solving structural engineering design optimization problems using an artificial bee colony algorithm. J Ind Manag Optim 10(3):777–794
Garneau M Lemyre (2015) Modelling of a solar thermal power plant for benchmarking blackbox optimization solvers. Master’s thesis, École Polytechnique de Montréal. https://publications.polymtl.ca/1996/
Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning, 1st edn. Addison-Wesley, Boston
Gramacy RB, Gray GA, Le Digabel S, Lee HKH, Ranjan P, Wells G, Wild SM (2016) Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics 58(1):1–11
Gramacy RB, Le Digabel S (2015) The mesh adaptive direct search algorithm with treed Gaussian process surrogates. Pac J Optim 11(3):419–447
Haftka RT, Gurdal Z (1992) Elements of structural optimization, volume 11 of solid mechanics and its applications. Springer, Berlin
Hastie T, Tibshirani R, Friedman J (2001) The elements of statistical learning. Springer series in statistics. Springer, New York
Kannan A, Wild SM (2012) Benefits of deeper analysis in simulation-based groundwater optimization problems. In: Proceedings of the XIX international conference on computational methods in water resources (CMWR 2012)
Lall U, Moon YI, Kwon HH, Bosworth K (2006) Locally weighted polynomial regression: parameter choice and application to forecasts of the Great Salt Lake. Water Resour Res 42(W05422):1–11
Le Digabel S (2011) Algorithm 909: NOMAD: nonlinear optimization with the MADS algorithm. ACM Trans Math Softw 37(4):44:1–44:15
Le Digabel S, Wild SM (2015) A taxonomy of constraints in simulation-based optimization. Technical report G-2015-57, Les cahiers du GERAD
Loader C (1999) Local regression and likelihood. Springer, New York
Macaulay FR (1931) Curve fitting and graduation. In: The smoothing of time series. National Bureau of Economic Research. NBER, pp 31–42
Matott LS, Rabideau AJ, Craig JR (2006) Pump-and-treat optimization using analytic element method flow models. Adv Water Resour 29(5):760–775
Matott LS, Leung K, Sim J (2011) Application of MATLAB and Python optimizers to two case studies involving groundwater flow and contaminant transport modeling. Comput Geosci 37(11):1894–1899
McKay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245
Minka TP (2002) Estimating a Gamma distribution. https://tminka.github.io/papers/minka-gamma.pdf
Moré JJ, Wild SM (2009) Benchmarking derivative-free optimization algorithms. SIAM J Optim 20(1):172–191
Müller H-G (1987) Weighted local regression and Kernel methods for nonparametric curve fitting. J Am Stat Assoc 82(397):231–238
Müller J, Piché R (2011) Mixture surrogate models based on Dempster–Shafer theory for global optimization problems. J Global Optim 51(1):79–104
Natrella M (2010) NIST/SEMATECH e-handbook of statistical methods. NIST/SEMATECH. https://www.itl.nist.gov/div898/handbook/
Orr MJL (1996) Introduction to radial basis function networks. Technical report, Center for Cognitive Science, University of Edinburgh
Pourmohamad T (2016) Combining multivariate stochastic process models with filter methods for constrained optimization. Ph.D. thesis. UC Santa Cruz: Statistics and Applied Mathematics
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucher PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28
Ruppert D, Wand MP (1994) Multivariate locally weighted least squares regression. Ann Stat 22(3):1346–1370
Stone M (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. J Roy Stat Soc Ser B (Methodol) 39(1):44–47
Ting J-A, Vijayakumar S, Schaal S (2010) Locally weighted regression for control. Springer, Boston, pp 613–624
Wilks DS (1990) Maximum likelihood estimation for the Gamma distribution using data containing zeros. J Clim 3(12):1495–1501
Acknowledgements
All the authors acknowledge the partial support of FRQNT Grant PR-182098; B. Talgorn and M. Kokkolaras are also grateful for the partial support of NSERC/Hydro-Québec Grant EGP2 498903-16; such support does not constitute an endorsement by the sponsors of the opinions expressed in this article. B. Talgorn would like to thank Stéphane Alarie of IREQ, Patricia Gillett-Kawamoto of GERAD and Sylvain Arreckx of GERAD for their invaluable insights and comments.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Talgorn, B., Audet, C., Le Digabel, S. et al. Locally weighted regression models for surrogate-assisted design optimization. Optim Eng 19, 213–238 (2018). https://doi.org/10.1007/s11081-017-9370-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11081-017-9370-5