Structural and Multidisciplinary Optimization

, Volume 59, Issue 1, pp 93–116 | Cite as

Efficient global optimization with ensemble and selection of kernel functions for engineering design

  • Pramudita Satria PalarEmail author
  • Koji Shimoyama


In this paper, we investigate the use of multiple kernel functions for assisting single-objective Kriging-based efficient global optimization (EGO). The primary objective is to improve the robustness of EGO in terms of the choice of kernel function for solving a variety of black-box optimization problems in engineering design. Specifically, three widely used kernel functions are studied, that is, Gaussian, Matérn-3/2, and Matérn-5/2 function. We investigate both model selection and ensemble techniques based on Akaike information criterion (AIC) and cross-validation error on a set of synthetic (noiseless and noisy) and non-algebraic (aerodynamic and parameter tuning) optimization problems; in addition, the use of cross-validation-based local (i.e., pointwise) ensemble is also studied. Since all the constituent surrogate models in the ensemble scheme are Kriging models, it is possible to perform EGO since the Kriging uncertainty structure is still preserved. Through analyses of empirical experiments, it is revealed that the ensemble techniques improve the robustness and performance of EGO. It is also revealed that the use of Matérn-kernels yields better results than those of the Gaussian kernel when EGO with a single kernel is considered. Furthermore, we observe that model selection methods do not yield any substantial improvement over single kernel EGO. When averaged across all types of problem (i.e., noise level, dimensionality, and synthetic/non-algebraic), the local ensemble technique achieves the best performance.


Efficient global optimization Kernel function Surrogate model Model selection Model ensemble 



Koji Shimoyama was supported in part by the Grant-in-Aid for Scientific Research (B) No. H1503600 administered by the Japan Society for the Promotion of Science (JSPS).


  1. Acar E (2010) Various approaches for constructing an ensemble of metamodels using local measures. Struct Multidiscip Optim 42(6):879–896CrossRefGoogle Scholar
  2. Acar E (2013) Effects of the correlation model, the trend model, and the number of training points on the accuracy of Kriging metamodels. Expert Syst 30(5):418–428CrossRefGoogle Scholar
  3. Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37(3):279–294CrossRefGoogle Scholar
  4. Archambeau C, Bach F (2011) Multiple gaussian process models. arXiv:1110.5238
  5. Bach FR (2009) Exploring large feature spaces with hierarchical multiple kernel learning. In: Advances in neural information processing systems, pp 105–112Google Scholar
  6. Bader J, Zitzler E (2011) Hype An algorithm for fast hypervolume-based many-objective optimization. Evol Comput 19(1):45–76CrossRefGoogle Scholar
  7. Bartoli N, Bouhlel M-A, Kurek I, Lafage R, Lefebvre T, Morlier J, Priem R, Stilz V, Regis R (2016) Improvement of efficient global optimization with application to aircraft wing design. In: 17th AIAA/ISSMO Multidisciplinary analysis and optimization conference, p 4001Google Scholar
  8. Ben Salem M, Tomaso L (2018) Automatic selection for general surrogate models. Struct Multidisc Optim 58(2):719–734CrossRefGoogle Scholar
  9. Bishop CM (1995) Neural networks for pattern recognition. Oxford University PressGoogle Scholar
  10. Burnham KP, Anderson DR (2003) Model selection and multimodel inference: a practical information-theoretic approach. SpringerGoogle Scholar
  11. Couckuyt I, Dhaene T, Demeester P (2014) oodace toolbox: a flexible object-oriented Kriging implementation. J Mach Learn Res 15:3183–3186zbMATHGoogle Scholar
  12. Drela M (1989) Xfoil An analysis and design system for low Reynolds number airfoils. In: Low Reynolds number aerodynamics. Springer, pp 1–12Google Scholar
  13. Dubrule O (1983) Cross validation of Kriging in a unique neighborhood. J Int Assoc Math Geol 15(6):687–699MathSciNetCrossRefGoogle Scholar
  14. Durrande N, Ginsbourger D, Roustant O, Carraro L (2011) Additive covariance kernels for high-dimensional Gaussian process modeling. arXiv:1111.6233
  15. Forrester AI, Keane AJ, Bressloff NW (2006) Design and analysis of “noisy” computer experiments. AIAA J 44(10):2331–2339CrossRefGoogle Scholar
  16. Ginsbourger D, Helbert C, Carraro L (2008) Discrete mixtures of kernels for Kriging-based optimization. Qual Reliab Eng Int 24(6):681–691CrossRefGoogle Scholar
  17. Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidiscip Optim 33(3):199–216CrossRefGoogle Scholar
  18. Hennig P, Schuler CJ (2012) Entropy search for information-efficient global optimization. J Mach Learn Res 13(Jun):1809–1837MathSciNetzbMATHGoogle Scholar
  19. Jeong S, Murayama M, Yamamoto K (2005) Efficient optimization design method using Kriging model. J Aircraft 42(2):413–420CrossRefGoogle Scholar
  20. Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Global Optim 21 (4):345–383MathSciNetCrossRefGoogle Scholar
  21. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13(4):455–492MathSciNetCrossRefGoogle Scholar
  22. Kanazaki M, Matsuno T, Maeda K, Kawazoe H (2015) Efficient global optimization applied to wind tunnel evaluation-based optimization for improvement of flow control by plasma actuators. Eng Optim 47(9):1226–1242CrossRefGoogle Scholar
  23. Kleijnen JP, van Beers W, Van Nieuwenhuyse I (2012) Expected improvement in efficient global optimization through bootstrapped Kriging. J Global Optim, 1–15Google Scholar
  24. Krige D (1951) A statistical approach to some mine valuation and allied problems on the Witwatersrand: By DG KrigeGoogle Scholar
  25. Kulfan BM (2008) Universal parametric geometry representation method. J Aircr 45(1):142–158CrossRefGoogle Scholar
  26. Liu H, Xu S, Wang X, Meng J, Yang S (2016) Optimal weighted pointwise ensemble of radial basis functions with different basis functions. AIAA JGoogle Scholar
  27. Marelli S, Sudret B (2014) Uqlab A framework for uncertainty quantification in matlab. In: Vulnerability, uncertainty, and risk: quantification, mitigation, and management, pp 2554–2563Google Scholar
  28. Martin JD, Simpson TW (2005) Use of Kriging models to approximate deterministic computer models. AIAA J 43(4):853–863CrossRefGoogle Scholar
  29. Matheron G (1969) Les cahiers du centre de morphologie mathématique de fontainebleau fascicule, vol 1. Le krigeage universel. Ecole de Mines de, Paris, FontainebleauGoogle Scholar
  30. Močkus J (1975) On Bayesian methods for seeking the extremum. In:Optimization Techniques IFIP technical conference. Springer, pp 400–404Google Scholar
  31. Mukhopadhyay T, Chakraborty S, Dey S, Adhikari S, Chowdhury R (2017) A critical assessment of Kriging model variants for high-fidelity uncertainty quantification in dynamics of composite shells. Arch Comput Methods Eng 24(3):495–518MathSciNetCrossRefGoogle Scholar
  32. Namura N, Obayashi S, Jeong S (2016) Efficient global optimization of vortex generators on a supercritical infinite wing. Journal of AircraftGoogle Scholar
  33. Palacios F, Alonso J, Duraisamy K, Colonno M, Hicken J, Aranake A, Campos A, Copeland S, Economon T, Lonkar A et al (2013) Stanford university unstructured (su 2): an open-source integrated computational environment for multi-physics simulation and design. In: 51st AIAA Aerospace Sciences meeting including the new horizons forum and aerospace exposition, pp 287Google Scholar
  34. Picheny V, Wagner T, Ginsbourger D (2013) A benchmark of Kriging-based infill criteria for noisy optimization. Struct Multidiscip Optim 48(3):607–626CrossRefGoogle Scholar
  35. Rasmussen CE, Williams CK (2006) Gaussian processes for machine learning, vol 1. MIT Press, CambridgezbMATHGoogle Scholar
  36. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Statist Sci, 409–423Google Scholar
  37. Samad A, Kim K-Y, Goel T, Haftka RT, Shyy W (2008) Multiple surrogate modeling for axial compressor blade shape optimization. J Propuls Power 24(2):302–310CrossRefGoogle Scholar
  38. Shi H, Gao Y, Wang X (2010) Optimization of injection molding process parameters using integrated artificial neural network model and expected improvement function method. Int J Adv Manuf Technol 48(9–12):955–962CrossRefGoogle Scholar
  39. Sóbester A, Leary SJ, Keane AJ (2004) A parallel updating scheme for approximating and optimizing high fidelity computer simulations. Struct Multidiscip Optim 27(5):371–383CrossRefGoogle Scholar
  40. Sobieczky H (1999) Parametric airfoils and wings. In: Recent development of aerodynamic design methodologies. Springer, pp 71–87Google Scholar
  41. Stein ML (2012) Interpolation of spatial data: some theory for Kriging. Springer Science & Business MediaGoogle Scholar
  42. Tesch M, Schneider J, Choset H (2011) Using response surfaces and expected improvement to optimize snake robot gait parameters. In: 2011 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, pp 1069–1074Google Scholar
  43. Viana FA, Haftka RT, Steffen V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39(4):439–457CrossRefGoogle Scholar
  44. Zhang J, Chowdhury S, Messac A (2012) An adaptive hybrid surrogate model. Struct Multidiscip Optim 46(2):223–238CrossRefGoogle Scholar
  45. Zhou X, Ma Y, Tu Y, Feng Y (2013) Ensemble of surrogates for dual response surface modeling in robust parameter design. Qual Reliab Eng Int 29(2):173–197CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018
corrected publication September/2018

Authors and Affiliations

  1. 1.Institute of Fluid ScienceTohoku UniversitySendaiJapan

Personalised recommendations