Soft Computing

, Volume 23, Issue 23, pp 12417–12436 | Cite as

A comparison of quality measures for model selection in surrogate-assisted evolutionary algorithm

  • Haibo YuEmail author
  • Ying Tan
  • Chaoli Sun
  • Jianchao Zeng
Methodologies and Application


Choosing a proper approximation model should be the first and the most fundamental problem to be solved when dealing with surrogate-assisted evolutionary algorithms. Till now, most of the model selection methods emphasize on obtaining the best surrogate model basing on model accuracy assessments. As the population ranking is of the most important part in evolutionary optimization, the target function of surrogate model should focus on the right ranking of candidate solutions. Therefore, in this paper, we make a comparison study on several model quality measures which basically dedicated to measuring the capability of surrogate model in selecting and ranking the candidate solutions. In order to investigate the compatibility between accuracy assessments and ranking correlation methods, four algorithms with different model selection strategies based on different quality measures are designed and comparative study is made by contrasting them to three specific surrogate-assisted evolutionary algorithms as well as the standard particle swarm optimization. Simulation results on ten commonly used benchmark problems and one engineering case demonstrate the efficacy of the designed model selection strategies and meanwhile provide further insight into the three model quality measures studied in this paper in model selection.


Surrogate Model selection Bootstrap sampling Quality measure Particle swarm optimization 



This work is supported by National Natural Science Foundation of China (Grant Nos. 61472269 and 61403272) and the State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, China, as well as the Fund Program for the Scientific Activities of Selected Returned Overseas Professionals in Shanxi Province.

Compliance with ethical standards

Conflict of interest

Author Haibo Yu declares that he has no conflict of interest. Author Ying Tan declares that she has no conflict of interest. Author Chaoli Sun declares that she has no conflict of interest. Author Jianchao Zeng declares that he has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. Buche D, Schraudolph NN, Koumoutsakos P (2005) Accelerating evolutionary algorithms with Gaussian process fitness function models. IEEE Trans Syst Man Cybern C 35:183–194. CrossRefGoogle Scholar
  2. Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 1953, pp 1–1957.
  3. Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1:3–18. CrossRefGoogle Scholar
  4. Díaz-Manríquez A, Toscano-Pulido G, Gómez-Flores W (2011) On the selection of surrogate models in evolutionary optimization algorithms. In: 2011 IEEE congress of evolutionary computation (CEC), pp 2155–2162.
  5. Díaz-Manríquez A, Toscano G, Coello Coello CA (2016) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput. CrossRefGoogle Scholar
  6. Díaz-Manríquez A, Toscano G, Coello Coello CA (2017) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput 21:5647–5663. CrossRefGoogle Scholar
  7. Efron B, Tibshirani RJ (1994) An introduction to the bootstrap. CRC Press, Boca RatonzbMATHGoogle Scholar
  8. Emmerich MTM, Giannakoglou KC, Naujoks B (2006) Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10:421–439. CrossRefGoogle Scholar
  9. Fisher NI, Hall P (1991) Bootstrap algorithms for small samples. J Stat Plan Inference 27:157–169. MathSciNetCrossRefGoogle Scholar
  10. Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45:50–79. CrossRefGoogle Scholar
  11. Hüsken M, Jin Y, Sendhoff B (2005) Structure optimization of neural networks for evolutionary design optimization. Soft Comput 9:21–28. CrossRefGoogle Scholar
  12. Ingu T, Takagi H (1999) Accelerating a GA convergence by fitting a single-peak function. In: Fuzzy systems conference proceedings, FUZZ-IEEE ‘99. 1999 IEEE International, vol 1413, pp 1415–1420.
  13. Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9:3–12CrossRefGoogle Scholar
  14. Jin Y (2011) Surrogate-assisted evolutionary computation: recent advances and future challenges. Swarm Evol Comput 1:61–70. CrossRefGoogle Scholar
  15. Jin Y, Michael H (2003) Quality measures for approximate models in evolutionary computation. In: GECCO, pp 170–173Google Scholar
  16. Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscipl Optim 23:1–13. CrossRefGoogle Scholar
  17. Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13:455–492. MathSciNetCrossRefzbMATHGoogle Scholar
  18. Lawrence CT, Tits AL (2001) A computationally efficient feasible sequential quadratic programming algorithm. SIAM J Optim 11:1092–1118. MathSciNetCrossRefzbMATHGoogle Scholar
  19. Le MN, Ong YS, Menzel S, Jin Y, Sendhoff B (2013) Evolution by adapting surrogates. Evol Comput 21:313–340. CrossRefGoogle Scholar
  20. Lendasse A, Wertz V, Verleysen M (2003) Model selection with cross-validations and bootstraps: application to time series prediction with RBFN models. In: Artificial neural networks and neural information processing–ICANN/ICONIP 2003, pp 573–580. Springer, BerlinGoogle Scholar
  21. Lesh FH (1959) Multi-dimensional least-squares polynomial curve fitting. Commun ACM 2:29–30. CrossRefzbMATHGoogle Scholar
  22. Liang K-H, Yao X, Newton C (2000) Evolutionary search of approximated n-dimensional landscapes. Int J Knowl Based Intell Eng Syst 4:172–183Google Scholar
  23. Lim D, Ong Y-S, Jin Y, Sendhoff B (2007) A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. Paper presented at the Proceedings of the 9th annual conference on Genetic and evolutionary computation, London, EnglandGoogle Scholar
  24. Lim D, Ong Y-S, Jin Y, Sendhoff B (2008) Evolutionary optimization with dynamic fidelity computational models. In: Huang D-S, Wunsch DC, Levine DS, Jo K-H (eds) Advanced intelligent computing theories and applications. With Aspects of artificial intelligence: 4th international conference on intelligent computing, ICIC 2008 Shanghai, China, September 15-18, 2008. Springer, Berlin, pp 235–242
  25. Lim D, Jin Y, Ong YS, Sendhoff B (2010) Generalizing surrogate-assisted evolutionary computation. IEEE Trans Evol Comput 14:329–355. CrossRefGoogle Scholar
  26. Liu H, Hervas J-R, Ong Y-S, Cai J, Wang Y (2018) An adaptive RBF-HDMR modeling approach under limited computational budget. Struct Multidiscipl Optim 57:1233–1250. CrossRefGoogle Scholar
  27. Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0Google Scholar
  28. Matheron G (1963) Principles of geostatistics. Econ Geol 58:1246–1266CrossRefGoogle Scholar
  29. Mokarram V, Banan MR (2018) A new PSO-based algorithm for multi-objective optimization with continuous and discrete design variables. Struct Multidiscipl Optim 57:509–533. MathSciNetCrossRefGoogle Scholar
  30. Munk DJ, Vio GA, Steven GP (2015) Topology and shape optimization methods using evolutionary algorithms: a review. Struct Multidiscipl Optim 52:613–631. MathSciNetCrossRefGoogle Scholar
  31. Nakayama H, Arakawa M, Sasaki R (2001) A computational intelligence approach to optimization with unknown objective functions. In: Artificial neural networks—ICANN 2001. Springer, Berlin, pp 73–80Google Scholar
  32. Powell M (2001) Radial basis function methods for interpolation to functions of many variables. In: HERCMA. Citeseer, pp 2–24Google Scholar
  33. Praveen C, Duvigneau R (2009) Low cost PSO using metamodels and inexact pre-evaluation: application to aerodynamic shape design. Comput Methods Appl Mech Eng 198:1087–1096. CrossRefzbMATHGoogle Scholar
  34. Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Kevin Tucker P (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28. CrossRefzbMATHGoogle Scholar
  35. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4:409–423MathSciNetCrossRefGoogle Scholar
  36. Sadoughi M, Hu C, MacKenzie CA, Eshghi AT, Lee S (2018) Sequential exploration-exploitation with dynamic trade-off for efficient reliability analysis of complex engineered systems. Struct Multidiscipl Optim 57:235–250. MathSciNetCrossRefGoogle Scholar
  37. Shi L, Rasheed K (2008) ASAGA: an adaptive surrogate-assisted genetic algorithm. Paper presented at the Proceedings of the 10th annual conference on Genetic and evolutionary computation, Atlanta, GA, USAGoogle Scholar
  38. Shi L, Rasheed K (2010) A survey of fitness approximation methods applied in evolutionary algorithms. In: Tenne Y, Goh C-K (eds) Computational intelligence in expensive optimization problems. Springer, Berlin, pp 3–28.
  39. Simpson TW, Poplinski JD, Koch PN, Allen JK (2001) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17:129–150. CrossRefzbMATHGoogle Scholar
  40. Stern RE, Song J, Work DB (2017) Accelerated Monte Carlo system reliability analysis through machine-learning-based surrogate models of network connectivity. Reliab Eng Syst Safe 164:1–9. CrossRefGoogle Scholar
  41. Tang Y, Chen J, Wei J (2013) A surrogate-based particle swarm optimization algorithm for solving optimization problems with expensive black box functions. Eng Optim 45:557–576. MathSciNetCrossRefGoogle Scholar
  42. Wang GG, Shan S (2006) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129:370–380. CrossRefGoogle Scholar
  43. Wang D, Wu Z, Fei Y, Zhang W (2014) Structural design employing a sequential approximation optimization approach. Comput Struct 134:75–87. CrossRefGoogle Scholar
  44. Wang H, Jin Y, Doherty J (2017) Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems. IEEE Trans Cybern 47:2664–2677. CrossRefGoogle Scholar
  45. Wang H, Jin Y, Sun C, Doherty J (2018) Offline data-driven evolutionary optimization using selective surrogate ensembles. IEEE Trans Evol Comput. CrossRefGoogle Scholar
  46. Wu Z, Wang D, Okolo NP, Hu F, Zhang W (2016) Global sensitivity analysis using a gaussian radial basis function metamodel. Reliab Eng Syst Safe 154:171–179. CrossRefGoogle Scholar
  47. Yang D, Flockton SJ (1995) Evolutionary algorithms with a coarse-to-fine function smoothing. In: IEEE international conference on evolutionary computation. IEEE, pp 657–662Google Scholar
  48. Yang Q, Xue D (2015) Comparative study on influencing factors in adaptive metamodeling. Eng Comput 31:561–577. CrossRefGoogle Scholar
  49. Yang Q, Kianimanesh A, Freiheit T, Park SS, Xue D (2011) A semi-empirical model considering the influence of operating parameters on performance for a direct methanol fuel cell. J Power Sources 196:10640–10651. CrossRefGoogle Scholar
  50. Yew Soon O, Keane AJ (2004) Meta-Lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8:99–110. CrossRefGoogle Scholar
  51. Younis A, Dong Z (2010) Trends, features, and tests of common and recently introduced global optimization methods. Eng Optim 42:691–718. MathSciNetCrossRefGoogle Scholar
  52. Yu H, Tan Y, Sun C, Zeng J, Jin Y (2016) An adaptive model selection strategy for surrogate-assisted particle swarm optimization algorithm. In: 2016 IEEE symposium series on computational intelligence (SSCI), pp 1–8
  53. Yu H, Tan Y, Sun C, Zeng J (2018a) A generation-based optimal restart strategy for surrogate-assisted social learning particle swarm optimization. Knowl Based Syst 10:1–12. CrossRefGoogle Scholar
  54. Yu H, Tan Y, Zeng J, Sun C, Jin Y (2018b) Surrogate-assisted hierarchical particle swarm optimization. Inf Sci 454–455:59–72. MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Mechanical EngineeringTaiyuan University of Science and TechnologyTaiyuanChina
  2. 2.Division of Industrial and System EngineeringTaiyuan University of Science and TechnologyTaiyuanChina
  3. 3.School of Computer Science and TechnologyTaiyuan University of Science and TechnologyTaiyuanChina
  4. 4.School of Computer Science and Control EngineeringNorth University of ChinaTaiyuanChina

Personalised recommendations