Concurrent surrogate model selection (COSMOS): optimizing model type, kernel function, and hyper-parameters

Abstract

This paper presents an automated surrogate model selection framework called the Concurrent Surrogate Model Selection or COSMOS. Unlike most existing techniques, COSMOS coherently operates at three levels, namely: 1) selecting the model type (e.g., RBF or Kriging), 2) selecting the kernel function type (e.g., cubic or multiquadric kernel in RBF), and 3) determining the optimal values of the typically user-prescribed hyper-parameters (e.g., shape parameter in RBF). The quality of the models is determined and compared using measures of median and maximum error, given by the Predictive Estimation of Model Fidelity (PEMF) method. PEMF is a robust implementation of sequential k-fold cross-validation. The selection process undertakes either a cascaded approach over the three levels or a more computationally-efficient one-step approach that solves a mixed-integer nonlinear programming problem. Genetic algorithms are used to perform the optimal selection. Application of COSMOS to benchmark test functions resulted in optimal model choices that agree well with those given by analyzing the model errors on a large set of additional test points. For the four analytical benchmark problems and three practical engineering applications – airfoil design, window heat transfer modeling, and building energy modeling – diverse forms of models/kernels are observed to be selected as optimal choices. These observations further establish the need for automated multi-level model selection that is also guided by dependable measures of model fidelity.

Appendices

Appendix A: Surrogate model candidates

1.1 Radial basis function (RBF)

The idea of using Radial Basis Functions (RBF) as approximation functions was conceived by (Hardy 1971). The RBF approximation is a linear combination of the basis functions (Ψ) computed with respect to each sample point, as given by

$$ \bar{F}(x)=W^{T}\Psi=\sum\limits_{i=1}^{n_{p}} w_{i} \psi\left( \|x-x^{i}\|\right) $$

(9)

In (9), n _p denotes the number of selected sample points; w _i ’s are the weights estimated using the pseudo inverse method, based on the training data; and ψ is the basis function that is expressed in terms of the Euclidean distance, r = ∥x − x ⁱ∥, of a point x from a given sample point, x ⁱ. The most effective forms of the basis functions are listed in Table 5 where σ represents the shape parameter of the basis function. The shape parameter in a basis function has a strong impact on the accuracy of the trained RBF. A smaller shape parameter often corresponds to a wider basis function, and the shape parameter, σ = 0, corresponds to a constant basis function (Mongillo 2011). The different RBFs considered in this paper are listed in Table 5.

Table 5 Basis or Kernel functions and their hyper-parameters in the candidate surrogate models

1.2 Kriging

Kriging (Giunta and Watson 1998) is an approach to approximate irregular data. The kriging approximation function consists of two components: (i) a global trend function, and (i i) a deviation function representing the departure from the trend function. The trend function is generally a polynomial (e.g., constant, linear, or quadratic). The general form of the kriging surrogate model is given by Cressie (1993):

$$ \bar{F}(x)= \hat{f}(x,\varphi) + Z(x) $$

(10)

where $\bar {F}(x)$ is the unknown function of interest, Z(x) is the realization of a stochastic process with the mean equal to zero, and a nonzero covariance, and $\hat {f}$ is the known approximation function

$$ \hat{f}(x,\varphi)=f(x)^{T}\varphi $$

(11)

where φ is the regression parameter matrix. The i,j − t h element of the covariance matrix, Z(x), is given by

$$ COV[Z(x^{i}),Z(x^{j})] = {\sigma_{z}^{2}} R_{ij} $$

(12)

where R _{i j} is the correlation function between the i ^th and the j ^th data points; and ${\sigma _{z}^{2}}$ is the process variance, which scales the spatial correlation function. The popular types of correlation functions are listed in Table 5. The correlation function controls the smoothness of the Kriging model estimation, based on the influence of other nearby points on the point of intrest. In Kriging, the regression function coefficients, the process variance, and the correlation function parameters, $\{\varphi ,{\sigma _{z}^{2}},\theta \}$, each can be predefined or estimated using parameter estimation methods such as Maximum Likelihood Estimation (MLE). In this paper, the regression function coefficients and the process variance are estimated using MLE, as given by

$$\begin{array}{@{}rcl@{}} \varphi&=&(F^{t} R^{-1}F)^{-1} F^{T}R^{-1}Y \\ {\sigma_{z}^{2}}&=&\frac{1}{n}(Y-F\widetilde{\varphi})^{T} R^{-1} (Y-F\widetilde{\varphi}) \end{array} $$

(13)

where Y = [y ₁ y ₂...] represents the vector of the actual output at the training points; R is a correlation matrix; and F is a matrix of f(x) evaluated by Kriging at each training point (Martin and Simpson 2005). The hyper-parameter, 𝜃, in the correlation function is determined by solving the nonlinear hyper-parameter optimization problem. In this paper, Kriging model with first-order regression polynomial function is used. To estimate the regression function coefficients and the process variance in Kriging, the DACE (design and analysis of computer experiments) package, developed by Lophaven et al. (2002), is used in this paper.

1.3 Support vector regression (SVR)

Support Vector Regression (SVR) is a relatively newer regression type surrogate model. For a given training set of instance-label pairs (x _i , y _i ), i = 1,...,n _p , where x _i ∈ R ⁿ and y ∈1,−1^m, a linear SVR is defined by f(x) =< w,x > + b, where b is a bias and < .,. > denotes the dot product. To train the SVR, the error, |ξ| = |y − f(x)|, is minimized by solving the following convex optimization problem:

$$\begin{array}{@{}rcl@{}} & & \text{Min} \left\{ \frac{1}{2} \| w \|^{2}+C {\Sigma}_{i=1}^{n_{p}} \xi_{i}+\tilde{\xi}_{i}\right\} \\ & & \text{subject to}\\ & & (w^{T}x_{i}+b)-y_{i}\leq \varepsilon+\xi_{i} \\ & & y_{i}-(w^{t}x_{i}+b)\leq \varepsilon+\tilde{\xi}_{i} \\ & & \xi_{i}, \tilde{\xi}_{i}\geq0, i=1,2,...,n_{p} \end{array} $$

(14)

In (14), ε ≥ 0 represents the difference between the actual and the predicted values; ξ _i and $\tilde {\xi }_{i}$ are the slack variables; C represents the flatness of the function; n _p represents the number of training points. By applying kernel functions, K(α,β) =< ϕ(α),ϕ(β) > , under KKT conditions, the original problem is mapped into a higher dimensional space. The dual form of SVR for nonlinear regression could be represented as

$$\begin{array}{@{}rcl@{}} & & \text{Max}\ \left\{\sum\limits_{i=1}^{n_{p}}\alpha_{i} y_{i} \,-\, \varepsilon \sum\limits_{i=1}^{n_{p}} | \alpha | \,-\, \frac{1}{2} \sum\limits_{i,j=1}^{n_{p}} \alpha_{i}.\alpha_{j} \!<\!\phi(x_{i}),\phi(x_{j})>\right\} \\ & & \text{subject to}\\ & & \sum\limits_{i=1}^{n_{p}} \alpha_{i}=0, -C \leq \alpha_{i} \leq C for i=1,...,n_{p} \end{array} $$

(15)

The standard Kernel functions used in SVR are listed in Table 5. The performance of SVR depends on its penalty parameter C and kernel parameters γ, r, and d. Using hyper-parameter optimization, correlation parameters can be estimated such that it minimizes the model error. To implement SVR in this paper, the LIBSVM (A Library for Support Vector Machines) package (Chang and Lin 2011) is used.

Appendix B: Analytical test functions

Branin-Hoo function (2 variables):

$$\begin{array}{@{}rcl@{}} f(x) \,=\, \left( x_{2} - \frac{5.1{x_{1}^{2}}}{4\pi^{2}} + \frac{5x_{1}}{\pi} -6 \right)^{2} \,+\, 10 \left( 1 - \frac{1}{8\pi} \right) cos(x_{1}) + 10 \end{array} $$

(16)

where x ₁ ∈ [−510],x ₂ ∈ [015].

Hartmann function (3 variables):

$$\begin{array}{@{}rcl@{}} f(x) \,=\, - \sum\limits_{i=1}^{4} c_{i} exp \left\{ - \sum\limits_{j=1}^{n} A_{ij} \left( x_{j} - P_{ij} \right)^{2} \right\} \end{array} $$

(17)

where x = (x ₁ x ₂…x _n )x _i ∈ [01].

In this function, the number of variables, n = 3; the constants c, A, and P, are respectively a 1 × 4 vector, a 4 × 3 matrix, and a 4 × 3 matrix:

$$\begin{array}{@{}rcl@{}} \mathrm{c} &\,=\,& [1.0, 1.2, 3.0, 3.2]^{T};\\ A&\,=\,& \left[ \begin{array}{ccc} 3.0 &10 & 30 \\ 0.1& 10& 35\\ 3.0 &10 & 30 \\ 0.1& 10& 35 \end{array} \right]\!, \text{ \!and\! } P\,=\, 10^{(-4)} \!\times\! \left[ \begin{array}{ccc} 3689 & 1170 & 2673 \\ 4699 & 4387 & 7470\\ 1091 & 8732 & 5547\\ 381 & 5743 & 8828 \end{array}\right] \end{array} $$

Perm Function (10 variables):

$$\begin{array}{@{}rcl@{}} f(x) = \sum\limits_{k=1}^{n}\left\{\sum\limits_{j=1}^{k}(j^{k}+0.5)\left[\left( \frac{x_{j}}{j}\right)^{k}-1\right]\right\}^{2} \\ where ~~~~x_{i} \in [-n n+1], i=1,...,n \\ n=10 \end{array} $$

(18)

Dixon & Price Function (50 variables):

$$\begin{array}{@{}rcl@{}} f(x) = \left( x_{1} - 1 \right)^{2} + \sum\limits_{i=2}^{n} i\left( 2{x_{i}^{2}} - x_{i-1} \right)^{2} \\ where ~~~~x_{i} \in [-10 10], i=1,...,n \\ n=50 \end{array} $$

(19)

Appendix C: Relationship between the error of the model selected by COSMOS and the size and distribution of the training data set

In this section, we explore the relationship between the error of the model selected by COSMOS and the size and distribution of the training data set for the Branin-Hoo benchmark function. A single model-kernel combination is considered here (RBF with Multiquadric function). A set of 200 paired training data set is randomly generated: X₁, X₂, X₃,...,X₂₀₀. The size of each training data set is defined to be: {X_{1t o40}} = 30, {X_{41t o80}} = 60, {X_{81t o120}} = 90, {X_{121 to 160}} = 120, {X_{161 to 200}} = 150. The distribution of samples is different (random) for sets with the same size. For each data set, COSMOS is applied to find the hyper-parameter value that minimizes the median error metric.

The median error of the selected model for the different data sets is illustrated in Fig. 21 as a series of boxplots, with each boxplot corresponding to sample sets of one given size. It is readily evident that error of the selected model is highly sensitive to the size of the training data sets. Although the influence of the training data distribution is also evident from the significant observed variance in the resulting model error, no particular trend is observed.

Fig. 21

Variation of the estimated error measures of the COSMOS-yielded models, with respect to training set size and variation in sample distribution (for Branin-Hoo Function)

Appendix D: Implementation of COSMOS on Hartmann and Perm functions

The final solutions, including the best trade-offs between the median and the maximum errors (in the Φ ₀, Φ ₁, and Φ ₂ classes) in Hartmann Perm functions are illustrated in Figs. 22 and 23. For the Hartmann test function, RBF with the Gaussian and Multiquadratic basis functions and Kriging with the Gaussian correlation function under the Φ ₁ class constitute the set of Pareto models in both Cascaded and One-Step techniques (Table 4). It is observed from Fig. 22 that in this problem, the Pareto solutions given by COSMOS for different Hyper-parameter classes have a larger spread than those given by the actual error. However, in terms of the best trade-off models, there is a fair agreement between the results of COSMOS and those determined from the actual errors (Table 4). Unlike the Branin-Hoo test function, in the Hartmann function, there is noticeable overlap between the final solutions from the different Hyper-parameter classes.

Fig. 22

Trade-offs between modal values of median and maximum error - Hartmann test function (3-variable): Pareto optimal models and final population of models from all Φ classes

Figure 23 and Table 4 show that for the Perm test function, at least one model-kernel combination from each of the three classes (Φ ₀, Φ ₁, and Φ ₂) contribute to the Pareto optimal set. In this test problem, Kriging with Linear correlation function and SVR with Sigmoid kernel function are selected as the best models with the lowest median error and the lowest maximum error, respectively. It can be seen from Table 4 that there is promising agreement between the model-kernel combinations chosen by COSMOS and those chosen based on the actual error. From the COSMOS and the actual error results, we also observe that a Pareto solution from the Φ ₀ class (RBF-Linear) is located at the elbow of the Pareto Frontier, which could be considered to represents a practically attractive, best trade-off model choice.

Fig. 23

Trade-offs between modal values of median and maximum error - Perm test function (10-variable): Pareto optimal models and final population of models from all Φ classes