Comparative study of HDMRs and other popular metamodeling techniques for high dimensional problems

Abstract

The efficiency of optimization for the high dimensional problem has been improved by the metamodeling techniques in multidisciplinary in the past decades. In this study, comparative studies are implemented for high dimensional problems on the accuracy of four popular metamodeling methods, Kriging (KRG), radial basis function (RBF), least square support vector regression (LSSVR) and cut-high dimensional model representation (cut-HDMR) methods. Besides, HDMR methods with different basis functions are considered, including KRG-HDMR, RBF-HDMR and SVR-HDMR. Four factors that might influence the quality of metamodeling methods involving parameter interaction of problems, sample sizes, noise level and sampling strategies are considered. The results show that the LSSVR with Gaussian kernel, using Latin hypercube sampling (LHS) strategy, constructs more accurate metamodels than the KRG. The RBF with Gaussian basis function performs poor in the group. Generally, cut-HDMR methods perform much better than the other metamodeling methods when handling the function with weak parameter interaction, but not better when handling the function with strong parameter interaction.

Appendices

Appendix 1: Test functions

Appendix 2: Validation by cross validation

It is often prohibitive to generate large quantities of testing sample points for practical simulation-based engineering problems. Thus, researchers often employ leave-one-out cross validation (LOOCV) since it only takes training sample points to estimate the metamodels’ accuracy. Accordingly, the generalized mean square error (GMSE) is considered as the accuracy criterion. The time that LOOCV takes is proportional to the number of sample points and the efficiency of metamodeling method. When given a large number of sample points, a low-efficiency metamodeling method definitely requires much time to estimate the metamodel accuracy. However, considering the effort to obtain a sample point using time-demanding FE or CFD analysis, the GMSE criterion is relatively time-saving in contrast with the RMSE measure. There is a necessity to check whether the RMSE measure can be reflected by the GMSE measure. Figure 14 depicts the average values of NRMSE and normalized GMSE (NGMSE) for 12 test functions after 50 replicas. Given large sample sets, three offline metamodeling methods with spline basis function and kernel functions are used. For each function, it is noticeable that the discrepancy of two accuracy criteria derived from the same metamodeling method is negligible. It indicates that the LOOCV can also be utilized to verify the performances of metamodeling methods.

Fig. 14

Average values of NRMSE and normalized GMSE for functions after 50 replicas using KRG-Sp, RBF-Sp and LSSVR-Sp with LHS strategy generating large sample sets

Appendix 3: Variability in performance of metamodels with multiple sampling strategies

This section aims to identify the variability in performance of metamodeling methods in view of multiple samples generated by the same sampling strategy (i.e., LHS and HSS). Relative error of accuracy criteria, defined as (3.1 and 3.2), is introduced to represent the robustness of metamodels. The smaller the relative error is, the more robust the metamodel is:

$$ {\mathrm{NRMSE}}_{\mathrm{Relative}\kern0.4em \mathrm{error}}=\left({\mathrm{NRMSE}}_{\mathrm{max}}\hbox{-} {\mathrm{NRMSE}}_{\mathrm{min}}\right)/{\mathrm{NRMSE}}_{\mathrm{mean}} $$

(3.1)

$$ {\mathrm{NMAX}}_{\mathrm{Relative}\kern0.4em \mathrm{error}}=\left({\mathrm{NMAX}}_{\mathrm{max}}\hbox{-} {\mathrm{NMAX}}_{\mathrm{min}}\right)/{\mathrm{NMAX}}_{\mathrm{mean}} $$

(3.2)

Figures 15 and 16 respectively show the relative error of NRMSE and NMAX obtained from 50 replicas with LHS and HSS strategies generating small sample sets. Using the LHS strategy, it can be observed from Fig. 15a for NRMSE that RBF-Ga seems the most robust metamodeling method, followed by LSSVR-Sp and KRG. RBF-Sp and LSSVR-Ga are not robust enough due to their large relative errors. However, RBF-Ga performs much worse for each function and each NRMSE values estimated approach to 1, which makes its relative error extremely small. Thus, LSSVR-Sp is the most robust one, and it can be proved by Figs.15b where LSSVR-Sp possesses the least values of the relative error of NMAX for most of functions. When referring to HSS strategy, LSSVR-Sp remains to be the best alternative, and the overall consequence is similar with that when using LHS strategy, as presented in Fig. 16.

Fig. 15

Relative error of accuracy obtained from 50 replicas with LHS strategy generating small sample sets. a NRMSE. b NMAX

Fig. 16

Relative error of accuracy obtained from 50 replicas HSS strategy generating small sample sets. a NRMSE. b NMAX

Similarly, Figs. 17 and 18 respectively show the relative error of two measures with LHS and HSS strategies generating large sample sets. It is observable that LSSVR-Sp remains to be the best and has higher robustness compared to Figs. 15 and 16 when sample points increase.

Fig. 17

Relative error of accuracy obtained from 50 replicas with LHS strategy generating large sample sets. a NRMSE. b NMAX

Fig. 18

Relative error of accuracy obtained from 50 replicas with HSS strategy generating large sample sets. a NRMSE. b NMAX

Appendix 4: Counting of the parameter interaction

The number of parameter interactions should be measured via series representation of the form (16) rather than the previous definition. Therefore, we choose the analysis of variance (ANOVA) to determine the number of parameter interaction of a test function. The implementation of ANOVA is derived from Ref. (Toshimitsu and Andrea 1996). The importance of parameter x_i and parameter interactions among ${x}_{i_1},{x}_{i_2}$ and ${x}_{i_s}$ to the output f(x) can be quantitively measured by the Sobol sensitivity index S_i and ${S}_{i_1{i}_2\cdots {i}_s}$, respectively. One can argue that parameter interaction among parameters ${x}_{i_1}$, ${x}_{i_2}$,…, ${x}_{i_s}$ is insignificant if ${S}_{i_1{i}_2\cdots {i}_s}$ equals or approaches zero.

Given an integrable function f(x) defined in Iⁿ, the ANOVA representation of this function can be expressed as follow:

$$ f\left(\mathbf{x}\right)={f}_0+\sum \limits_{i=1}^n{f}_i\left({x}_i\right)+\sum \limits_{1\le i<j\le n}{f}_{ij}\left({x}_i,{x}_j\right)+\cdots +{f}_{12\cdots n}\left({x}_1,{x}_2,\cdots, {x}_n\right) $$

(4.1)

Assuming the sample points are independent among parameters, the estimation of ${S}_{i_1{i}_2\cdots {i}_s}$ is defined as follows:

$$ {S}_{i_1{i}_2\cdots {i}_s}=\frac{D_{i_1{i}_2\cdots {i}_s}}{D} $$

(4.2)

where ${D}_{i_1{i}_2\cdots {i}_s}$ and D respectively indicate the variance of ${f}_{i_1{i}_2\cdots {i}_s}$ and f(x):

$$ {D}_{i_1{i}_2\cdots {i}_s}={\int}_{I^s}{f}_{i_1{i}_2\cdots {i}_s}^2{dx}_{i_1}{dx}_{i_2}\cdots {dx}_{i_s} $$

(4.3)

$$ D={\int}_{I^n}{f}^2\left(\mathbf{x}\right)d\mathbf{x}-{\widehat{f}}_0^2 $$

(4.4)

where ${\widehat{f}}_0$ is the mean of outputs of sample points.

All ${S}_{i_1{i}_2\cdots {i}_s}$ are nonnegative and the sum is

$$ \sum \limits_{s=1}^n\sum \limits_{i_1<\cdots <{i}_s}^n{S}_{i_1{i}_2\cdots {i}_s}=1 $$

(4.5)

Using the MC method, the estimation of ${D}_{i_1{i}_2\cdots {i}_s}$ is approximated as follows:

$$ {D}_{i_1{i}_2\cdots {i}_s}\approx {\widehat{D}}_{i_1{i}_2\cdots {i}_s}-\sum {\widehat{D}}_{i_1{i}_2\cdots {i}_{s-1}}+\sum {\widehat{D}}_{i_1{i}_2\cdots {i}_{s-2}}-\cdots +{\left(-1\right)}^s{\widehat{f}}_0^2 $$

(4.6)

In the above equation, ${\widehat{D}}_{i_1{i}_2\cdots {i}_s}$ is generated by summing products of two function values: u with all the variables sampled and the other one v with all the variables re-sampled except variables ${x}_{i_1}$, ${x}_{i_2}$,…, ${x}_{i_s}$. The following equation is the mathematical expression:

$$ {\widehat{D}}_{i_1{i}_2\cdots {i}_s}=\frac{1}{N}\sum \limits_{m=1}^Nf\left({\mathbf{u}}_m\right)f\left({\mathbf{v}}_m,{\mathbf{x}}_m^{\mathbf{u}}\right) $$

(4.7)

where $\left({v}_m,{x}_m^u\right)$ indicates the matrix v has the same column vector with the matrix u corresponding to variables ${x}_{i_1}$, ${x}_{i_2}$,…, ${x}_{i_s}$.

In this study, we divide test functions into two groups: low parameter interaction and high parameter interaction. The former kind of test functions is represented by the definition that parameter interactions of test functions are not larger than 2, with $0.999\le \sum \limits_i{S}_i+\sum \limits_{i<j}{S}_{ij}\le 1$. The latter ones indicate test functions have 3 or higher parameter interactions, with $\sum \limits_i{S}_i+\sum \limits_{i<j}{S}_{ij}<0.999$. Therefore, a simple procedure is implemented to determine the parameter interaction of a test function.

To begin with, calculate the first order GSI S_i based on (4.2). If $\sum \limits_i{S}_i\ge 0.999$, parameter interaction of the function is 1; Otherwise, continue calculating the second order GSI S_ij. If $\sum \limits_i{S}_i+\sum \limits_{i<j}{S}_{ij}\ge 0.999$, parameter interaction is 2; Otherwise, parameter interaction is 3 or higher.

The parameter interactions of twelve test functions are summarized in Table 9.