Modeling Techniques

Abstract

Let $y=f(\varvec{x})=f(x_1,\ldots ,x_s)$ be the true model of a system where $X_1,\ldots ,X_s$ are factors that take values $(x_1,\ldots ,x_s)$ on a domain $\mathcal {X}$, and y is response. For physical experiments with model unknown, the true model $f(\cdot )$ is unknown and for computer experiments the function $f(\cdot )$ is known, but it may have no an analytic formula. We are requested to find a metamodel $\hat{y}=g(\varvec{x})$ with a high quality to approximate the true model in a certain sense. How to find a metamodel is a challenge problem. This chapter gives a brief introduction to various modeling techniques such as radial basis function, polynomial regression model, spline and Fourier model, wavelets basis and Kriging models in applications by the use of uniform designs. Readers can find more discussion on modeling methods in Eubank (1988), Wahba (1990), Hastie et al. (2001), and Fang et al. (2006).

Exercises

5.1

Suppose the response y and factor x have the following underlying relationship

$$\begin{aligned} y=f(x)+\varvec{e}=1-e^{-x^2}+\varepsilon ,~~\varepsilon \sim N(0,0.1^2), x\in [0, 3], \end{aligned}$$

but the experimenter does not know this model and he/she wants to find an approximation model to the real one by experiments. Therefore, he/she considers four designs as follows:

$$\begin{aligned} D_3= & {} \{0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2,2\},\\ D_4= & {} \left\{ \frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{3}{4}, \frac{3}{4}, \frac{3}{4}, \frac{5}{4}, \frac{5}{4}, \frac{5}{4}, \frac{7}{4}, \frac{7}{4}, \frac{7}{4},\right\} ,\\ D_6= & {} \left\{ \frac{1}{6}, \frac{1}{6}, \frac{3}{6}, \frac{3}{6}, \frac{5}{6}, \frac{5}{6}, \frac{7}{6},\frac{7}{6},\frac{9}{6},\frac{9}{6},\frac{11}{6},\frac{11}{6}\right\} ,\\ D_{12}= & {} \left\{ \frac{1}{12}, \frac{3}{12}, \frac{5}{12}, \frac{7}{12}, \frac{9}{12}, \frac{11}{12}, \frac{13}{12},\frac{15}{12},\frac{17}{12},\frac{19}{12},\frac{21}{12},\frac{23}{12}\right\} . \end{aligned}$$

Implement the following steps:

• 1. Plot the function

  $$ y=f(x)=1-e^{-x^2},~~ x\in [0,3]. $$

• 2. Generate a data set for each design by the statistical simulation.

• 3. Find a suitable regression model and related ANOVA table for each data set. Plot the fitting models.

• 4. Randomly generate $N=1000$ points $x_1, \ldots , x_{1000}$ and calculate the mean square error (MSE) defined by

  $$ \text{ MSE }=\frac{1}{N}\sum _{i=1}^N (y_i-\hat{y}_i)^2 $$

  for each model, where $\hat{y}_i$ is the estimated value of $y_i$ under the model.

• 5. According to the plots, MSE and $SS_E$, give your conclusions based on your comparisons among the above models.

5.2

For comparing different kinds of designs and modeling techniques in computer experiments, there is a popular way to consider several case studies. The models are known. Choose several designs like the orthogonal design (OD), Latin hypercube sampling (LHS), uniform design (UD), and modeling techniques. Then compare all design-modeling combinations.

Suppose that the following models are given. Please consider three kinds of designs OD, UD, and LHS (with $n=16,25,29,64$) and modeling techniques: the quadratic regression models, a power spline basis with the following general form of $1,x,x^2,\ldots , x^p, (x-\kappa _1)_+^p,\ldots , (x-\kappa _m)_+^p$ in (5.1.8), a Kriging model $y(\varvec{x})= \sum _{i=1}^m \beta _ih_i(\varvec{x}) +z(\varvec{x})$ defined in Definition 5.2.2 and artificial neural network. Give your comparisons for all possible design-modeling combinations.

Model 1:

$$\begin{aligned} Y=\frac{{\ln }(x_{1})\times (\sin (x_{2})+4)}{e^{x_{3}}}+{\ln }(x_{1})e^{x_{3}} \end{aligned}$$

(5.3.5)

where the ranges of the independent variables are $x_{1}:[0.1, 10],$ $x_{2}:[-\pi /2, \pi /2],$ and $x_{3}:[0, 1]$, respectively.

Model 2:

$$\begin{aligned} Y= & {} -\left[ 2\exp \left\{ -\frac{1}{2}(x_{1}^{2}+(x_{2}-4))^{2}\right\} \right. \nonumber \\&\left. +\exp \left\{ -\frac{1}{2}((x_{1}-4)^{2}+\frac{x_{2}^{2}}{4}\right\} +\exp \left\{ -\frac{1}{2}\left( \frac{(x_{1}+4)^2}{4}+x_{2}^{2}\right) \right\} \right] \end{aligned}$$

(5.3.6)

where the ranges of the independent variables are $x_{1}:[-10, 7],$ $x_{2}:[-6, 7],$ respectively.

Model 3:

$$\begin{aligned} Y= & {} 10(x_{2}-x_{1}^{2})^{2}+(1-x_{1})^{2}+9(x_{4}-x_{3}^{2})+(1-x_{3})^{2}\nonumber \\&+1.01[(x_{2}-1)^{2}+(x_{4}-1)^{2}] +1.98(x_{2}-1)(x_{4}-1)^{2} \end{aligned}$$

(5.3.7)

where the ranges of the independent variables are $x_{i}:[-2, 2],i=1,2,3,4$.

Model 4:

$$\begin{aligned} Y=\sum _{k=1}^{4}[100(x_{k+1}-x_{k}^{2})^{2}+(1-x_{k})^{2}], \end{aligned}$$

(5.3.8)

where the ranges of the independent variables are $x_{i}:[-2, 2],i=1,\ldots ,5$.

5.3

Example 1.1.7 is a good platform for comparing various design-modeling combinations. Let y be the distance from the end of the arm to the origin expressed as a function of 2m variables $\theta _j\in [0,2\pi ]$ and $L_j\in [0,1]$, where $y=\sqrt{u^2+v^2}$ and (u, v) are defined in (1.1.3). Consider three kinds of designs OD, UD, and LHS and three kinds of metamodel: polynomial regression model, Kriging model, and empirical Kriging model. Give your comparisons for possible design-modeling combinations with $m=2$ and $m=3$, respectively.