Computer Experiments

Abstract

In contrast to experiments discussed in previous chapters which are carried out in the physical world, the experiments introduced in this chapter are carried out on a computer. Computer experiments can be run if the physical process of interest can be described by a mathematical model and if computer code can be written to compute the response from the mathematical model. A computer experiment is usually deterministic and will return the same response if observed more than once at the same input variable settings. Maximin Latin hypercube designs are introduced as commonly used designs for computer experiments. The chapter describes Gaussian stochastic process models for modeling the responses. The model parameters are estimated using the maximum likelihood technique. Prediction of output values and construction of prediction intervals are also discussed. The concepts introduced in this chapter are illustrated through a real experiment and the use of SAS and R software.

Exercises

• 1. Gaussian correlation function

  The Gaussian correlation function $R(\varvec{x}_i - \varvec{x}_j|\varvec{\xi })$ introduced in Sect. 20.3, p. 768, quantifies the correlation between outputs at two points $\varvec{x}_i$ and $\varvec{x}_j$ based on the distance between them. For parts (a) and (b) below, consider the case of $d = 1$ input variable.

  • (a) To investigate the effect of the value of parameter $\theta$ on the correlation between outputs at two points, calculate the twenty five correlations $R(x_i - x_j|\theta )$ for $\theta \in \{0.5, 2, 5, 20, 100\}$ and $|x_i - x_j| \in \{0.1, 0.2, 0.4, 0.6, 0.7\}$, where $|x_i - x_j|$ is the absolute value of the distance between $x_i$ and $x_j$.

  • (b) Construct a plot with $|x_i - x_j|$ on the x-axis and $R(x_i - x_j|\theta )$ on the y-axis, plot $R(x_i - x_j|\theta )$ for each value of $\theta$ on the same plot, and comment on the relationship between $\theta$ and $R(x_i - x_j|\theta )$.

• 2. Power exponential correlation function

  The Gaussian correlation function (20.3.3) is a special case of a Power exponential correlation function. The latter is given by

  $$\begin{aligned} R(\varvec{x}_i - \varvec{x}_j|\varvec{\xi })= \prod _{k=1}^d\exp (-\theta _k|x_{ik} - x_{jk}|^{p_k}) \end{aligned}$$

  where $\varvec{\xi }$ represents the parameters $(\theta _1, \ldots , \theta _d, p_1, \ldots , p_d)$, with all $\theta _k\ge 0$, and ${0<p_k\le 2}$.

  • (a) Suppose there is $d=1$ input variable. To investigate the effect of $\theta$ on the correlation between outputs at two points $x_i$ and $x_j$ for the Power exponential correlation function, calculate the nine correlations $R(x_i - x_j|\theta )$ for $\theta \in \{0.5, 5, 100\}$ and $|x_i - x_j| \in \{0.1, 0.4, 0.7\}$, for each value of $p\in \{0.5, 1, 1.5, 2\}$.

  • (b) Construct four plots, one for each value of p, with $|x_i - x_j|$ on the x-axis and $R(x_i - x_j|\theta )$ on the y-axis. Plot $R(x_i - x_j|\theta )$ for each value of $\theta$ on the same plot, and comment on the relationship between $\theta$ and $R(x_i - x_j|\theta )$ for each value of p.

• 3. Cubic correlation function

  The cubic correlation function is given by

  $$\begin{aligned} R(\varvec{x}_i-\varvec{x}_j|\varvec{\xi })=\prod _{k=1}^dR(x_{ik}-x_{jk}|\varvec{\xi }) \end{aligned}$$

  where

  $$\begin{aligned} R(x_{ik}-x_{jk}|\varvec{\xi }) = \left\{ \begin{array}{l l} 1-6\left( \frac{x_{ik}-x_{jk}}{\theta _k}\right) ^2+6\left( \frac{|x_{ik}-x_{jk}|}{\theta _k}\right) ^3 &{} \text{ if } |x_{ik}-x_{jk}|\le \frac{\theta _k}{2} \\ 2\left[ 1- \left( \frac{|x_{ik}-x_{jk}|}{\theta _k}\right) ^3\right] &{} \text{ if } \frac{\theta _k}{2}< |x_{ik}-x_{jk}| \le \theta _k \\ 0 &{} \text{ if } \theta _k < |x_{ik}-x_{jk}| \end{array} \right. , \end{aligned}$$

  where $\varvec{\xi }$ represents the parameters $(\theta _1, \theta _2 \ldots , \theta _d)$, and $\theta _k> 0$. Repeat Exercise 1 with the cubic correlation function and $d = 1$.

• 4. Bohman correlation function

  The Bohman correlation function is given by

  $$\begin{aligned} R(\varvec{x}_i-\varvec{x}_j|\varvec{\xi })=\prod _{k=1}^dR(x_{ik} - x_{jk}|\varvec{\xi }) \end{aligned}$$

  where $R(x_{ik} - x_{jk}|\varvec{\xi })$ is given by

  $$\begin{aligned} \left\{ \begin{array}{l l} 1-\frac{x_{ik} - x_{jk}}{\theta _k}\cos \left( \frac{\pi (x_{ik} - x_{jk})}{\theta _k}\right) +\frac{1}{\pi }\sin \left( \frac{\pi (x_{ik} - x_{jk})}{\theta _k}\right) &{} \text{ if } |x_{ik} - x_{jk}|< \theta _k \\ 0 &{} \text{ if } \theta _k \le |x_{ik} - x_{jk}| \end{array} \right. , \end{aligned}$$

  and $\varvec{\xi }$ represents the parameters $(\theta _1, \theta _2 \ldots , \theta _d)$, and $\theta _k> 0$. Repeat Exercise 1 with the Bohman correlation function and $d = 1$.

• 5. Euclidean interpoint distance

  For two points $\varvec{x}_i=(x_{i1}, x_{i2}, \ldots , x_{id})$ and $\varvec{x}_j=(x_{j1}, x_{j2}, \ldots , x_{jd})$ in a d-dimensional space, the Euclidean distance between $\varvec{x}_i$ and $\varvec{x}_j$ is defined by (20.4.8), p. 775. For the questions below assume that the range of each input variable is [0, 1].

  • (a) What is the largest possible distance between two points in (i) a 1-dimensional space? (ii) a 2-dimensional space? (iii) a d-dimensional space?

  • (b) What is the smallest possible distance between two points in a d-dimensional space? Why would it be undesirable to have two input points with this minimum distance between them in a computer experiment?

• 6. Maximin Latin hypercube designs

  Consider the three LHDs shown below. Note that the location of the points is not in the center of each cell but is randomly chosen.

  $$\begin{aligned} \varvec{X}_1 = \left[ \begin{array}{cc} 0.66 &{} 0.65 \\ 0.23 &{} 0.38 \\ 0.78 &{} 0.21 \\ 0.38 &{} 0.98 \end{array} \right] , \varvec{X}_2 = \left[ \begin{array}{cc} 0.50 &{} 0.68 \\ 0.24 &{} 0.39 \\ 0.89 &{} 0.19 \\ 0.35 &{} 0.92 \end{array} \right] , \text{ and } \varvec{X}_3 = \left[ \begin{array}{cc} 0.98 &{} 0.10 \\ 0.39 &{} 0.54 \\ 0.18 &{} 0.76 \\ 0.57 &{} 0.39 \end{array} \right] \end{aligned}$$

  We can think of maximin designs in the following way. Suppose we need to place n grocery stores (design points, corresponding to rows in $\varvec{X}$) in a particular county, where the county is the experimental region, taken as a rectangle determined by the ranges of the input variables $x_i$ and then scaled to $[0, 1]^2$. We would like to choose the store locations in a way that prevents any two stores from being close together. In other words, we are maximizing the minimum distance between the stores and are constructing a maximin design.

  • (a) For each design, calculate all $\left( {\begin{array}{c}4\\ 2\end{array}}\right)$ Euclidean interpoint distances (20.4.8), p. 775, i.e. calculate the distances between each pair of proposed store locations.

  • (b) For each design, identify the two closest points and their distance.

  • (c) Between $\varvec{X}_1$, $\varvec{X}_2$, and $\varvec{X}_3$ identify the design that maximizes the minimum interpoint distance; i.e. identify the maximin design.

• 7. Minimax designs

  Referring back to the intuitive explanation of the maximin designs in Exercise 6, consider now the distance between each customer and the location of the stores in a county. A reasonable placement of the stores could be such that no customer is too far from the closest store. In other words, we are minimizing the maximum distance of each customer to the closest store. When we are minimizing the maximum distance of any point in the experimental region (input space) from the closest design point, we are constructing a minimax design.

  Minimax designs are notoriously difficult to construct. When building a maximin design we have to calculate only the $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$ distances among the design points but, when constructing a minimax design, we have to consider infinitely many distances (since there are infinitely many points in the experimental region).

  Consider the three LHDs from Exercise 6 (i.e. the potential grocery store locations) and suppose that we are interested in the points in the experimental region (i.e. the customer locations) given by

  $$\begin{aligned} \varvec{C} = \left[ \begin{array}{cc} 0.59 &{} 0.10 \\ 0.89 &{} 0.55 \\ 0.14 &{} 0.35 \\ 0.38 &{} 0.90 \\ 0.72 &{} 0.63 \end{array} \right] . \end{aligned}$$

  • (a) For each of five points (customer locations) in $\varvec{C}$, determine the closest of the four design points in $\varvec{X}_1$ by calculating four appropriate Euclidean distances (20.4.8), p. 775. Identify a point (customer) in $\varvec{C}$ with the largest distance to its closest point in $\varvec{X}_1$.

  • (b) Repeat part (a) for $\varvec{X}_2$ and $\varvec{X}_3$.

  • (c) Identify which of the three designs is minimax for this scenario.

• 8. Minimum average reciprocal distance designs

  An alternative measure of space-fillingness is that of minimum average reciprocal distance. If design points are spaced out, then the distance between any pair of points will not be small, and so their reciprocal distance will not be large. Thus an alternative to constructing a maximin design is to construct a design with smallest possible average reciprocal distance between pairs of design points. The Euclidean distances between the points are calculated as in (20.4.8), p. 775, and the average of their reciprocals is the measure of goodness of the design.

  • (a) Of the three designs $\varvec{X}_1$, $\varvec{X}_2$, $\varvec{X}_3$ in Exercise 6, which has the minimum average reciprocal distance? Does this coincide with the maximin design?

  • (b) Of the three designs $\varvec{X}_1$, $\varvec{X}_2$, $\varvec{X}_3$ in Example 20.4.2, p. 775, which has the minimum average reciprocal distance? Does this coincide with the maximin design?

• 9. Prediction

  Suppose that a computer simulator with one input variable was run at 3 input points and the GaSP model with Gaussian correlation function (20.3.3) was fitted to the data:

  $$\begin{aligned} \begin{array}{cc} { x} &{} { y} \\ \hline 0.20 &{} -0.3635 \\ 0.50 &{} -0.1353 \\ 0.80 &{} -0.0330 \end{array} \end{aligned}$$

  leading to parameter estimates $\hat{\beta }_0 = -0.2104$, $\hat{\sigma }^2 = 0.0264$, and $\hat{\theta } = 4.9003$. Based on this information, we would like to predict Y at $x_a = 0.20$ using the predictor in (20.3.4), p. 770.

  • (a) Calculate $\hat{\varvec{R}}$ (defined below (20.3.4)).

  • (b) Calculate $\hat{\varvec{r}}$ (defined below (20.3.4)).

  • (c) Calculate $\hat{Y}(x_a)$ in (20.3.4) at $x_a = 0.20$. Is this the answer you expected? Explain.

  • (d) Calculate the estimated variance $\hat{s}^2(x_a)$ (20.3.5) of your prediction. Is this the answer you expected? Explain.

  • (e) Repeat parts (b), (c), and (d) for $x_a = 0.49$ and $x_a = 0.65$.

• 10. Tool coating experiment

  D. Draguljić, S. Nekkanty , T. J. Santner , A. M. Dean , and R. Shivpuri, in 2015, described a computer experiment used to develop multilayer coatings which are used to protect tools, drills, cutting blades, bearings, etc. The experiment consisted of modeling the effect of the number of coating layers and the thicknesses of those layers on the normalized measures of maximum normal radial stress, $Y_1$, and the maximum shear stress, $Y_2$. The two responses were modeled independently. Large values of either stress would lead to coating failures (either peeling of of the coating or occurrence of cracks in the coating). Here we will focus on coatings with only two layers. Therefore we have input variables $x_1$ and $x_2$ (both in $\mu$m), the thicknesses of the top and the bottom layer, respectively. The data for this experiment are given in Table 20.8. Note that $x_1$ and $x_2$, as shown in Table 20.8, are scaled from their original (0, 6) $\mu$m scale to (0, 1) $\mu$m scale.

  Table 20.8 Data for the tool coating experiment

  • (a) Estimate $\theta _1$ and $\theta _2$ from the GaSP model that relates $x_1$ and $x_2$ to $y_1$ using the data from Table 20.8.

  • (b) For this experiment the total thickness of the coating was required to be between 1/3 and 1, i.e. $1/3\le x_{1} + x_2\le 1$ while the thickness of any individual layer needed to be a multiple of 1/24. Construct a grid with mesh size of 1/24 that satisfies this constraint and predict the values of $Y_1$ for this grid.

  • (c) Based on your predictions, what pair of coating thicknesses ($x_1, x_2$) seems to minimize $Y_1$?

  • (d) Repeat parts (a)–(c) for $Y_2$.

  • (e) Based on your predictions for $Y_1$ and $Y_2$, what single pair of coating thicknesses ($x_1, x_2$) would you suggest to try to minimize both $Y_1$ and $Y_2$ simultaneously (as well as you can)? It may help to make a plot of the predicted values of $Y_1$ and $Y_2$.

• 11. Borehole function

  A borehole is a narrow tunnel drilled in the ground. Boreholes serve numerous purposes, including extraction of water or gases, mineral exploration, temperature measurement, etc. The borehole function (see Surjanovic and Bingham 2013) models water flow through a borehole and is given by

  $$\begin{aligned} y = \frac{2\pi T_u(H_u - H_l)}{a\left( 1+ + \frac{2LT_u}{ar^2_w K_w} + \frac{T_u}{T_l}\right) } \end{aligned}$$

  (20.7.9)

  where $a = \ln (r/r_w)$. The output y measures the water flow rate in m$^3/$year. There are eight inputs to the borehole function. Their names and ranges are given in Table 20.9.

  • (a) Construct an $80\times 8$ maximin LHD, $\varvec{X}$, where each element of $\varvec{X}$ is in [0, 1]. Let the elements $x_{i1}$ in the first column of $\varvec{X}$ represent the scaled values of the first variable $r_w$ for which we will “observe” (calculate) y from the simulator, the elements $x_{i2}$ in the second column of $\varvec{X}$ represent the scaled values of the second variable r, and so on for all 8 columns.

  • (b) To be able to calculate the simulator data $y(\varvec{x}_i)$ using (20.7.9), where $\varvec{x}_i$ is the $i^{th}$ row (input combination) of $\varvec{X}$, we need to transform the value of each input variable in $\varvec{X}$ (which has range [0, 1]) to the variables and matching ranges given in Table 20.9. This will allow the appropriate values to be entered into (20.7.9). For transforming $x_{1}$ to $r_w$ with range in Table 20.9, the scaling is done via

    $$ r_{w}^{[0.05, 0.15]} = (0.15 - 0.05)x_{1}^{[0, 1]} + 0.05\,. $$

    The superscripts represent the ranges for the input variable $x_1$ and transformed variable $r_w$. The scaling for other variables is done in a similar fashion. Scale each value in each row $\varvec{x}_i$ of $\varvec{X}$ to the appropriate range. Calculate the output ${y}(\varvec{x}_i), i = 1, 2, \ldots , 80$ using (20.7.9).

  • (c) One of the simple ways to assess how sensitive Y is to the changes in a particular input $x_k$ is to plot y versus the $x_{ik}$’s and examine the plot for any obvious patterns. Construct eight scatterplots, one for each input variable $x_1$–$x_8$, and identify the variables that seem to influence the response the most.

  • (d) Using $\varvec{X}$ and ${y}(\varvec{x}_i)$, fit the GaSP model. What are the maximum likelihood estimates of the parameters? Do the values of $\hat{\theta }_k$, $k = 1, 2, \ldots , 8$, support your conclusion from part (c)?

  • (e) Construct a grid of mesh size roughly equal to 0.5 (i.e. $x_{ik}\in \{0, 0.5, 1\}$) and predict the outputs ${y}(\varvec{x}_i)$ over the grid. Based on your predictions, what are the values of $x_1, \ldots , x_8$ that result in the predicted maximal water flow? How about the minimal water flow? Using the ranges in Table 20.9, again, scale these values to show the values of the original variables that result in predicted maximal and minimal water flow.

Table 20.9 Input variables for the borehole function