Evaluating the Effectiveness of DART^® Buoy Networks Based on Forecast Accuracy

Abstract

A performance measure for a DART^® tsunami buoy network has been developed. DART^® buoys are used to detect tsunamis, but the full potential of the data they collect is realized through accurate forecasts of inundations caused by the tsunamis. The performance measure assesses how well the network achieves its full potential through a statistical analysis of simulated forecasts of wave amplitudes outside an impact site and a consideration of how much the forecasts are degraded in accuracy when one or more buoys are inoperative. The analysis uses simulated tsunami amplitude time series collected at each buoy from selected source segments in the Short-term Inundation Forecast for Tsunamis database and involves a set for 1000 forecasts for each buoy/segment pair at sites just offshore of selected impact communities. Random error-producing scatter in the time series is induced by uncertainties in the source location, addition of real oceanic noise, and imperfect tidal removal. Comparison with an error-free standard leads to root-mean-square errors (RMSEs) for DART^® buoys located near a subduction zone. The RMSEs indicate which buoy provides the best forecast (lowest RMSE) for sections of the zone, under a warning-time constraint for the forecasts of 3 h. The analysis also shows how the forecasts are degraded (larger minimum RMSE among the remaining buoys) when one or more buoys become inoperative. The RMSEs provide a way to assess array augmentation or redesign such as moving buoys to more optimal locations. Examples are shown for buoys off the Aleutian Islands and off the West Coast of South America for impact sites at Hilo HI and along the US West Coast (Crescent City CA and Port San Luis CA, USA). A simple measure (coded green, yellow or red) of the current status of the network’s ability to deliver accurate forecasts is proposed to flag the urgency of buoy repair.

7. Appendix: Details on Simulations and Evaluation of Forecasts

Here we give details about how we have chosen to simulate and evaluate open-ocean wave amplitude forecasts. This material is summarized in Sect. 3, where we noted four tasks to be carried out, which are described in Sects. 7.1, 7.2, 7.3, 7.4. We illustrate the four tasks by focusing on a representative triad consisting of one predesignated central unit source (ac026b), one DART^® buoy (46403) and one impact site (Hilo HI).

1.1 7.1. Simulation of Tsunami Signal and Open-Ocean Wave Amplitudes

Our first task is to simulate a tsunami event, which manifests itself as a tsunami signal that passes by a particular DART^® buoy, say, 46403 (Fig. 1 shows the location of this buoy). We denote this signal by ${\varvec{t}}$. A simple way to specify ${\varvec{t}}$ would be to use one of the precomputed models predicting what 46403 should see if the tsunami event were to originate from within a particular predesignated unit source, say, ac026b (shown in Fig. 1). We would thus just set ${\varvec{t}}$ equal to ${\varvec{g}}_5$ depicted in Fig. 2; however, we have chosen not to adopt this simple approach because an important factor impacting open-ocean wave amplitude forecasts is modeling of the tsunami signal ${\varvec{t}}$ (this is the subject of Sect. 7.3). Because we cannot model the signal perfectly in an actual tsunami event, there is a danger of generating unrealistic simulated forecasts if we were to assume the signal to be identical to its model. One of many sources of mismatch is uncertainty in the location of the source within the subduction zone. Figure 11 illustrates the effect of relocating unit source ac026b, producing a relocated 100 km by 50 km source that overlaps (in this example) with three additional predesignated sources. To create the relocated source, we picked a random point (the red diamond) within ac026b and used this point as the center for the relocated unit source (blue rectangle). We can now compute a model predicting what would be observed at buoy 46403 if an earthquake were to originate from the randomly relocated unit source. This new model will differ from the precomputed models shown in Fig. 2. If we take this new model to be our tsunami signal ${\varvec{t}}$ and if we were then to entertain modeling ${\varvec{t}}$ using a subset of the models shown in Fig. 2, then the constructed signal need not be exactly equal to its model. Randomly choosing a set of locations within a central unit source (ac026b in this example) then leads to an ensemble of relocated sources that reflects the uncertainty in source location.

Because it is time consuming to generate geophysical models, we make an assumption of linearity and construct the tsunami signal ${\varvec{t}}$ using a linear combination of precomputed models. For the example shown in Fig. 11, the signal would be a linear combination of the models for unit sources ac025a, ac026a, ac025b and ac026b (the central unit source). These models are labeled in Fig. 2 as ${\varvec{g}}_1$, ${\varvec{g}}_2$, ${\varvec{g}}_4$ and ${\varvec{g}}_5$. The weights $w_k$ in the linear combination are based on geometry and are dictated by the degree of intersection of the relocated source with the predesignated sources—in this example, ${\varvec{g}}_5$ (corresponding to ac026b) gets the most weight because it intersects the most, while ${\varvec{g}}_1$ (corresponding to ac025a) gets the least nonzero weight. Figure 12 illustrates the construction of the N-dimensional column vector ${\varvec{t}}$ in terms of the six vectors ${\varvec{g}}_k$ shown in Fig. 2:

$$\begin{aligned} {\varvec{t}} = \sum _{k=1}^6 w_k {\varvec{g}}_k \end{aligned}$$

(1)

(black curve, lower middle plot). Note that, since neither ${\varvec{g}}_3$ nor ${\varvec{g}}_6$ are involved in constructing ${\varvec{t}}$, the weights $w_3$ and $w_6$ are zero. Past experience with actual tsunami events suggests that the weights should not be arbitrarily set if we want to regard the simulated event as originating from a 100 km by 50 km region ((Hanks and Kanamori, 1979; Papazachos et al., 2004)). We use the normalization $\sum _k w_k = 4$ in all of our simulations because a larger setting than this would result in a tsunami signal whose magnitude would be more realistically associated with multiple unit sources. For the example shown in Fig. 11, we have $w_1 \doteq 0.28$, $w_2 \doteq 1.04$, $w_4 \doteq 0.57$ and $w_5 \doteq 2.11$.

We can construct additional tsunami signals by selecting other points at random within the rectangle for the central unit source ac026b. Figure 13 breaks the rectangle for ac026b up into four quadrants of equal size (labeled as I, II, III and IV). The specific models ${\varvec{g}}_k$ that are used to construct the tsunami signal depend upon which quadrant the randomly selected point falls in—these are listed in Fig. 13 for the four quadrants. The random pick in Fig. 11 falls in quadrant II, and hence, as previously noted, the signal is constructed using ${\varvec{g}}_1$, ${\varvec{g}}_2$, ${\varvec{g}}_4$ and ${\varvec{g}}_5$. No matter into which quadrant the random pick falls, the model ${\varvec{g}}_5$ is always included with a weight $w_5$ satisfying $1 \le w_5 \le 4$, and this weight is close to 4 when the random pick is close to the center of the rectangle. The upper two quadrants involve three models in addition to ${\varvec{g}}_5$, whereas the lower quadrants involve just one (either ${\varvec{g}}_4$ or ${\varvec{g}}_6$). The characteristics of the constructed tsunami signals will thus depend upon the randomly selected quadrant and where the randomly selected point occurs within the quadrant. No matter where the random pick ends up, we can write the resulting constructed tsunami signal as per Eq. (1), with the understanding that either two or four of the $w_k$s will be zero.

We can also construct tsunami signals by focusing on a unit source other than ac026b; however, for simplicity, we will use as a central unit source only those having a setup similar to ac026b, namely, that they are in row b and are surrounded by five other unit sources. This restriction for central unit sources simplifies computer code somewhat and arguably should not adversely impact the evaluation of the effectiveness of a buoy network (a hypothesis that is yet to be investigated).

Given a relocated unit source, there are two methods for generating the corresponding open-ocean wave amplitudes ${\varvec{h}}$ outside of an impact site of interest. The more accurate method is via high-resolution model runs, which are time consuming; the less accurate makes use of lower-resolution runs already available in a precomputed propagation data base, which has the advantage of being easy to extract. We tested both methods and found that results based on the propagation data base differed little from those based on high-resolution model runs. We have thus elected to use the propagation data base approach, for which we take the presumed wave amplitudes to be

$$\begin{aligned} {\varvec{h}} = \sum _{k=1}^6 w_k {\varvec{h}}_k, \end{aligned}$$

(2)

where the weights $w_k$ are identical to those in Eq. 1, while ${\varvec{h}}_k$ is the forecast of the wave amplitudes in the open ocean that would occur from an event occurring in the same predesignated unit source associated with ${\varvec{g}}_k$ (note, however, that, while ${\varvec{h}}_k$ depends upon the location of the unit source and the location in the open ocean, it does not depend upon any of the buoy locations).

As an example, Fig. 14 shows wave amplitudes ${\varvec{h}}_k$ at an open-ocean location outside of Hilo associated with the same six predesignated unit sources considered in Figs. 1 and 2. The left-hand plot of Fig. 15 shows ${\varvec{h}}$ formed using the weights stated in the caption to Fig. 12 (the right-hand plot is explained in Sect. 7.3).

1.2 7.2. Simulation of 1-Min Stream

Our second task is to simulate data as it would be recorded at a DART^® buoy. Due both to the complexity of recorded tsunami data and the way in which these data are used to create wave amplitude forecasts, simulations of observations and forecasts that are fully realistic are difficult and time consuming to create. The approach we take is to impose simplifications that nonetheless retain the salient factors limiting the ability of a buoy network to contribute to accurate forecasts. As a starting point, we assume the 1-min stream recorded at a particular buoy is given by

$$\begin{aligned} \bar{\varvec{y}} = {\varvec{x}} + {\varvec{t}} + {\varvec{\epsilon }}, \end{aligned}$$

(3)

where $\bar{\varvec{y}}$ is a vector containing N consecutive values from the 1-min stream; ${\varvec{x}}$ represents tidal fluctuations; $\varvec{t}$ is the tsunami signal constructed as per Eq. (1); and $\varvec{\epsilon }$ is background noise. Tidal fluctuations and background noise in the DART^® data are two factors that can adversely impact inundation forecasts, so it is important to handle these realistically. Rather than simulating these factors, we can use archived DART^® data that were recorded under ambient conditions and retrieved during routine servicing of the buoy. The tsunami signal in Eq. (3) is missing during ambient conditions, so random samples from historical data can serve to generate $\varvec{x} + \varvec{\epsilon }$ (for details about the sampling procedure, see Sect. 3, (Percival et al., 2015)). A complication is that not all currently deployed buoys have associated archived data (and proposed buoys certainly don’t). To handle such buoys, we use data from a surrogate buoy with good matching oceanographic conditions.

Figure 16 shows a simulated 1-min stream $\bar{\varvec{y}}$ constructed as per Eqs. (1) and (3). The top plot shows the tsunami signal $\varvec{t}$ (this is the same as the black curve in middle bottom plot of Fig. 12). The middle plot shows the sum of tidal fluctuations $\varvec{x}$ and background noise $\varvec{\epsilon }$, which were obtained from archived data for DART^® buoy 46403. The bottom plot shows $\bar{\varvec{y}}$, which is the sum of the time series in the top and middle plots.

1.3 7.3. Estimation of Source Coefficients and Forecasting of Open-Ocean Wave Amplitudes

With simulated 1-min stream $\bar{\varvec{y}}$ in hand, the third task is to estimate the source coefficients $\alpha _k$. In practice factors limiting the accurate estimation of $\alpha _k$ include tidal fluctuations, background noise, an insufficient amount of data and imperfect knowledge about the underlying tsunami signal. We assume a linear regression model of the form

$$\begin{aligned} \bar{\varvec{y}} = \mu {\varvec{1}} + \beta _1 {\varvec{c}} + \beta _2 {\varvec{s}} + \sum _{k\in \mathcal{K}} \alpha _k {\varvec{g}}_k + {\varvec{e}}, \end{aligned}$$

(4)

where $\varvec{1}$ is an N-dimensional vector of ones associated with the regression coefficient $\mu$; ${\varvec{c}}$ is a vector with elements $\cos \,(\omega n\,\Delta )$, $n=0,1, \ldots , N-1$, and is associated with $\beta _1$—here $\omega$ is the tidal frequency M2 and $\Delta = 1$ min; ${\varvec{s}}$ is associated with $\beta _2$ and is similar to ${\varvec{c}}$ but its elements are given by $\sin \,(\omega n\,\Delta )$; $\mathcal K$ is a subset of $\{1, 2, \ldots , 6 \}$ that specifies the ${\varvec{g}}_k$ to be used to model the tsunami signal; and ${\varvec{e}}$ is a vector of stochastic errors assumed to have zero mean (note that this vector is not taken to be the same as ${\varvec{\epsilon }}$ in Eq. (3)). As discussed in Percival et al. (2015), the coefficients $\mu$, $\beta _1$ and $\beta _2$ and their associated vectors serve to model the tidal fluctuations in Eq. (3) in a simple—but statistically efficient—manner that is superior to other methods for handling the tides including filtering. Modeling of the signal $\varvec{t}$ is handled through specification of $\mathcal K$. We consider four protocols for setting $\mathcal K$, which are of interest because they make different assumptions about what is known about the random pick and because they lead to different degrees of mismatch between the constructed signal and its model.

• (a) The single unit source protocol (‘1 protocol’ for brevity) sets $\mathcal{K}$ to be $\{5 \}$; i.e., the model for the signal is $\alpha _5 {\varvec{g}}_5$. With probability one, this protocol yields an incorrect signal model in the sense that the constructed signal consists of ${\varvec{g}}_5$ combined with either one or three additional ${\varvec{g}}_k$s, whereas our model for it involves just ${\varvec{g}}_5$. This protocol in effect assumes we only know within which predesignated unit source the earthquake occurred (i.e., we know nothing about the random pick).

• (b) The two unit sources protocol (‘2 protocol’) assumes we have limited information about the random pick, namely, whether the earthquake originates from the left-hand or right-hand side of the rectangle (the random pick in Fig. 11 is left-handed). For a left-hand pick, we set $\mathcal{K}$ to $\{4, 5 \}$, and we set it to $\{5, 6 \}$ for a right-hand pick. For both settings, there are two ${\varvec{g}}_k$s in the model. With this protocol, there is a 50% chance of having a mismatch between the constructed signal and its model, and the nature of the mismatch is that the model has two too few unit sources.

• (c) The four unit sources protocol (‘4 protocol’) also assumes we know the left- or right-hand location of the earthquake. Now we set $\mathcal{K}$ to $\{1, 2, 4, 5 \}$ for a left-hand pick and to $\{2, 3, 5, 6 \}$ otherwise. There is again a 50% chance of having a mismatch, but now the nature of the mismatch is that there are two too many unit sources.

• (d) The matched protocol sets $\mathcal{K}$ so that it contains exactly the indices used in forming the constructed signal; i.e., we use the same set of ${\varvec{g}}_k$s both to form the signal and to model it so there is no mismatch. This protocol essentially presumes knowledge of the quadrant in which the random pick falls.

After selection of the protocol $\mathcal{K}$, the regression model of Eq. (3) is now fully specified. The model involves three coefficients for the tidal model ($\mu$, $\beta _1$ and $\beta _2$) and either one, two or four source coefficients $\alpha _k$ for the signal model. We estimate the regression coefficients using constrained least squares; i.e., the estimated coefficients $\hat{\mu }$, $\hat{\beta }_1$, $\hat{\beta }_2$ and $\hat{\alpha }_k$ are those minimizing

$$\begin{aligned} \left\| \bar{\varvec{y}} - \mu {\varvec{1}} - \beta _1 {\varvec{c}} - \beta _2 {\varvec{s}} - \sum _{k\in \mathcal{K}} \alpha _k {\varvec{g}}_k \right\| _2^2~\hbox{subject\,to}\,\alpha _k \ge 0\,\hbox{for\,all}\,k, \end{aligned}$$

where $\Vert {\varvec{x}} \Vert _2^2 = \sum _n x^2_n$ is the squared Euclidean norm of a vector ${\varvec{x}}$ with elements $x_n$. The nonnegativity constraint on each source coefficient is critical because, without it, there is nothing to prevent the least square estimate of $\alpha _k$ from being negative, which would render it inconsistent with the type of generating event we are assuming (a reverse fault earthquake).

Another important factor that impacts the quality of the inundation forecasts is the amount of data N available for estimating the source coefficients. In general, more data in $\bar{\varvec{y}}$ improves the quality of the estimated coefficients, which, in turn, improves inundation forecasts. To study the effect of N, we consider three settings intended to loosely mimic what would be available during different stages of an ongoing tsunami event. These settings are based on the location of the first peak of the constructed signal $\varvec{t}$ (alternatively we could use the first peak of the noisy data $\bar{\varvec{y}}$, but, to avoid issues that arise in automatic detection of a peak possibly distorted by noise, we have chosen to use the constructed signal instead). The first setting is 1 min past the first peak in the signal, and the second and third settings are 11 and 21 min past the peak. The three settings are illustrated in Fig. 16a, c using red circles. For this particular example, the settings correspond to 24, 34 and 44 min after the start of the earthquake, and we would estimate the coefficients based on placing the corresponding first $N=25$, 35 or 45 simulated buoy measurements in (c) into $\bar{\varvec{y}}$. For tsunami signals other than this example, we identify the first peak and the same related three points, but only use a maximum of 60 min worth of data prior to the first peak in cases where this peak occurs more than 1 hr after the start of the earthquake (this mimics the amount of the 1-min stream available during an actual event).

As an example of how the sample size influences source coefficient estimation, let us focus on the simulated DART^® buoy data $\bar{\varvec{y}}$ shown in Fig. 16c and set $\mathcal K$ as per the 2 protocol. Because the random pick is left-handed, the model for the tsunami signal becomes $\alpha _4 {\varvec{g}}_4 + \alpha _5 {\varvec{g}}_5$. There is thus a mismatch between the model and signal since the latter makes use of ${\varvec{g}}_1$ and ${\varvec{g}}_2$ in addition to ${\varvec{g}}_4$ and ${\varvec{g}}_4$. Table 1 shows the estimated source coefficients for the three sample sizes along with the presumed weights $w_4$ and $w_5$. The estimates improve with increasing N in the sense that $\hat{\alpha }_k$ gets closer to $w_k$.

Table 1 Estimated source coefficient $\hat{\alpha }_4$ and $\hat{\alpha }_5$ for three sample sizes N along with presumed weights $w_4$ and $w_5$

The forecasted wave amplitudes are

$$\begin{aligned} \hat{\varvec{h}} = \sum _{k\in \mathcal{K}} \hat{\alpha }_k {\varvec{h}}_k. \end{aligned}$$

(5)

For the 2 protocol with a left-handed pick, these amplitudes are given by

$$\begin{aligned} \hat{\varvec{h}} = \hat{\alpha }_4 {\varvec{h}}_4 + \hat{\alpha }_5 {\varvec{h}}_5, \end{aligned}$$

where ${\varvec{h}}_4$ and ${\varvec{h}}_5$ are depicted in Fig. 14. Use of the estimates for $\alpha _k$ corresponding to $N=45$ in Table 1 leads to the forecasted wave amplitudes shown in the right-hand plot of Fig. 15.

To summarize, estimates of the source coefficients $\alpha _k$ are imperfect in practice due to major factors including, inter alia,

• 1. an imperfect tidal model;

• 2. background noise;

• 3. a limited amount of data;

• 4. a mismatch between the assumed model and the actual tsunami signal; and

• 5. seismic noise, which is potentially important, but, because of timing, often does not come into play.

Our simulation study takes into account 1–4, but ignores seismic noise (which the next generation of DART^® buoys is designed to suppress entirely).

1.4 7.4. Evaluation of Forecasted Open-Ocean Wave Amplitudes

Our final task is to quantify how well the forecasted wave amplitudes $\hat{\varvec{h}}$ match the presumed amplitudes ${\varvec{h}}$. We considered three metrics. The first is the maximum cross-correlation between $\hat{\varvec{h}}$ and either ${\varvec{h}}$ or a lagged versions thereof; the second is the squared difference between the maximum amplitudes in $\hat{\varvec{h}}$ and ${\varvec{h}}$; and the third is the squared difference between the maximum 4-h ‘energies’ in $\hat{\varvec{h}}$ and ${\varvec{h}}$, where the energies in question are the sum of squares of data over all possible 4-h stretches contained within $\hat{\varvec{h}}$ or ${\varvec{h}}$. The latter two metrics are of more operational interest than maximum cross-correlation, which is also the most sensitive of the three to innocuous misalignments in time between $\hat{\varvec{h}}$ and ${\varvec{h}}$. In tests to date, we have found that use of either maximum wave amplitudes or maximum energies leads to evaluations of network effectiveness that are qualitatively similar. Since maximum wave amplitudes are less time consuming to compute, we stick with these in all that follows. For the example shown in Fig. 15, we have $\max \,\{{\varvec{h}} \} \doteq \,2.3$ cm and $\max \,\{\hat{\varvec{h}} \} \,\doteq\, 3.9$ cm, and the metric is $(\max \,\{\hat{\varvec{h}} \} - \max \,\{{\varvec{h}} \})^2 \, \doteq \, 2.6$ $\hbox {cm}^2$.

A more thorough assessment of how well we can forecast wave amplitudes outside of Hilo requires repeating what is set forth in the previous three subsections. We do so by generating 1000 different tsunami signals $\varvec{t}$ as per Eq. 1. The signals are associated with 1000 independent random picks from within unit source ac026b. Each pick serves as a center for a relocated unit source whose location dictates the weights $w_k$ used to form both $\varvec{t}$ and the presumed open-ocean wave amplitudes $\varvec{h}$ of Eq. 2. Each signal $\varvec{t}$ is then added to an independently drawn random sample of data recorded under ambient conditions by an appropriate DART^® buoy. This addition forms the simulated 1-min stream $\bar{\varvec{y}}$ as per Eq. 3. For each realization, we then compute twelve sets of estimated source coefficients $\hat{\alpha }_k$ corresponding to twelve formulations of the linear model of Eq. 4. The formulations involve the four protocols for setting $\mathcal K$ in combination with the three ways of determining the amount of data N to be placed in $\bar{\varvec{y}}$. For a given formulation, we use the source coefficient estimates $\hat{\alpha }_k$ to generate—as per Eq. 5—forecasted wave amplitudes $\hat{\varvec{h}}$ outside of the chosen impact site (Hilo in our representative triad). We then compare the peak values in ${\varvec{h}}$ and $\hat{\varvec{h}}$.

Figure 17 shows forecasted maximum amplitudes versus corresponding presumed amplitudes for 1000 realizations of our representative triad (central unit source ac026b, DART^® buoy 46403 and a location in the open ocean outside of the impact site Hilo). Each realization corresponds to a single point in each of the four plots. In addition to the 1000 points, each plot has a diagonal line indicating where a point would fall if the forecasted and presumed amplitudes were in perfect agreement. For all four plots, we set N such that 21 min worth of data past the peak value in the signal $\varvec{t}$ is used to estimate the source coefficients and other parameters in the regression model (Eq. 4). Plots (a) to (d) correspond to, respectively, the 1, 2, 4 and matched protocols $\mathcal K$ for fully specifying the model.

The colors of the points in Fig. 17 indicate the quadrant in which the random pick for a particular realization fell. Of the 1000 picks, 245 were in quadrant I (colored red); 247, in II (green); 268, in III (blue); and 240, in IV (black). This distribution is not inconsistent with each pick being equally likely to fall in any of the four quadrants. As noted in Fig. 13, picks in quadrants I and II involve linear combinations of four ${\varvec{g}}_k$s. For the 1 and 2 protocols, forecasted amplitudes are thus based on mismatched models. The mismatch is due to relevant ${\varvec{g}}_k$s being left out of the regression model. As a result, the red and green points in plots (a) and (b) have prominent scatter off the diagonal line. Plots (c) and (d) indicate that the forecasts are markedly better when there is no model mismatch, as occurs in the 4 and matched protocols (note that the red and green points in (c) and (d) are necessarily identical). In the absence of model mismatch, the remaining inaccuracies in the forecasts are mainly due to background noise and the tidal component.

By contrast, picks in quadrants III and IV (blue and black points) correspond to linear combinations of two ${\varvec{g}}_k$s. There is a model mismatch with the 1 and 4 protocols, but not for the 2 and matched protocols (note that the blue and black points in (b) and (d) are necessarily identical). Interestingly, when the mismatch is due to one less ${\varvec{g}}_k$ in the 1 protocol, there are highly structured deviations from the diagonal line. On the other hand, when the mismatch is due to two extraneous ${\varvec{g}}_k$s, as happens with the 4 protocol, the scatter about the diagonal line visually does not increase much over that of the 2 protocol (no mismatch).

We can summarize how well forecasted maximum wave amplitudes match up with the presumed amplitudes by computing a root-mean-square error (RMSE) for each scatterplot. Letting $\max \,\{\hat{\varvec{h}}_l \}$ and $\max \,\{{\varvec{h}}_l \}$ represent the forecasted and presumed amplitudes for the lth realization, this error is defined as

$$\begin{aligned} \hbox {RMSE} = \left[ \frac{1}{1000}\sum _{l=1}^{1000}\left( \max \,\{\hat{\varvec{h}}_l \} - \max \,\{{\varvec{h}}_l \}\right) ^2\right] ^{1/2}. \end{aligned}$$

(6)

Figure 18 shows the RMSEs for the four protocols in combination with three settings N for the amount of data used to estimate the source coefficients (dictated by up to either 1, 11 or 21 min past the first peak in the signal $\varvec{t}$). The four RMSEs corresponding to the scatterplots in Fig. 17 are shown above the ‘21’ label on the horizontal axis. In addition this figure has a dashed line showing the RMSE for a so-called seismic solution, for which a forecast is generated using $4 {\varvec{g}}_5$. This procedure assumes idealized seismic information about the tsunami event, namely, that the generating earthquake occurs within the central unit source ac026b and that the magnitude of the earthquake suggests setting the source coefficient to 4, which corresponds to our standard normalization $\sum _k \alpha _k = 4$ for the relocated unit source. The seismic-based forecast is of interest because it is arguably the best that can achieved without any information supplied by data from DART^® buoy 46403.

For all four protocols for formulating the regression model, the RMSEs decrease as more data are collected by the DART^® buoy. The decrease is minimal for the 1 protocol, but substantial for the other three protocols (in particular, there is a drop of more than two orders of magnitude for the 4 and matched protocols). The 1 protocol always involves a mismatch between the signal and its model. For the other protocols, there is at least a 50% chance of no mismatch. The poor performance of the 1 protocol points out the need for an adequate regression model. On the other hand, there is little difference in the RMSEs for the 4 and matched protocols even though the former involves mismatches in about 50% of the realizations, whereas the latter involves none. In contrast to the 1 protocol, the nature of the mismatch with the 4 protocol is too many ${\varvec{g}}_k$s rather than too few. This result suggests that having extraneous ${\varvec{g}}_k$s in the model (overfitting) does not significantly impact the forecasted wave amplitudes.

The seismic-based forecast outperforms the buoy-based forecasts either when there is an insufficient amount of data or when the regression model is always inadequate due to missing ${\varvec{g}}_k$s (the 1 protocol). When there is too little data, forecasts deteriorate considerably as the complexity of the regression model increases (i.e., more coefficients must be estimated). For our representative triad, there is some value to be gained by waiting the extra 10 min from 11 min past the first peak to 21 min (the RMSEs drop by 60% to 70%). For other triads involving different central unit sources and different buoys, the gain can be more substantial.

For the network assessment discussed in Sect. 4, we concentrate on the 4 protocol with N selected as dictated by 21 min after the first peak in the signal. This choice is close to the best RMSE for most—but not all—triads and should reflect the contribution of a particular buoy to wave amplitude forecasts at an impact site once enough data have been collected and once an adequate regression model has been identified.