How to evaluate a monitoring system for adaptive policies: criteria for signposts selection and their model-based evaluation

Abstract

Adaptive policies have emerged as a valuable strategy for dealing with uncertainties by recognising the capacity of systems to adapt over time to new circumstances and surprises. The efficacy of adaptive policies hinges on detecting on-going change and ensuring that actions are indeed taken if and when necessary. This is operationalised by including a monitoring system composed of signposts and triggers in the design of the plan. A well-designed monitoring system is indispensable for the effective implementation of adaptive policies. Despite the importance of monitoring for adaptive policies, the present literature has not considered criteria enabling the a-priori evaluation of the efficacy of signposts. In this paper, we introduce criteria for the evaluation of individual signposts and the monitoring system as a whole. These criteria are relevance, observability, completeness, and parsimony. These criteria are intended to enhance the capacity to detect the need for adaptation in the presence of noisy and ambiguous observations of the real system. The criteria are identified from an analysis of the information chain, from system observations to policy success, focusing on how data becomes information. We illustrate how models, in particular, the combined use of stochastic and exploratory modelling can be used to assess individual signposts, and the whole monitoring system according to these criteria. This analysis provides significant insight into critical factors that may hinder learning from data. The proposed criteria are demonstrated using a hypothetical case, in which a monitoring system for a flood protection policy in the Niger River is designed and tested.

Appendices

Appendix A: Decomposition of informativeness

In Eq. A.1, we use the law of total probability to decompose the monitoring of critical uncertainties into relevance and observability.

$$ P(\lambda_{i} | d_{t})= {\int}_{\delta} \mathcal{M}^{-1}(\lambda_{i} | \delta )\cdot P(\delta| d_{t}) \cdot d \delta $$

(A.1)

In Eq. A.1, λ_i are the i th critical uncertainty, d_t is the signpost estimate, i.e. the data obtained from observations of the real system, and δ_j is the signpost parameter, $\mathcal {M}$ is the system model, and P(λ_i|d_t) is the relation between critical uncertainties and signpost estimate. The system model, $\mathcal {M}(\delta |\lambda )$, contains the relationship between the critical uncertainties and the signpost parameter, and it is used backwards to estimate the latter from the earlier. $\mathcal {M}^{-1}(\lambda _{i} | \delta )$ is a measure of how critical uncertainties change with respect to the signpost parameter, hence the signpost relevance. P(δ|d_t) represents the possible values of the signposts parameter given its estimate, hence a measure of its observability.

Appendix B: Test-case values identification

Equation B.1 represents the Gumbel distribution.

$$ \mathcal{F}(q;\mu ,\sigma )=\exp\left( -\exp\left( -\frac{q-\mu }{\sigma} \right)\right) $$

(B.1)

where q is the yearly maximum discharge, μ and σ the location and scale parameter. The Gumbel distribution is equivalent to a generalised extreme value distribution for ξ = 0, a.k.a. type-I GEV.

1.1 B.1. Critical region

The boundary of the critical region in μ and σ is the set of [μ^∗,σ^∗], space in R², can be found by inverting Eq. B.1, conditioning the flood frequency to be equal to the critical flood frequency, i.e. F = F^∗, and the discharge equal to the flood threshold level, i.e. q = q_flood. Then, one can find the relation at Eq. B.2.

$$ \mu^{*}= q_{\text{flood}} + \log \left( -\log \left( 1-{F^{*}}\right)\right) \cdot \sigma^{*} $$

(B.2)

In Eq. B.2, the relation [μ^∗,σ^∗] is a straight line, in which q_flood is the intercept for σ = 0, and $\log (-\log \left (1- {F^{*}}\right ))$ its slope.

1.2 B.2. Signpost

Signposts distributions are derived analytically from Eq. B.1, assuming quasistationary condition.

S_E is distributed according to Eq. B.3.

$$ S_{E} \stackrel{\cdot}{\sim} \mathcal{N} \left( \mathbb{E}(q),\frac{\text{VAR}(q)}{n}\right) $$

(B.3)

In Eq. B.3, $\mathcal {N}$ is the normal distribution, $\mathbb {E}(q)$ and VAR(q) are the expected value and variance of Eq. B.1, being $\mathbb {E}(q)=\mu +\sigma \cdot \gamma$, and VAR(q) = (π² ⋅ σ²)/6, where $\pi \simeq 3.14$ is Greek pi, and $\gamma \simeq 0.577$ is the Euler–Mascheroni constant.

$S_{\max }$ is distributed according to Eq. B.4.

$$ S_{\max} \sim \mathcal{F}^{-1}\left( \hat{\mathcal{F}}_{n}(t)\right) $$

(B.4)

where

$$ \hat{\mathcal{F}}_{n}(t) \stackrel{\cdot}{\sim} \mathcal{N} \left( \mathcal{F}(t), \frac{\mathcal{F}(t)\cdot (1-\mathcal{F}(t))}{n}\right) $$

(B.5)

Equation B.4 use the property that quantile q convergences to der Vaart (2000) to In Eq. B.4, F^− 1(⋅) is the inverse of the original Gumbel distribution, as defined in Eq. B.1, and $\hat {F}_{n}(t)$ its empirical distribution. The empirical distribution converge to the original Gumbel distribution as in Eq. B.5 (der Vaart 2000). In Eq. B.5, $\mathcal {N}$ is the normal distribution, t the quantile, and n the sample size; for S₁₀, t = 24/25, and n = 25. Because of the low rate of convergency, however, in this application, we estimated the quantile distribution by a montecarlo approach, using 15000 sampling for each pair of (μ,σ).

1.3 B.3. Trigger point selection

A trigger point is the signpost value at which adaptation is required. In the test case, policy requires adaptation if flood frequency F exceeds the threshold level F^∗, of one over 25 years. Flood frequency level corresponds to a [μ^∗,σ^∗], that is a two-dimensional space in model parameters, as in Eq. B.2.

Trigger points are selected by sampling three equidistant combinations of location and scale parameter from the parameters space, in proximity of the parameters historically observed. Then, the three sets of parameters are mapped over the expected signpost values, finding the signpost values that would be measured, on average, for each set of parameters.