Feature Clustering for Extreme Events Analysis, with Application to Extreme Stream-Flow Data

Abstract

The dependence structure of extreme events of multivariate nature plays a special role for risk management applications, in particular in hydrology (flood risk). In a high dimensional context ($d>50$), a natural first step is dimension reduction. Analyzing the tails of a dataset requires specific approaches: earlier works have proposed a definition of sparsity adapted for extremes, together with an algorithm detecting such a pattern under strong sparsity assumptions. Given a dataset that exhibits no clear sparsity pattern we propose a clustering algorithm allowing to group together the features that are ‘dependent at extreme level’, i.e.,that are likely to take extreme values simultaneously. To bypass the computational issues that arise when it comes to dealing with possibly $O(2^d)$ subsets of features, our algorithm exploits the graphical structure stemming from the definition of the clusters, similarly to the Apriori algorithm, which reduces drastically the number of subsets to be screened. Results on simulated and real data show that our method allows a fast recovery of a meaningful summary of the dependence structure of extremes.

A Appendix: Proof of Lemma 1

Step 1. As a first step we show that $\mathcal {M}\subset \mathbb {M}, \text {i.e.,} \mu (\mathcal {C}_\alpha )>0 \Rightarrow \mu (\varGamma _\alpha )>0.$

Proof

Write $\mathcal {C}_\alpha = \bigcup _{\epsilon >0,\epsilon \in \mathbb {Q}} R_{\alpha ,\epsilon }$, where $R_{\alpha ,\epsilon } = \{x \in \mathbb {R}_+^d:\; \Vert x\Vert _{\infty }\ge 1; \quad x_j > \epsilon ~ (j\in \alpha ); \quad x_i = 0 ~ (i\notin \alpha ) \}.$ Assume $\mu (\mathcal {C}_\alpha )>0$. Since $\mu (\mathcal {C}_{\alpha } )<\infty$, by the monotonous limit property of the measure $\mu$, we have $\mu (\mathcal {C}_\alpha ) = \lim _{\epsilon \rightarrow 0} \mu (R_{\alpha ,\epsilon })$. Also, from the definitions, $R_{\alpha ,\epsilon }\subset \epsilon \varGamma _{\alpha }$. Thus,

$$\begin{aligned} \mu (\mathcal {C}_\alpha )>0&\Rightarrow \exists \epsilon \in (0,1) : \mu (R_{\alpha ,\epsilon })>0 \qquad \Rightarrow \mu (\epsilon \varGamma _{\alpha })>0 \\&\Rightarrow \rho _\alpha = \mu (\varGamma _{\alpha }) = \epsilon \mu (\epsilon \varGamma _{\alpha })>0. \end{aligned}$$

Step 2. We now prove the reverse inclusion for maximal elements of $\mathbb {M}$, i.e., 

$$\begin{aligned} \alpha \text { is maximal in } \mathbb {M} \quad \Rightarrow \alpha \in \mathcal {M}. \end{aligned}$$

(13)

Proof

Consider, for $i \notin \alpha$, the set $\;\Delta _{i, \epsilon } = \varGamma _{\alpha }\cap \{ x \in \mathbb {R}_+^d : \quad x_i > \epsilon \},$ so that $\varGamma _{\alpha } = \big \{ \bigcup _{\begin{array}{c} i \in \{1,\ldots ,d\}\setminus \alpha \\ \epsilon \in \mathbb {Q} \cap (0,1) \end{array}} \Delta _{i, \epsilon }\big \} \cup R_{\alpha , 1}.$ Thus,

$$\begin{aligned} \alpha \in \mathbb {M} ~~\Rightarrow ~~\mu (\varGamma _{\alpha } )>0 ~~\Rightarrow ~~\Big (\exists i, \mu (\Delta _{i,\epsilon })>0 ~~\text { or }~~ \mu ( R_{\alpha ,1} )>0\Big ) \end{aligned}$$

(14)

To prove (13), it is enough to show that

$$\begin{aligned} \alpha \in \mathbb {M}\quad \Rightarrow \quad \text { for } i\notin \alpha , \;\mu (\Delta _{i,\epsilon }) =0 . \end{aligned}$$

(15)

Indeed if (15) is true, and if $\alpha \in \mathbb {M}$, then (14) implies that $\mu (R_{\alpha ,1})>0$, and the result follows from the inclusion $R_{\alpha ,1}\subset \mathcal {C}_\alpha$. We show (15) by contradiction. If $\mu (\Delta _{i,\epsilon })>0$ for some $i\notin \alpha$, then

$$\frac{1}{\epsilon } \Delta _{i,\epsilon } = \left( \frac{1}{\epsilon }\,\varGamma _{\alpha }\right) \cap \{x\in \mathbb {R}_+^d : x_i>1 \}\subset \varGamma _{\alpha \cup \{i\}},$$

thus $\mu (\varGamma _{\alpha \cup \{i\}})>0$, which contradicts the maximality of $\alpha$ in $\mathbb {M}$.

Step 3. From (13), if $\alpha \text { is maximal in } \mathbb {M}$ then $\alpha \in \mathcal {M}$. Now if $\alpha$ is maximal in $\mathbb {M}$ but not in $\mathcal {M}$, there exists $\beta \supsetneq \alpha$ in $\mathcal {M}$. Thus from Step 1, $\beta \in \mathbb {M}$, a contradiction. Hence $\alpha$ is also maximal in $\mathcal {M}$. Conversely, if $\alpha$ is maximal in $\mathcal {M}$ then (Step 1) $\alpha \in \mathbb {M}$. If $\alpha$ was not maximal in $\mathbb {M}$, there would exist $\beta \supsetneq \alpha$ maximal in $\mathbb {M}$, and from (13), $\beta \in \mathcal {M}$, contradicting the maximality of $\alpha$ in $\mathcal {M}$.