Dimension Reduction for Systems with Slow Relaxation

Abstract

We develop reduced, stochastic models for high dimensional, dissipative dynamical systems that relax very slowly to equilibrium and can encode long term memory. We present a variety of empirical and first principles approaches for model reduction, and build a mathematical framework for analyzing the reduced models. We introduce the notions of universal and asymptotic filters to characterize ‘optimal’ model reductions for sloppy linear models. We illustrate our methods by applying them to the practically important problem of modeling evaporation in oil spills.

Appendices

Appendix 1: The Memory Kernel for Multiple Observables

One other comment is that we can indeed compute the memory kernel explicitly for the evaporation process (4), not just for the case with one observable, the mass $M_n$, but also more generally if we have a vector-valued linear observable $\varPhi$, i.e l scalar-valued observables $\varPhi = \{\phi ^1,\phi ^2,\ldots ,\phi ^l\}^T$. Each scalar linear observable $\phi ^i$ is given by an element of $\mathcal {H}^*$, and we will denote the corresponding bra-vector by $\langle {\phi ^i}|$. Using the Gram–Schmidt procedure if necessary, we can assume that the vectors $\langle {\phi ^i}|$ given an orthonormal basis for their span, a l-dimensional subspace of $\mathcal {H}^*$. The orthogonal projection $P^* : \mathcal {H}^* \rightarrow \mathcal {H}^*$ onto this subspace is given by

$$\begin{aligned} \mathcal {P}^* = \big \vert {\phi ^1}\big \rangle \langle {\phi ^1}| + \big \vert {\phi ^2}\big \rangle \langle {\phi ^2}| + \cdots + \big \vert {\phi ^2}\big \rangle \langle {\phi ^2}|. \end{aligned}$$

It follows that $\langle {\phi ^i}| P^* = \langle {\phi ^i}| P = \langle {\phi ^i}|$ and (11) gives

$$\begin{aligned} \langle {\phi ^i}|\big \vert {\xi _{n+1}}\big \rangle = \sum _{k=0}^n \sum _{j=1}^l \langle {\phi ^i}| \varLambda (Q \varLambda )^k\big \vert {\phi ^j}\big \rangle \langle {\phi ^j}|{\xi _n}\rangle + \langle {\phi ^i}| (\varLambda Q)^{n+1} |{\rho _0}\rangle . \end{aligned}$$

The quantities $\langle {\phi ^i}|{\xi _{n+1}}\rangle$ are the entries of the “vector” observable $\varPhi _{n+1}$. Defining the matrices $H_k$ by $(H_k)_{ij} = \langle {\phi ^i}| \varLambda (Q \varLambda )^k\big \vert {\phi ^j}\big \rangle$ for $k= 0,1,2,\ldots$ and the (column) vectors $\beta _n$ by the entries $\beta ^i_n = \langle {\phi ^i}| (\varLambda Q)^{n+1} \big \vert {\rho _0}\big \rangle$, we have the Mori–Zwanzig decomposition

$$\begin{aligned} \varPhi _{n+1} = \sum _{k=0}^n H_k \varPhi _{n-k} + \beta _n. \end{aligned}$$

(41)

If $\big \vert {\rho _0}\big \rangle$ is in the span of $\big \vert {\phi ^i}\big \rangle$, then $Q \big \vert {\rho _0}\big \rangle = 0$ so that the noise $\beta _n$ is identically zero. Taking $\big \vert {\rho _0}\big \rangle = \big \vert {\phi ^1}\big \rangle ,\big \vert {\phi ^2}\big \rangle ,\ldots ,\big \vert {\phi ^l}\big \rangle$ in turn, and collecting the corresponding column vectors $\varPhi _n$ into a $l \times l$ matrix $\varXi _n$, we have

$$\begin{aligned} (\varXi _n)_{ij} = \langle {\phi ^i}| \varLambda ^n \big \vert {\phi ^j}\big \rangle = \int _0^1 \phi ^i(w) e^{-n w \tau } \phi ^j(w) dw \end{aligned}$$

is a symmetric matrix for each n, and

$$\begin{aligned} \varXi _{n+1} = \sum _{k=0}^n H_k \varXi _{n-k}, \quad \text{ for }\; n = 0,1,2,\ldots \end{aligned}$$

As before, we can determine the memory kernel $H_k$ using the $\mathcal {Z}$-transform. Defining the matrices

$$\begin{aligned} \hat{\varXi }(z) = \sum _{n=0}^\infty z^{-n} \varXi _n, \quad \hat{H}(z) = \sum _{n=0}^\infty z^{-n} H_n \end{aligned}$$

we get

$$\begin{aligned} \hat{H}(z) = z(I - \hat{\varXi }(z)^{-1}). \end{aligned}$$

The matrix $\varXi _n$ is symmetric for all n, so that $H_n$ is also symmetric for all n. We expect that the norm $\Vert \varXi _n\Vert$ typically decays no faster than 1/n. This is true for instance if the constant functions are in the range of P, or more generally if there are continuous functions $\psi$ with $\psi (0) > 0$ in the range of P. In this case, we expect that the norm of $H_n$ decays no faster that $1/(n \log ^2(n))$ indicating again that, generically, one expects fat tails in the memory kernel for the system (4) if we use the Mori–Zwanzig decomposition based on any finite set of linear observables.

Appendix 2: Orthogonal Dynamics

We will now compute the statistics of the noise process $\beta _n$ in the Mori–Zwanzig decomposition (12) with the usual approach through the study of the projection equation (9) and the orthogonal dynamics (10). Since the orthogonal dynamics are linear, it suffices to solve the system

$$\begin{aligned} \langle {F_0}| = \langle {\delta (w-x)}| Q, \quad \langle {F_{n+1}}| = \langle {F_n}| \varLambda Q,\ \ n = 0,1,2,\ldots \end{aligned}$$

where $x \in [0,1]$ is fixed. A calculation reveals that, for any continuous function $\phi$,

$$\begin{aligned} \langle {F_0}|{\phi }\rangle = \langle {\delta (w-x)}|{\phi }\rangle - \langle {\delta (w-x)}| P \big \vert {\phi }\big \rangle = \phi (x) - \int _0^1 \phi (w) dw. \end{aligned}$$

We will thus associate $\langle {F_0}|$ with the “function” $F_0(w) = \delta (w-x) - 1$. We can follow this computation to solve the orthogonal dynamics equations recursively. For example,

$$\begin{aligned} \langle {F_1}|{\phi }\rangle&= \langle {\delta (w-x)-1}| \varLambda \big \vert {\phi }\big \rangle - \langle {\delta (w-x)-1}| \varLambda P \big \vert {\phi }\big \rangle \\&= \int _0^1 (\delta (w-x)-1) e^{-w \tau } \phi (w) dw - \int _0^1 (\delta (w-x)-1) e^{-w \tau } dw \int _0^1 \phi (w) dw \\&= \int _0^1 \left[ e^{-x\tau } \delta (w-x)- e^{-w \tau } - e^{-x \tau } + \frac{1-e^{-\tau }}{\tau }\right] \phi (w) dw, \end{aligned}$$

so that $\langle {F_1}|$ corresponds to the function $F_1(w) = e^{-x\tau } \delta (w-x)- e^{-w \tau } - e^{-x \tau } + \frac{1-e^{-\tau }}{\tau }$. Using the fact that Q and $\varLambda$ are self-adjoint operators on $\mathcal {H}$, and further $\langle {\psi }| \varLambda \big \vert {\phi }\big \rangle = \int \psi (w) e^{-w \tau } \phi (w) dw$ so that $\varLambda$ is diagonal on the “basis” $\{\delta (w-x)\}_{\{x \in [0,1]\}}$, an inductive argument shows that $F_n(w) = e^{-nx\tau } \delta (w-x) + \varPsi _n(w;x)$ where $\varPsi _n$ is a smooth, symmetric function $\varPsi _n(w;x) = \varPsi _n(x;w)$. We will use these conclusions to verify the full solution for $\langle {F_n}|$ that we obtain below by independent means.

Consider the $\mathcal {Z}$-transform $\hat{\langle {F}|} = \sum z^{-n} \langle {F_n}|$. The orthogonal dynamics imply

$$\begin{aligned} \hat{\langle {F}|} (1 - z^{-1} \varLambda Q) = \hat{\langle {F}|} - z^{-1} \hat{\langle {F}|} \varLambda + z^{-1} \hat{\langle {F}|} \varLambda \big \vert {1}\big \rangle \langle {1}| = \langle {F_0}|. \end{aligned}$$

Using the ansatz $\hat{F}(z,x,w) = \hat{A}(z,x) \delta (w-x) + \hat{\varPsi }(z,x,w)$ corresponding to a decomposition of $F_n$ into its singular and regular parts, we get the pair of equations

$$\begin{aligned} (1- z^{-1} e^{-x \tau }) \hat{A}&= 1, \\ \hat{\varPsi } - z^{-1} e^{-x \tau } \hat{\varPsi } + z^{-1} e^{-w \tau } \hat{A} + z^{-1} \int e^{-x \tau } \hat{\varPsi } dx&= -1, \end{aligned}$$

where we have suppressed the arguments (z, x, w) for $\hat{A}$ and $\hat{\varPsi }$ for clarity. We can solve the first equation to obtain

$$\begin{aligned} \hat{A} = \frac{1}{1-z^{-1} e^{-x \tau }}. \end{aligned}$$

Using this in the second equation, we obtain

$$\begin{aligned} \hat{\varPsi } = -\frac{1}{(1-z^{-1} e^{-x \tau }) (1-z^{-1} e^{-w \tau })} - \frac{z^{-1} C(z,w)}{1-z^{-1} e^{-x \tau }}, \end{aligned}$$

where $C(z,w) = \int e^{-x \tau } \hat{\varPsi } dx$ is determined in terms of the required solution $\hat{\varPsi }$ self-consistently. Multiplying by $e^{-x \tau }$ and integrating in x, and solving the resulting equation for C(z, w), we obtain

$$\begin{aligned} C(z,w) = -\frac{z\left( \tau -\log \left( 1-e^{\tau } z\right) +\log (1-z)\right) }{\left( 1-z^{-1} e^{-w \tau }\right) \left( \log (1-z)-\log \left( 1-e^{\tau } z\right) \right) }. \end{aligned}$$

Using this result in the computation for $\hat{\varPsi }$ gives

$$\begin{aligned} \hat{\varPsi }(z,x,w) = \frac{\tau }{\left( 1-z^{-1}e^{-w\tau }\right) \left( 1-z^{-1}e^{-x \tau }\right) \left( \log (1-z)-\log \left( 1-e^{\tau } z\right) \right) }. \end{aligned}$$

This gives a complete solution of orthogonal dynamics equation by

$$\begin{aligned} \hat{F}(z) = \frac{\delta (w-x)}{1-z^{-1} e^{-x \tau }} + \frac{\tau }{\left( 1-z^{-1}e^{-w\tau }\right) \left( 1-z^{-1}e^{-x \tau }\right) \left( \log (1-z)-\log \left( 1-e^{\tau } z\right) \right) }. \end{aligned}$$

The singular part of $F_n$ is therefore $e^{-n \tau x} \delta (w-x)$ as we noted above. Further, the regular part $\hat{\varPsi }$ is symmetric in w and x, implying this property for each of the functions $\varPsi _n$. Finally, for an observable given by a continuous function g, the solution to the orthogonal dynamics is given by

$$\begin{aligned} \sum _{n=0}^\infty z^{-n} \langle {g}|{Q (\varLambda Q)^n}{|\phi } \rangle = \int _0^1 \int _0^1 g(x) \hat{F}(z,w,x) \phi (w) dw dx. \end{aligned}$$

For the observable $M_{n}$, the prediction for the total mass at the next time step, we have $\langle {g}| = \langle {1}| \varLambda$. The $\mathcal {Z}$-transform of the memory kernel is given by

$$\begin{aligned} H(z) = \sum _{k \ge 1} z^{-k} h_k&= z^{-2} \sum _{k \ge 1} z^{-k+2} \langle {1}|{(\varLambda Q)^{k-1} \varLambda }|{1}\rangle \\&= z^{-1} \langle {1}|{\varLambda }|{1}\rangle +z^{-2}\sum _{n \ge 0} z^{-n} \langle {1}|{\varLambda Q (\varLambda Q)^n \varLambda }|{1}\rangle \\&= z^{-1} \frac{1-e^{-\tau }}{\tau } + z^{-2} \int _0^1 \int _0^1 e^{-x \tau } \hat{F}(z,w,x) e^{-w \tau } dw dx \\&= \left[ 1 - \frac{\tau }{\log (e^\tau z -1) - \log (z-1)}\right] . \end{aligned}$$

The $\mathcal {Z}$-transform of the expected values of the noise sequence $\beta _n$ is given by

$$\begin{aligned} \sum z^{-n} \mathbb {E}[\beta _n] = \sum z^{-n} \mathbb {E}[\langle {F_n}|{\rho _0}\rangle ] = \int _0^1 \int _0^1 e^{-x \tau } \hat{F}(z,w,x) dw dx = 0 \end{aligned}$$

and the correlations between the noise $\beta _n$ and the mass $M_j$ are given by $\mathbb {E}[\beta _n M_j] = \bar{\sigma }^2 \langle {1}|{(\varLambda Q)^n \varLambda ^j}|{1}\rangle$ (see Sect. 5). Taking the (two index) $\mathcal {Z}$-transform, noting that $\beta _0 = 0$, we have

$$\begin{aligned} \sum _{n \ge 1} \sum _{j \ge 0} z^{-n} \zeta ^{-j} \mathbb {E}[\beta _n M_j]&= \bar{\sigma }^2\sum _{n \ge 1} \sum _{j \ge 0} z^{-n} \zeta ^{-j} \langle {1}|{(\varLambda Q)^{n} \varLambda ^j}|{1}\rangle \\&= \bar{\sigma }^2z^{-1}\sum _{n \ge 0} \sum _{j \ge 0} z^{-n} \zeta ^{-j} \langle {1}|{\varLambda Q (\varLambda Q)^n \varLambda ^j}|{1}\rangle \\&= \bar{\sigma }^2z^{-1} \int _0^1 \int _0^1 \frac{e^{-x \tau } \hat{F}(z,w,x)}{1-\zeta ^{-1} e^{-w \tau }} dw dx \\&= \frac{\bar{\sigma }^2}{\tau } z^{-1} \Bigg [\frac{1-z^{-1}e^{ -\tau }}{(1-z^{-1})(1-z^{-1}e^{-\tau })} \\&\quad + \frac{\log \left( \frac{1-z^{-1}}{1-z^{-1}e^{-\tau }}\right) \left( \log \left( z \log \left( \frac{\zeta -1}{\zeta e^{\tau }-1}\right) - \zeta \log \left( \frac{z -1}{z e^{\tau }-1}\right) \right) \right) }{(z-\zeta ) \log \left( \frac{z-1}{e^{\tau } z-1} \right) }\Bigg ]. \end{aligned}$$

It is not true that $\mathbb {E}[\beta _n M_j] = 0$ if $n > j$, as one would expect in the Mori–Zwanzig decomposition for a system with an invariant measure. In particular,

$$\begin{aligned} \mathbb {E}[\beta _2 M_1] = \frac{\bar{\sigma }^2 (1-e^{-\tau })((\tau -2) +(\tau +2)e^{-\tau })}{2\tau ^2} \ne 0 \end{aligned}$$

Appendix 3: Sampling Initial Conditions

For any prescribed value $0< \bar{\sigma }^2 < \infty$, we can indeed find a family of I-dependent distributions such that

$$\begin{aligned} \mu _\gamma \rightarrow 1, \frac{\sigma _\gamma ^2}{I} \rightarrow \bar{\sigma }^2\;\text{ as }\; I \rightarrow \infty \end{aligned}$$

by appropriately truncating and rescaling a distribution that has finite mean but infinite variance. For example, the function

$$\begin{aligned} f(x) = {\left\{ \begin{array}{ll} \frac{9}{10} &{}\quad 0 \le x \le \frac{2}{3}, \\ \frac{9}{10}\left( \frac{3x}{2}\right) ^{5/2} &{}\quad x > \frac{2}{3} \end{array}\right. } \end{aligned}$$

satisfies $f \ge 0$ on $(0,\infty )$ and $\int _0^\infty f(x) dx = 1$, so f is indeed a nonmalized density on $(0,\infty )$. Further $\int _0^\infty x f(x) dx = 1$ and $\int _0^L x^2 f(x) dx \sim \sqrt{\frac{32}{75} L}$ for $L \gg 1$. We can therefore define a sequence of I dependent distributions by truncating the support of f and renormalizing to have unit mass, i.e.

$$\begin{aligned} f_I(x) = {\left\{ \begin{array}{ll} c_I f(x) &{} \quad 0 \le x \le L_I \\ 0 &{}\quad x > L_I, \end{array}\right. } \end{aligned}$$

where $L_I$ is any sequence satisfying $L_I \ge 2/3$ for all I, $L_I \nearrow \infty$ and ${\sqrt{\frac{32}{75 I^2} L_I} \rightarrow \bar{\sigma }^2}$ as $I \rightarrow \infty$. Given such a sequence $L_I$, the normalization $c_I$ is determined by $\int _0^I f_I(x) dx = 1$ so that $c_I \rightarrow 1$.

Appendix 4: Asymptotic Solutions of the Yule–Walker Equations

We seek a solution to (30) as an asymptotic series in n, i.e. solutions of the form

$$\begin{aligned} h^{(n)}_j = a^0_j + \frac{1}{n} a_1^j + \frac{1}{n^2} a_2^j + \cdots . \end{aligned}$$

(42)

The difficulty in solving this system is evident if we expand the coefficient matrix A as a power series in n:

$$\begin{aligned} A = \frac{1}{2n}\begin{pmatrix} 1 &{}\quad 1 &{}\quad \cdots &{}\quad 1 \\ 1 &{} \quad 1 &{} \quad \cdots &{}\quad 1 \\ \vdots &{}\quad \vdots &{}\quad \ddots &{} \quad \vdots \\ 1 &{}\quad 1 &{}\quad \cdots &{} \quad 1 \end{pmatrix} + \frac{1}{4n^2}\begin{pmatrix} 2 &{}\quad 3 &{} \quad \cdots &{}\quad L+1\\ 3 &{} \quad 4 &{} \quad \cdots &{} \quad L +2\\ \vdots &{}\quad \vdots &{} \quad \ddots &{}\quad \vdots \\ L +1&{} \quad L+2 &{} \quad \cdots &{}\quad 2L\end{pmatrix} + \cdots . \end{aligned}$$

Assuming $L \ge 3$, the two matrices displayed in the expansion of A are singular. The first matrix has rank 1, the second has rank 2. Indeed the first $L-1$ matrices in the expansion of A are all singular and their (row) nullspaces are nested

$$\begin{aligned} v^T \begin{pmatrix} 2 &{} \quad 3 &{}\quad \cdots &{}\quad L+1 \\ 3 &{}\quad 4 &{}\quad \cdots &{}\quad L+2 \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ L +1 &{}\quad L+2 &{} \quad \cdots &{}\quad 2L \end{pmatrix} = 0 \implies v^T \begin{pmatrix} 1 &{} \quad 1 &{}\quad \cdots &{} \quad 1 \\ 1 &{} \quad 1 &{} \quad \cdots &{} \quad 1 \\ \vdots &{}\quad \vdots &{} \quad \ddots &{}\quad \vdots \\ 1 &{} \quad 1 &{}\quad \cdots &{} \quad 1 \end{pmatrix} =0, \end{aligned}$$

and so on. The determinant of A is thus very close to zero ($\det \left\{ A \right\} \sim O(n^{-L^2})$ as we see below) so it is not clear that we have solutions for $h^{(n)}$ where the leading order behavior stays O(1) instead of diverging with n. Proving the boundedness of $h^{(n)}$ and determining the O(1) solution thus requires consideration of L solvability conditions given by the vectors that span the common (row)-nullspaces of the initial j terms in the expansion of A for $j=1,2,\ldots ,L-1$. Higher order terms will require even longer expansion of the matrices and more solvability conditions.

In the general case of a process with slowly decaying correlations, it is still true that the matrix of coefficients in the Yule–Walker equation is nearly singular, and one does have to go through the process described above to find optimal, reduced dimensional, models for such systems. For the evaporation process (4) however, the coefficient matrix has a special structure, that we exploit to find the solutions for the optimal filter $h^{(n)}$. The matrix A is a Cauchy matrix [48] i.e its entries are of the form $A_{ij} = 1/(x_i-y_j)$. In particular, we can choose $x_i = 2n-i$ and $y_j = j$. The determinant of a Cauchy matrix $A_{ij} = 1/(x_i-y_j)$ is given by [48]

$$\begin{aligned} \det {\left\{ {A}\right\} } = \frac{\prod _{i > j} (x_i-x_j)(y_j-y_i)}{\prod _i \prod _j (x_i-y_j)}. \end{aligned}$$

For the particular matrix A from above, the terms in the numerator are all bounded by L and the terms in the denominator are all $\approx 2n$ if $n \gg L$. Consequently, $\det {A} \sim O(n^{-L^2})$. The matrix $\hat{A}_m$ obtained by replacing the mth column of A by the vector $v_i = \frac{1}{2n -i}$ is also a Cauchy matrix $\hat{A}_{ij} = 1/(x_i - \hat{y}_j)$, with the same choice $x_i = 2n-i$ and

$$\begin{aligned} \hat{y}_j = {\left\{ \begin{array}{ll} y_j &{} \quad j \ne m, \\ 0 &{} \quad j = m. \end{array}\right. } \end{aligned}$$

Cramer’s rule now yields,

$$\begin{aligned} h^{(n)}_m = \frac{\det {\{ {\hat{A}}\}}}{\det {\{{{A}}\}}} = \prod _{i \ne m} \frac{i}{i-m} \prod _i \frac{2n - i -m}{2n - i}. \end{aligned}$$

(43)