Estimating Lyapunov Exponents from Time Series

Abstract

Lyapunov exponents are important statistics for quantifying stability and deterministic chaos in dynamical systems. In this review article, we first revisit the computation of the Lyapunov spectrum using model equations. Then, employing state space reconstruction (delay coordinates), two approaches for estimating Lyapunov exponents from time series are presented: methods based on approximations of Jacobian matrices of the reconstructed flow and so-called direct methods evaluating the evolution of the distances of neighbouring orbits. Most direct methods estimate the largest Lyapunov exponent, only, but as an advantage they give graphical feedback to the user to confirm exponential divergence. This feedback provides valuable information concerning the validity and accuracy of the estimation results. Therefore, we focus on this type of algorithms for estimating Lyapunov exponents from time series and illustrate its features by the (iterated) Hénon map, the hyper chaotic folded-towel map, the well known chaotic Lorenz-63 system, and a time continuous 6-dimensional Lorenz-96 model. These examples show that the largest Lyapunov exponent from a time series of a low-dimensional chaotic system can be successfully estimated using direct methods. With increasing attractor dimension, however, much longer time series are required and it turns out to be crucial to take into account only those neighbouring trajectory segments in delay coordinates space which are located sufficiently close together.

Appendix

Let $\varphi ^{k}: \mathbb{R}^{D} \rightarrow \mathbb{R}^{D}$ be the induced flow in reconstruction space mapping reconstructed states $\mathbf{x}_{n} = (s_{n},s_{n+L},\ldots,s_{n+(D-1)L})$ to their future values $\varphi ^{k}(\mathbf{x}_{n}) =\mathbf{ x}_{n+k}$. To estimate the D × D Jacobian matrices $D\varphi ^{k}(\mathbf{x})$ from the temporal evolution of the reconstructed states $\{\mathbf{x}_{n}\}_{n=1}^{N}$ the flow $\varphi ^{k}$ has to be approximated by a general ansatz (black-box model) like a neural network [20] or a superposition of I basis functions $b_{i}: \mathbb{R}^{D} \rightarrow \mathbb{R}$ providing an approximating function

$$\displaystyle{ \psi ^{k}(\mathbf{x}) = \left (\psi _{ 1}^{k}(\mathbf{x}),\ldots,\psi _{ D}^{k}(\mathbf{x})\right ) = \left (\sum _{ i=1}^{I}c_{ i1}b_{i}(\mathbf{x}),\ldots,\sum _{i=1}^{I}c_{ iD}b_{i}(\mathbf{x})\right ) =\mathbf{ b}(\mathbf{x}) \cdot C }$$

(1.50)

where C = (c _ij ) denotes a I × D matrix of coefficients with columns $\mathbf{c}^{(\,j)}$ ( j = 1, …, D) that have to be estimated and $\mathbf{b}(\mathbf{x}) = (b_{1}(\mathbf{x}),\ldots,b_{I}(\mathbf{x}))$ is a row vector consisting of the values of all basis functions evaluated at the state $\mathbf{x}$.⁴

For the special choice k = L (evolution time step equals the lag of the delay coordinates) the first D − 1 components of the map $\psi ^{k}(\mathbf{x}_{n})$ are known (due to the delay reconstruction) and only for the last component an approximation is required

$$\displaystyle\begin{array}{rcl} \psi ^{L}(\mathbf{x})& =& \left (s_{ n+L},s_{n+2L},\ldots,s_{n+(D-1)L},\sum _{i=1}^{I}c_{ i}b_{i}(\mathbf{x})\right ){}\end{array}$$

(1.51)

$$\displaystyle\begin{array}{rcl} & =& (x_{n2},\ldots,x_{nD},\mathbf{b}(\mathbf{x}) \cdot \mathbf{ c}).{}\end{array}$$

(1.52)

With this notation the approximation $D\psi ^{k}(\mathbf{x})$ of the desired Jacobian matrix $D\varphi ^{k}(\mathbf{x})$ of the (induced) flow $\varphi (\mathbf{x})$ in embedding space can be written as

$$\displaystyle{ D\psi ^{k}(\mathbf{x}) = \left (\begin{array}{ccc} \frac{\partial b_{1}} {\partial x_{1}} & \ldots & \frac{\partial b_{I}} {\partial x_{1}}\\ \vdots & \ddots & \vdots \\ \frac{\partial b_{1}} {\partial x_{D}} & \ldots & \frac{\partial b_{I}} {\partial x_{D}} \end{array} \right )\cdot C = G\cdot C }$$

(1.53)

where G will be called derivative matrix in the following.

For k = L the Jacobian matrix of the approximating function ψ ^k is given as

$$\displaystyle{ D\psi ^{L}(\mathbf{x}) = \left (\begin{array}{ccccc} 0 &1&0&\ldots & 0\\ 0 &0 &1 &\ldots & 0\\ \vdots & \vdots & \vdots &\ddots & \vdots \\ 0 &0&0&\ldots & 1 \\ \sum _{i=1}^{I}c_{i} \frac{\partial b_{i}} {\partial x_{1}} & \ldots & \ldots & \ldots & \sum _{i=1}^{I}c_{i} \frac{\partial b_{i}} {\partial x_{D}} \end{array} \right ) }$$

(1.54)

Linear basis functions $b_{i}(\mathbf{x})$ can be used to model the (linearized) flow (very) close to the reference points $\mathbf{x}_{n}$ along the orbits. To approximate the flow in a larger neighbourhood of $\mathbf{x}_{n}$ or even globally, nonlinear basis functions are required, like multidimensional polynomials [2, 4–7], or radial basis functions [23, 24, 35].

To estimate the coefficient matrix C in Eq. (1.50) or the coefficient vector $\mathbf{c}$ in Eq. (1.51) we select a set of representative states $\{\mathbf{z}^{j}\}$ whose temporal evolution $\varphi ^{k}(\mathbf{z}^{j})$ is known. For local modeling this set of states consists of nearest neighbours $\{\mathbf{x}_{m(n)}: m(n) \in \mathcal{U}_{n}\}$ of the reference point $\mathbf{x}_{n}$ where $\mathcal{U}_{n}$ defines the chosen neighbourhood that can be of fixed mass (a fixed number K of nearest neighbours of $\mathbf{x}_{n}$) or of fixed size (all points with distance smaller than a given bound ε). For global modeling of the flow the set $\{\mathbf{z}^{j}\}$ is usually a (randomly sampled) subset of all reconstructed states. Let

$$\displaystyle{ Y = \left (\begin{array}{cccc} \varphi _{1}^{k}({\boldsymbol z^{1}}) &\ldots & \varphi _{D}^{k}({\boldsymbol z^{1}})\\ \vdots &\vdots & \vdots \\ \varphi _{1}^{k}({\boldsymbol z^{J}})&\ldots &\varphi _{D}^{k}({\boldsymbol z^{J}})\\ \end{array} \right ) }$$

(1.55)

be a J × D matrix whose rows are components the (known) future values $\varphi ^{k}(\mathbf{z}^{j})$ of the J states $\{\mathbf{z}^{j}\}$ and let

$$\displaystyle{ B = \left (\begin{array}{cccc} b_{1}({\boldsymbol z^{1}}) &\ldots & b_{I}({\boldsymbol z^{1}})\\ \vdots &\vdots & \vdots \\ b_{1}({\boldsymbol z^{J}})&\ldots &b_{I}({\boldsymbol z^{J}})\\ \end{array} \right ) }$$

(1.56)

be the J × I (design) matrix [37] whose rows are the basis functions b _i (⋅ ) evaluated at the selected states $\{\mathbf{z}^{j}\}$. Using this notation the approximation task can be stated as a minimization problem with a cost function

$$\displaystyle{ g(\mathbf{c}^{(j)}) =\Vert B \cdot \mathbf{ c}^{(j)} -\mathbf{ y}^{(j)}\Vert ^{2} }$$

(1.57)

where y ^(j) denotes the j-th column of the matrix Y (given in Eq. (1.55)), or

$$\displaystyle{ g(C) =\Vert B \cdot C - Y \Vert _{F}^{2} }$$

(1.58)

where $\Vert \cdot \Vert _{F} =$ denotes the Frobenius matrix norm (also called Schur norm).

The solution of this optimization problem may suffer from the fact that typically the states $\{\mathbf{z}^{j}\}$ cover only some subspace of the reconstructed state space. Therefore, in particular for local modeling ill-posed optimization problems may occur with many almost equivalent solutions. For estimating Lyapunov exponents we prefer to select solutions for the coefficient matrix C that provide partial derivatives (elements of the Jacobian matrix) with small magnitudes, because in this way spurious Lyapunov exponents are shifted towards $-\infty$. This goal can be achieved by Tikhonov–Philips regularization where the cost function of the optimization problem (1.58) is extended by a term ρ ∥ A ⋅ C ∥ resulting in

$$\displaystyle{ g(\mathbf{c}^{(j)}) =\Vert B \cdot \mathbf{ c}^{(j)} -\mathbf{ y}^{(j)}\Vert ^{2} +\rho ^{2}\Vert A \cdot \mathbf{ c}^{(j)}\Vert ^{2} }$$

(1.59)

where A denotes a so-called stabilizer matrix and $\rho \in \mathbb{R}$ is the regularization parameter that is used to control the impact of the regularization term on the solution of the minimization problem. If the identity matrix is used as stabilizer A = I then $\Vert \mathbf{c}(j)\Vert$ is minimized and the solution with the smallest coefficients is selected (also called Tikhonov stabilization). Another possible choice is the derivative matrix (1.53) A = G. In this case we minimize the sum of all squared singular values $\sigma _{i}$ of $D\psi ^{k}(\mathbf{x}) = U \cdot S \cdot V ^{tr}$, because

$$\displaystyle\begin{array}{rcl} \Vert G \cdot C\Vert _{F}^{2}& =& \Vert D\psi ^{k}(\mathbf{x})\Vert _{ F}^{2} = \mbox{ trace}\left ([D\psi ^{k}(\mathbf{x})]^{tr} \cdot D\psi ^{k}(\mathbf{x})\right ){}\end{array}$$

(1.60)

$$\displaystyle\begin{array}{rcl} & =& \mbox{ trace}\left (V \cdot S^{2} \cdot V ^{tr}\right ) = \mbox{ trace}(S^{2}) =\sum _{ i=1}^{D}\sigma _{ i}^{2}{}\end{array}$$

(1.61)

and so we minimize Lyapunov exponents by maximizing contraction rates.

To solve the optimization problem (1.59) we rewrite it as an augmented least squares problem with a cost function

$$\displaystyle{ g(\mathbf{c}^{(j)}) =\Vert \left (\begin{array}{c} B \\ \rho A \end{array} \right )\cdot \mathbf{c}^{(j)}-\left (\begin{array}{c} \mathbf{y}^{(j)} \\ 0\end{array} \right )\Vert ^{2} =\Vert \hat{B} \cdot \mathbf{c}^{(j)}-\hat{\mathbf{y}} ^{(j)}\Vert ^{2} }$$

(1.62)

that can be minimized by a solution of the corresponding normal equations

$$\displaystyle{ \left (B^{tr} \cdot B +\rho ^{2}A^{tr} \cdot A\right ) \cdot \mathbf{ c}^{(j)} = B^{tr} \cdot \mathbf{ y}^{(j)} }$$

(1.63)

using a sequence of Householder transformations [35] or by employing the singular value decomposition of the matrix $\hat{B} = U_{\hat{B} } \cdot S_{\hat{B} } \cdot V _{\hat{B} }^{tr}$ providing the minimal solution [37]

$$\displaystyle{ \mathbf{c}^{(j)} = V _{\hat{ B} } \cdot S_{\hat{B} }^{-1} \cdot U_{\hat{ B} }^{tr} \cdot \hat{\mathbf{y}} ^{(j)}. }$$

(1.64)

for each column $\hat{\mathbf{c}} ^{(j)}$ or

$$\displaystyle{ C = V _{\hat{B} } \cdot S_{\hat{B} }^{-1} \cdot U_{\hat{ B} }^{tr} \cdot \hat{Y } }$$

(1.65)

for the full coefficient matrix C where $\hat{Y } = \left (\begin{array}{c} Y\\ 0 \end{array} \right )$.

For the stabilizer A = I the elements of the diagonal matrix $S_{\hat{B} }^{-1}$ are given by

$$\displaystyle{ \frac{\hat{\sigma }_{i}} {\hat{\sigma }_{i}^{2} +\rho ^{2}} }$$

(1.66)

where $\hat{\sigma }_{i}$ are the diagonal elements of $S_{\hat{B} }$ (i.e., the singular values of$\hat{B}$).