# Learning Ergodic Averages in Chaotic Systems

- 128 Downloads

## Abstract

We propose a physics-informed machine learning method to predict the time average of a chaotic attractor. The method is based on the hybrid echo state network (hESN). We assume that the system is ergodic, so the time average is equal to the ergodic average. Compared to conventional echo state networks (ESN) (purely data-driven), the hESN uses additional information from an incomplete, or imperfect, physical model. We evaluate the performance of the hESN and compare it to that of an ESN. This approach is demonstrated on a chaotic time-delayed thermoacoustic system, where the inclusion of a physical model significantly improves the accuracy of the prediction, reducing the relative error from 48% to 1%. This improvement is obtained at the low extra cost of solving a small number of ordinary differential equations that contain physical information. This framework shows the potential of using machine learning techniques combined with prior physical knowledge to improve the prediction of time-averaged quantities in chaotic systems.

## Keywords

Echo state networks Hybrid echo state networks Physics-informed echo state networks Chaotic dynamical systems## 1 Introduction

In the past decade, there has been a proliferation of machine learning techniques applied in various fields, from spam filtering [7] to self-driving cars [3], including the more recent physical applications in fluid dynamics [4, 6]. However, a major hurdle in applying machine learning to complex physical systems, such as those in fluid dynamics, is the high cost of generating data for training [6]. Nevertheless, this can be mitigated by leveraging prior knowledge (e.g. physical laws). Physical knowledge can compensate for the small amount of training data. These approaches, called physics-informed machine learning, have been applied to various problems in fluid dynamics [4, 6]. For example, [5, 14] improve the predictability horizon of echo state networks by leveraging physical knowledge, which is enforced as a *hard* constraint in [5], without needing more data or neurons. In this study, we use a hybrid echo state network (hESN) [14], originally proposed to time-accurately forecast the evolution of chaotic dynamical systems, to predict the long-term time averaged quantities, i.e., the ergodic averages. This is motivated by recent research in optimization of chaotic multi-physics fluid dynamics problems with applications to thermoacoustic instabilities [8]. The hESN is based on reservoir computing [10], in particular, conventional Echo State Networks (ESNs). ESNs have shown to predict nonlinear and chaotic dynamics more accurately and for a longer time horizon than other deep learning algorithms [10]. However, we stress that the present study is not focused on the accurate prediction of the time evolution of the system, but rather of its ergodic averages, which are obtained by the time averaging of a long time series (we implicitly assume that the system is ergodic, thus, the infinite time average is equal to the ergodic average [2].). Here, the physical system under study is a prototypical time-delayed thermoacoustic system, whose chaotic dynamics have been analyzed and optimized in [8].

## 2 Echo State Networks

*n*is the discrete time index. The ESN is composed of a reservoir, which can be represented as a directed weighted graph with \(N_x\) nodes, called neurons, whose state at time

*n*is given by the vector \(\varvec{x}(n) \in \mathbb {R}^{N_x}\). The reservoir is coupled to the input via an input-to-reservoir matrix, \(\varvec{W}_\mathrm {in}\), such that its state evolves according to

*j*to node

*i*, and the hyperbolic tangent is the activation function. Finally, the prediction is produced by a linear combination of the states of the neurons

*n*is the input at time step \(n+1\), i.e. \(\varvec{y}(n) = \varvec{u}(n+1)\) [14]. We wish to learn ergodic averages, given by

*can not*generate non-symmetric attractors. This symmetry needs to be broken to work with a general non-symmetric dynamical system. This can be done by including biases [10]. However, the addition of a bias can make the reservoir prone to saturation (results not shown), i.e. \(x_i \rightarrow \pm 1\), and thus care needs to be taken in the choice of hyperparameters. In this paper, we break the symmetry by exploiting prior knowledge on the physics of the problem under investigation with a hybrid ESN.

## 3 Physics-Informed and Hybrid Echo State Network

*hard*constraint with a physics loss term. The prediction is consistent with the physics, but the training requires nonlinear optimization. The authors of [14] introduced a hybrid echo state network (hESN), which incorporates incomplete physical knowledge by feeding the prediction of the physical model into the reservoir and into the output. This requires ridge regression. Here, we use an hESN (Fig. 1) because we are not interested in constraining the physics as a hard constraint for an accurate short-term prediction [5]. In the hESN, similarly to the conventional ESN, the input is fed to the reservoir via the input layer \(\varvec{W}_\mathrm {in}\), but also to a physical model, which is usually a set of ordinary differential equations that approximately describe the system that is to be predicted. In this work, that model is a reduced-order model (ROM) of the full system. The output of the ROM is then fed to the reservoir via the input layer and into the output of the network via the output layer.

## 4 Learning the Ergodic Average of an Energy

*u*,

*p*, \(\zeta \) and \(\dot{q}\) are the non-dimensionalized acoustic velocity, pressure, damping and heat-release rate, respectively. \(\delta \) is the Dirac delta. These equations are discretized by using \(N_g\) Galerkin modes

We stress that the optimal values of hyperparameters for a certain set of physical parameters, e.g. \((\beta _1, \tau _1)\), might not be optimal for a different set of physical parameters \((\beta _2, \tau _2)\). This should not be surprising, since different physical parameters will result in different attractors. For example, Fig. 4 shows that changing the physical parameters from \((\beta =7.0, \tau =0.2)\) to \((\beta =6.0, \tau =0.3)\) results in a change of type of attractor from chaotic to limit cycle. For the hESN to predict the limit cycle, the value of \(\sigma _\mathrm {in}\) must change from 0.2 to 0.03 Thus, if the hESN (or any deep learning technique in general) is to be used to predict the dynamics of various physical configurations (e.g. the generation of a bifurcation diagram), then it should be coupled with a robust method for the automatic selection of optimal hyperparameters [1], with a promising candidate being Bayesian optimization [15, 17].

## 5 Conclusion and Future Directions

We propose the use of echo state networks informed with incomplete prior physical knowledge for the prediction of time averaged cost functionals in chaotic dynamical systems. We apply this to chaotic acoustic oscillations, which is relevant to aeronautical propulsion. The inclusion of physical knowledge comes at a low cost and significantly improves the performance of conventional echo state networks from a 48% error to 1%, without requiring additional data or neurons. This improvement is obtained at the low extra cost of solving a small number of ordinary differential equations that contain physical information. The ability of the proposed ESN can be exploited in the optimization of chaotic systems by accelerating computationally expensive shadowing methods [13]. For future work, (i) the performance of the hybrid echo state network should be compared against those of other physics-informed machine learning techniques; (ii) robust methods for hyperparameters’ search should be coupled for a “hands-off” autonomous tool; and (iii) this technique is currently being applied to larger scale problems. In summary, the proposed framework is able to learn the ergodic average of a fluid dynamics system, which opens up new possibilities for the optimization of highly unsteady problems.

## References

- 1.Bengio, Y.: Practical recommendations for gradient-based training of deep architectures. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 437–478. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_26CrossRefGoogle Scholar
- 2.Birkhoff, G.D.: Proof of the ergodic theorem. Proc. Natl. Acad. Sci.
**17**(12), 656–660 (1931)CrossRefGoogle Scholar - 3.Bojarski, M., et al.: End to end learning for self-driving cars. arXiv e-prints, arXiv:1604.07316 (2016)
- 4.Brunton, S.L., Noack, B.R., Koumoutsakos, P.: Machine learning for fluid mechanics. Annu. Rev. Fluid Mech.
**52**(1), 477–508 (2020)CrossRefGoogle Scholar - 5.Doan, N.A.K., Polifke, W., Magri, L.: Physics-informed echo state networks for chaotic systems forecasting. In: Rodrigues, J.M.F., Cardoso, P.J.S., Monteiro, J., Lam, R., Krzhizhanovskaya, V.V., Lees, M.H., Dongarra, J.J., Sloot, P.M.A. (eds.) ICCS 2019. LNCS, vol. 11539, pp. 192–198. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22747-0_15CrossRefGoogle Scholar
- 6.Duraisamy, K., Iaccarino, G., Xiao, H.: Turbulence modeling in the age of data. Annu. Rev. Fluid Mech.
**51**(1), 357–377 (2019)MathSciNetCrossRefGoogle Scholar - 7.Guzella, T.S., Caminhas, W.M.: A review of machine learning approaches to spam filtering. Expert Syst. Appl.
**36**(7), 10206–10222 (2009)CrossRefGoogle Scholar - 8.Huhn, F., Magri, L.: Stability, sensitivity and optimisation of chaotic acoustic oscillations. J. Fluid Mech.
**882**, A24 (2020)MathSciNetCrossRefGoogle Scholar - 9.Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science
**304**(5667), 78–80 (2004)CrossRefGoogle Scholar - 10.Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev.
**3**(3), 127–149 (2009)CrossRefGoogle Scholar - 11.Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_36CrossRefGoogle Scholar
- 12.Nair, V., Sujith, R.: A reduced-order model for the onset of combustion instability: physical mechanisms for intermittency and precursors. Proc. Combust. Inst.
**35**(3), 3193–3200 (2015)CrossRefGoogle Scholar - 13.Ni, A., Wang, Q.: Sensitivity analysis on chaotic dynamical systems by non-intrusive least squares shadowing (NILSS). J. Comput. Phys.
**347**, 56–77 (2017)MathSciNetCrossRefGoogle Scholar - 14.Pathak, J.: Hybrid forecasting of chaotic processes: using machine learning in conjunction with a knowledge-based model. Chaos: Interdisc. J. Nonlinear Sci.
**28**(4), 41101 (2018)MathSciNetCrossRefGoogle Scholar - 15.Reinier Maat, J., Gianniotis, N., Protopapas, P.: Efficient optimization of echo state networks for time series datasets. arXiv e-prints, arXiv:1903.05071 (2019)
- 16.Traverso, T., Magri, L.: Data assimilation in a nonlinear time-delayed dynamical system with lagrangian optimization. In: Rodrigues, J.M.F., Cardoso, P.J.S., Monteiro, J., Lam, R., Krzhizhanovskaya, V.V., Lees, M.H., Dongarra, J.J., Sloot, P.M.A. (eds.) ICCS 2019. LNCS, vol. 11539, pp. 156–168. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22747-0_12CrossRefGoogle Scholar
- 17.Yperman, J., Becker, T.: Bayesian optimization of hyper-parameters in reservoir computing. arXiv e-prints, arXiv:1611.05193 (2016)