Assessment of end-to-end and sequential data-driven learning for non-intrusive modeling of fluid flows

Abstract

In this work, we explore the advantages of end-to-end learning of multilayer maps offered by feedforward neural networks (FFNNs) for learning and predicting dynamics from transient flow data. While data-driven learning (and machine learning) in general depends on data quality and quantity relative to the underlying dynamics of the system, it is important for a given data-driven learning architecture to make the most of this available information. To this end, we focus on data-driven problems where there is a need to predict over reasonable time into the future with limited data availability. Such function time series prediction of full and reduced states is different from many applications of machine learning such as pattern recognition and parameter estimation that leverage large datasets. In this study, we interpret the suite of recently popular data-driven learning approaches that approximate the dynamics as Markov linear model in higher dimensional feature space as a multilayer architecture similar to neural networks. However, there exist a couple of key differences: (i) Markov linear models employ layer-wise learning in the sense of linear regression whereas neural networks represent end-to-end learning in the sense of nonlinear regression. We show through examples of data-driven modeling of canonical fluid flows that FFNN-like methods owe their success to leveraging the extended learning parameter space available in end-to-end learning without overfitting to the data. In this sense, the Markov linear models behave as shallow neural networks. (ii) The second major difference is that while the FFNN is by design a forward architecture, the class of Markov linear methods that approximate the Koopman operator is bi-directional, i.e., they incorporate both forward and backward maps in order to learn a linear map that can provide insight into spectral characteristics. In this study, we assess both reconstruction and predictive performance of temporally evolving dynamic using limited data for canonical nonlinear fluid flows including transient cylinder wake flow and the instability-driven dynamics of buoyant Boussinesq flow.

Appendices

Appendix A: Nomenclature and definitions

1.1 A.1 Nomenclature

F	=	nonlinear dynamical operator	λ	=	regularization parameter
$\mathcal {K}$	=	Koopman operator	N	=	dimensions of state space
${\mathcal {K}_{a}}$	=	approximate Koopman operator	M	=	dimensions of snapshots
$\mathcal {F}$	=	functional space	K,R	=	dimension in features
C	=	convolution operator	N f	=	neurons/feature growth
					factor
C m l	=	multilayer convolution operator	p i	=	i th-order polynomial
g,h	=	observable functions	DMD	=	dynamic mode
					decomposition
x,y	=	state vectors	EDMD	=	extended dynamic mode
		(e.g., velocity field)			decomposition
X,Y	=	snapshot data pairs	POD	=	proper orthogonal
					decomposition
$\bar X,\bar Y$	=	POD weights data pairs	SVD	=	singular value
					decomposition
Θl	=	convolution operators or weights matrix	MSM	=	multilayer sequential map
$\mathcal {N}$	=	nonlinear map	MEM	=	multilayer end-to-end map
a	=	POD weights/coefficients	FFNN	=	feedforward neural networks
$\mathcal {J}$	=	cost function	()+	=	pseudo-inverse
$\mathcal {LP}$	=	learning parameter	$\bar {()}$	=	single convolution
			$\bar {()}^{i}$	=	multiple convolutions

1.2 A.2 Definitions

$\mathcal {LP}$ for MSMs	:	Number of elements in matrix ($\mathcal {K}_{a}$).
	:	For example, $\mathcal {K}_{a}$ with size [9,9] has $\mathcal {LP}$ = 81.
$\mathcal {LP}$ for MEMs	:	Summation of all of elements in weight matrices
		(Θ’s). For example, a two hidden layer network
	:	with Nf = 3 has three weight matrices with
		sizes Θ1 = [3, 9], Θ2 = [9, 9], Θ3 = [9, 3].
	:	The $\mathcal {LP} = 3\mathrm {x}9 + 9\mathrm {x}9 + 9\mathrm {x}3 = 135$
Feature growth factor (Nf)	:	Helps calculate the number hidden units or
		Neurons (Nh) in feedforward neural network
	:	based on the number of features(K) used
		for the model.
	:	For example, with Nf = 3 and 3 features, the
		number of neurons in each hidden layer
		would be 9

Polynomial feature growth pi	:	Order of polynomial used to grow the input
		features to learn MSM models.
	:	With 3 features and polynomial order of 2,
		the feature growth would be 9.

Appendix B: Learning parameter budget and POD space dimension

The number of learning parameters are dependent on the size of input POD space dimension and also the choice of model. For example, the MSM model with DMD architecture learns $\mathcal {LP}$ parameters that has a quadratic dependence on the POD space dimension, K or $\mathcal {LP} \propto O(K^{2})$. However, for the MSM/EDMD architecture, it turns out that $\mathcal {LP}$ depends on the extended functional basis dimension which is usually N_f(K) ∗ K. Here, N_f(K) is a prefactor to K that also depends on K. Therefore, $\mathcal {LP} \propto O((N_{f}(K)*K)^{2})$ to yield a super quadratic dependence on K. In fact, for P2, $N_{f}(K)\sim K$ whereas for P7, $N_{f}(K)\sim 40N$ when K = 3. For the MEM/FFNN methods, $\mathcal {LP}$ depends on both N, the neuron growth factor, N_f, and number of layers , L, i.e., $\mathcal {LP} \propto O((N_{f}*N)^{2}L)$. Typically, in our studies, N_f and L tend to be independent of N and therefore, $\mathcal {LP}$ a shows quadratic dependence on the POD-space dimension.

Appendix C: Algorithm complexity

The cost of computing $\mathcal {K}_{a}$ is directly related to number of input features, feature growth factor, and method used. In fact, this is not dissimilar to the parameter budgets discussed above. For the MSM method, where g,h are precomputed, we use Moore–Penrose inverse (pinv in MATLAB) for computing the transfer operator $\mathcal {K}_{a}$ using least squares error minimization. Given K features and M snapshots, the cost of computing the pseudo-inverse is O(M xK²) and the estimation of the evolved state, YX⁺ requires order O(K xM²). Therefore, the total complexity is O(KM ∗ (M + K)). When K ≪ M, the complexity of the MSM model learning is O(KM²) and O(K²M) otherwise.

For the MSM method with EDMD-P2, i.e., using 2nd-order polynomial features, the feature dimension R grows as K + K(K + 1)/2. In this case, computational cost of estimating $\mathcal {K}_{a}$ is of order O(M²xK² + M xK⁴). When using higher order polynomial such as in EDMD-P7, computational cost is expected to depend on higher powers of K. Therefore, EDMD methods become expensive when using a higher dimensional POD space. For example, even a 2nd-order polynomial can become a limiting factor with large initial feature set, say, 50 POD modes and 400 snapshots, the resulting $\mathcal {K}_{a}$ matrix would be of size [1325,1325] and the number of elements stored and computed is O(1, 755, 625), O(212 × 10⁶) operations, respectively.

The computation cost of MEM (FFNN) method primarily depends on the number of hidden layers (L) and feature growth factor (N_f). The method consists a forward sweep of matrix multiplications, error/cost estimation, and backsweep of predominantly matrix multiplications for computing backpropagation. A single forward propagation operation consists of matrix multiplications based on the number of hidden layers (L) and number of elements in the hidden layers (N_fxK). The computational cost associated with single forward propagation is of order O((L + 1)xN_fxK xM). For N_iter iterations, the order of computations would be O(N_iterx(L + 1)xN_fxK xM). Similarly, the cost of computation for backpropagation is also of the orderO(N_iterx(L + 1)xN_fxK xM). For example, given 50 POD modes and 400 snap shots, N_f = 3 with 2 hidden layers, the number of elements stored and computational cost are O(67500), O(N_iterx1.8x10⁶) operations, respectively.

Further, the cost of computing the POD basis is of order O(N xM²), where N is the dimension of full state vector. In the case of reconstruction, the computing cost is of the order O(N xK xM).

Appendix D: Effect of bias on predictions

The results presented in the main sections of this article for MEM architectures were based on FFNNs devoid of the bias term. It is well known from machine learning literature [47] that the presence of a bias term helps with function approximation provided sufficient $\mathcal {LP}$s are used to capture the dynamics. In our studies, the FFNN had difficulty predicting the shift mode for the transient cylinder wake dynamics (Section ??) whereas the modes with zero mean were predicted accurately. Since the bias term helps in quantitative translation (shift) of the learned dynamics into higher or lower values, we expect its inclusion to improve predictions. In Fig. 25, we show predictions of the features obtained from FFNNs with N_f = 1, 3, 9 for the TR-I regime for the cylinder flow with Re = 100. In Fig. 26, we include the predictions for the TR-II cylinder flow data. In both these cases, the shift mode (third POD feature) is accurately predicted with a bias term. Fig. 27 shows the corresponding predictions with bias term for the Boussinesq flow.

Fig. 25

Times series of predicted Re = 100 POD features obtained from a 6-MEM-TS1, b 6-MEM-TS3, and c 6-MEM-TS9 compared with their respective original coefficients for TR-I region

Fig. 26

Times series of predicted Re = 100 POD features obtained from a 6-MEM-TS3 and b 6-MEM-TS9 compared with their respective original coefficients for TR-II region

Fig. 27

Comparison of the time evolution of the posteriori prediction of the 3 POD features. The different plots correspond to a MEM with N_f = 1, b MEM with N_f = 3, and c MEM with N_f = 5