Laplace based approximate posterior inference for differential equation models

Abstract

Ordinary differential equations are arguably the most popular and useful mathematical tool for describing physical and biological processes in the real world. Often, these physical and biological processes are observed with errors, in which case the most natural way to model such data is via regression where the mean function is defined by an ordinary differential equation believed to provide an understanding of the underlying process. These regression based dynamical models are called differential equation models. Parameter inference from differential equation models poses computational challenges mainly due to the fact that analytic solutions to most differential equations are not available. In this paper, we propose an approximation method for obtaining the posterior distribution of parameters in differential equation models. The approximation is done in two steps. In the first step, the solution of a differential equation is approximated by the general one-step method which is a class of numerical numerical methods for ordinary differential equations including the Euler and the Runge-Kutta procedures; in the second step, nuisance parameters are marginalized using Laplace approximation. The proposed Laplace approximated posterior gives a computationally fast alternative to the full Bayesian computational scheme (such as Makov Chain Monte Carlo) and produces more accurate and stable estimators than the popular smoothing methods (called collocation methods) based on frequentist procedures. For a theoretical support of the proposed method, we prove that the Laplace approximated posterior converges to the actual posterior under certain conditions and analyze the relation between the order of numerical error and its Laplace approximation. The proposed method is tested on simulated data sets and compared with the other existing methods.

Appendices

Appendix 1: Computation of ${\ddot{g}}_n(x_1)$.

Recall that

$$\begin{aligned} g_n(x_1) = \frac{1}{n} \sum _{i=1}^n \Vert y_i - x_i\Vert ^2, \end{aligned}$$

where $x_i = x(t_i)$ for $i=1,2,\ldots ,n$ and $x(t) = (x_1(t), x_2(t),\ldots , x_p(t))^T$. For the following discussion, we use the following convention for vectors and matrices. Suppose we have an array of real numbers $a_{ijk}$ with indices $i=1,2,\ldots , I$, $j=1,2,\ldots , J$ and $k=1,2,\ldots , K$. Let $(a_{ijk})_{(i)}$ denote the column vector with dimension I

$$\begin{aligned} (a_{ijk})_{(i)} = (a_{1jk}, a_{2jk},\ldots , a_{Ijk})^T \end{aligned}$$

and $(a_{ijk})_{(j,k)}$ denote the matrix with dimensions $J \times K$

$$\begin{aligned} (a_{ijk})_{(j,k)}= \left[ \begin{array}{cccc} a_{i,1,1} &{}a_{i,1,2} &{} \ldots &{} a_{i,1,K}\\ a_{i,2,1} &{} a_{i,2,2} &{} \ldots &{} a_{i,2,K}\\ \ldots &{}\ldots &{}\ldots &{}\ldots \\ a_{i,J,1} &{} a_{i,J,2} &{} \ldots &{} a_{i,J,K} \end{array} \right] . \end{aligned}$$

The indices in the the subscript with parenthesis are the indices running in the vector or the matrix. The object with one running index is a column vector, while the object with two running indices a matrix where the first and the second running index are for the row and column, respectively.

Note that

$$\begin{aligned} g_n(x_1) = \frac{1}{n} \sum _{i=1}^n g_{n i}(x_1), \end{aligned}$$

where $g_{ni}(x_1) = y_i^T y_i - 2 x_i^T y_i + x_i^T x_i$. Thus, the (l, k)th element of ${\ddot{g}}_n(x_1)$ is

$$\begin{aligned} \frac{\partial ^2 g_n}{\partial x_{1l} \partial x_{1k}} = \frac{1}{n} \sum _{i=1}^n \frac{\partial ^2 g_{ni}}{\partial x_{1l} \partial x_{1k}}. \end{aligned}$$

Note

$$\begin{aligned} \frac{\partial g_{ni}}{\partial x_{1k}} = -2 \sum _{j=1}^p y_{ij} \frac{\partial x_{ij}}{\partial x_{1k}} + 2 \sum _{j=1}^p x_{ij} \frac{\partial x_{ij}}{\partial x_{1k}} \end{aligned}$$

and

$$\begin{aligned} \frac{\partial ^2 g_{ni}}{\partial x_{1l}\partial x_{1k}}= & {} -2 \sum _{j=1}^p y_{ij} \frac{\partial ^2 x_{ij}}{\partial x_{1l}\partial x_{1k}}\\&+ 2 \sum _{j=1}^p \Big (\frac{\partial x_{ij}}{\partial x_{1l}} \frac{\partial x_{ij}}{\partial x_{1k}} + x_{ij} \frac{\partial ^2 x_{ij}}{\partial x_{1l} \partial x_{1k}} \Big ). \end{aligned}$$

The above equation can be written in a matrix form

$$\begin{aligned} \Big (\frac{\partial ^2 g_{ni}}{\partial x_{1l} \partial x_{1k}}\Big )_{(l,k)}= & {} -2 \sum _{j=1}^p \Big (\frac{\partial ^2 x_{ij}}{\partial x_{1l} \partial x_{1k}}\Big )_{(l,k)} y_{ij}\\&+ \, 2 \Big (\frac{\partial x_{ij}}{\partial x_{1l}}\Big )_{(l,j)} \left( \frac{\partial x_{ij}}{\partial x_{1k}}\right) _{(j,k)}\\&+\, 2 \sum _{j=1}^p \Big (\frac{\partial ^2 x_{ij}}{\partial x_{1l} \partial x_{1k}}\Big )_{(l,k)} x_{ij}\\= & {} 2 \Big (\frac{\partial x_{ij}}{\partial x_{1l}}\Big )_{(l,j)} \Big (\frac{\partial x_{ij}}{\partial x_{1k}}\Big )_{(j,k)}\\&+ \, 2 \sum _{j=1}^p \Big (\frac{\partial ^2 x_{ij}}{\partial x_{1l} \partial x_{1k}}\Big )_{(l,k)} (x_{ij}-y_{ij}). \end{aligned}$$

Thus,

$$\begin{aligned} {\ddot{g}}_n(x_1)= & {} \frac{2}{n} \sum _{i=1}^n \Bigg (\Big (\frac{\partial x_{ij}}{\partial x_{1l}}\Big )_{(l,j)} \Big (\frac{\partial x_{ij}}{\partial x_{1k}}\Big )_{(j,k)}\\&+ \sum _{j=1}^p \Big (\frac{\partial ^2 x_{ij}}{\partial x_{1l} \partial x_{1k}}\Big )_{(l,k)} (x_{ij}-y_{ij})\Bigg ). \end{aligned}$$

The derivatives of $x_i$ with respect to $x_1$ can be computed by using the sensitivity equation for ODE. See Hooker (2009). Let

$$\begin{aligned} z_{jl}(t) = \frac{\partial x_j(t)}{\partial x_{1l}} \text { ~or~} Z(t) = \Big (\frac{\partial x_j(t)}{\partial x_{1l}} \Big )_{(j, l)} , ~~ j,l =1,\ldots , p \end{aligned}$$

be the sensitivity of the state $x_{j}$ with respect to the initial value $x_{1l}$. The sensitivity equation is given by

$$\begin{aligned} {\dot{z}}_{jl}(t) = \frac{\partial }{\partial t}\frac{\partial x_j(t)}{\partial x_{1l}}= & {} \frac{\partial }{\partial x_{1l}} {\dot{x}}_j(t)\\= & {} \sum _{u=1}^p \frac{\partial f_j(x, t ; \theta )}{\partial x_u(t)} \frac{\partial x_u(t)}{\partial x_{1l}}\\= & {} \sum _{u=1}^p \frac{\partial f_j(x, t ; \theta )}{\partial x_u(t)} z_{ul}(t), \end{aligned}$$

or in matrix notation,

$$\begin{aligned} {\dot{Z}}(t) = \Big (\frac{\partial f_j(x, t ; \theta )}{\partial x_u(t)} \Big )_{(j, u)} \cdot Z(t) \end{aligned}$$

(14)

with an initial condition $Z(t_1) = I_p$. For given $\theta$ and t, the coefficient $\partial f_j(x, t ;\theta ) / \partial x_u(t)$ is calculated easily. It is a linear ODE problem whose initial condition is known. We can solve (14) using some numerical methods such as Runge-Kutta method. $\partial ^2 x_{ij} / (\partial x_{1l} \partial x_{1k})$ can be computed similarly.

Appendix 2: Proof of Theorem 1

Proof

The results of Tierney and Kadane (1986) and Azevedo-Filho and Shachter (1994) assume several regularity conditions such as the existence of a unique global maximum as well as the existence of higher order derivatives (up to sixth order) of the likelihood function. In particular, our methods for approximating the ODE model work only under the assumption of a unique maximum of the likelihood function. Thus, we assume that the likelihood surface does not include any ridges (that is, continuum regions with equal values of the global maximum).

Using the result in Tierney and Kadane (1986) and Azevedo-Filho and Shachter (1994), we have

$$\begin{aligned} \pi _m(\theta , \tau ^2 \mid \mathbf{y}_n)= & {} c_m^{-1} \int L_m(\theta , \tau ^2, x_1) \pi (\theta ,\tau ^2,x_1) dx_1\\&\times (1+ O(n^{-3/2})), \end{aligned}$$

where $c_m = \int L_m(\theta ,\tau ^2,x_1)\pi (\theta ,\tau ^2,x_1) dx_1 d\theta d\tau ^2$. Note the full likelihood $L(\theta , \tau ^2, x_1)$ is

$$\begin{aligned} L(\theta , \tau ^2, x_1) \propto e^{-\frac{\tau ^2}{2}ng_n(x_1)} \times (\tau ^2)^{np/2} , \end{aligned}$$

and $L_m(\theta , \tau ^2, x_1)$ is the corresponding term with $g_n$ replaced by $g_n^m$. If $L_m(\theta ,\tau ^2,x_1)$ converges to $L(\theta ,\tau ^2,x_1)$ as $m \rightarrow \infty$ for all $\theta \in \varTheta , \tau ^2 >0, x_1 \in {\mathbb {R}}^p$ and $\mathbf{y}_n$, by the dominated convergence theorem, $c_m \longrightarrow c$ as $m \rightarrow \infty$. Thus,

$$\begin{aligned} \lim _{m\rightarrow \infty } \pi _m(\theta ,\tau ^2 \mid \mathbf{y}_n)= & {} c^{-1} \int L(\theta ,\tau ^2,x_1)\pi (\theta ,\tau ^2,x_1)dx_1\\&\times (1+ O(n^{-3/2}))\\= & {} \pi (\theta ,\tau ^2 \mid \mathbf{y}_n)\times (1+ O(n^{-3/2})) \end{aligned}$$

which is the desired result.

To complete the proof, we need to prove $L_m(\theta ,\tau ^2,x_1) \longrightarrow L(\theta ,\tau ^2,x_1)$ as $m\rightarrow \infty$, and it suffices to prove $ng_n^m(x_1) \longrightarrow ng_n(x_1)$ as $m\rightarrow \infty$. Since we assume the Lipschitz continuity of f, the ODE has a unique solution with initial condition $x(t_1) = x_1$. Assumptions A1 and A3 implies

$$\begin{aligned} \sup _{x, t} \Vert \frac{d^K}{dt^K} f(x,t;\theta ) \Vert =: B < \infty \end{aligned}$$

for some constants $B >0$. The local errors of the Kth order numerical method are given by

$$\begin{aligned}&\Vert x(t_i) - x(t_{i-1}) - h \phi (x_{i-1} , t_{i-1} ; \theta ) \Vert \\&\quad \le B' h^{K+1}, ~ i=2,\ldots , n \end{aligned}$$

for some $B'>0$, which depends only on $\sup _{ t} \Vert d^K f(x,t;\theta )/(dt^K) \Vert$ $\le B$ (Palais and Palais 2009). Thus, the local errors are uniformly bounded. It implies the global errors uniformly bounded

$$\begin{aligned} \Vert x_i - x^h_i\Vert \le C h^K \end{aligned}$$

for some constant $C>0$.

Thus,

$$\begin{aligned} | ng_n(x_1) - ng_n^m(x_1)|&= \Big | \sum _{i=1}^n \Vert y_i - x_i \Vert ^2 - \sum _{i=1}^n\Vert y_i - x_i^m\Vert ^2 \Big | \nonumber \\&= \sum _{i=1}^n \Big (\Vert y_i - x_i \Vert + \Vert y_i - x_i^m \Vert \Big ) \nonumber \\&\quad \times \Big | \Vert y_i - x_i \Vert - \Vert y_i - x_i^m \Vert \Big | \nonumber \\&\le \sum _{i=1}^n \Big (2 \Vert y_i - x_i \Vert + \Vert x_i -x_i^m\Vert \Big ) \Vert x_i-x_i^m\Vert \nonumber \\&\le \sum _{i=1}^n \Big (2C_y + 2 C_x + \Vert x_i -x_i^m\Vert \Big ) \Vert x_i -x_i^m\Vert \nonumber \\&\le \sum _{i=1}^n \Big (2C_y + 2 C_x + C \Big (\frac{h}{m}\Big )^K \Big ) C \Big (\frac{h}{m}\Big )^K \nonumber \\&\asymp n \Big (\frac{h}{m} \Big )^K, \text { as } m \longrightarrow \infty , \end{aligned}$$

(15)

where $\sup _{t \in [T_0,T_1]}\Vert y(t)\Vert < C_y < \infty$, $\sup _{t \in [T_0,T_1]}\Vert x(t)\Vert < C_x <\infty$. This completes the proof. $\square$

Appendix 3: Proof of Theorem 2

Proof

If $\alpha > 5/(2K)$, as n goes to infinity, $n(h/m)^K = O(n^{1-\alpha K}) = O(n^{-3/2})$ and it converges to to zero. Under $A1-A3$, we have shown in the proof of Theorem 1 that $| ng_n(x_1) - ng_n^m(x_1)| = O(n (h/m)^K)$. For fixed $\tau ^2>0$,

$$\begin{aligned} e^{-\frac{\tau ^2}{2}ng_n^m(x_1)}= & {} e^{-\frac{\tau ^2}{2}\left[ ng_n(x_1)+ng_n^m(x_1)-ng_n(x_1)\right] }\\= & {} e^{-\frac{\tau ^2}{2}ng_n(x_1)} \times e^{-\frac{\tau ^2}{2}\left[ ng_n^m(x_1)-ng_n(x_1)\right] }\\= & {} e^{-\frac{\tau ^2}{2}ng_n(x_1)} \times e^{-\frac{\tau ^2}{2}O\Big (n \Big (\frac{h}{m}\Big )^K\Big )}\\= & {} e^{-\frac{\tau ^2}{2}ng_n(x_1)} \times \Big (1+ O\Big (n \Big (\frac{h}{m}\Big )^K\Big ) \Big ) \end{aligned}$$

because $e^{x} = 1+ O(x)$ for sufficiently small x. It implies

$$\begin{aligned} \pi _m(\theta , \tau ^2 \mid \mathbf{y}_n)\propto & {} \int L_m(\theta , \tau ^2, x_1) \pi (\theta ,\tau ^2,x_1) dx_1\\&\times (1+ O(n^{-3/2})) \nonumber \\= & {} \int L(\theta , \tau ^2, x_1) \pi (\theta ,\tau ^2,x_1) dx_1\\&\times (1+ O(n^{-3/2}))\times \Big (1+ O\Big (n \Big (\frac{h}{m}\Big )^K \Big ) \Big )~~~\nonumber \\\propto & {} \pi (\theta ,\tau ^2\mid \mathbf{y}_n) \times (1+ O(n^{-3/2}))\\&\times \Big (1+ O\Big (n \Big (\frac{h}{m}\Big )^K \Big ) \Big ) \nonumber \end{aligned}$$

for sufficiently large n. If $\alpha > 5/(2K)$, i.e., $n(h/m)^K \le n^{-3/2}$, $(1+ O(n^{-3/2})) \times (1+ O(n (h/m)^K))$ is $(1+ O(n^{-3/2}))$.$\square$