Random time step probabilistic methods for uncertainty quantification in chaotic and geometric numerical integration

Abstract

A novel probabilistic numerical method for quantifying the uncertainty induced by the time integration of ordinary differential equations (ODEs) is introduced. Departing from the classical strategy to randomise ODE solvers by adding a random forcing term, we show that a probability measure over the numerical solution of ODEs can be obtained by introducing suitable random time steps in a classical time integrator. This intrinsic randomisation allows for the conservation of geometric properties of the underlying deterministic integrator such as mass conservation, symplecticity or conservation of first integrals. Weak and mean square convergence analysis is derived. We also analyse the convergence of the Monte Carlo estimator for the proposed random time step method and show that the measure obtained with repeated sampling converges in the mean square sense independently of the number of samples. Numerical examples including chaotic Hamiltonian systems, chemical reactions and Bayesian inferential problems illustrate the accuracy, robustness and versatility of our probabilistic numerical method.

Appendix

1.1 A modified stochastic differential equation

In Remark 5, we claim the existence of a modified stochastic differential equation (SDE) whose solution is well approximated by the RTS-RK method. Let us denote by ${\widetilde{f}}$ the function defining the modified equation corresponding to the numerical flow $\Psi _h$ truncated after l terms, i.e.

$$\begin{aligned} {\widetilde{f}}(y) = f(y) + h^q f_{q+1}(y) + h^{q+1} f_{q+2}(y) + \cdots + h^l f_{l+1}(y). \end{aligned}$$

Details about the construction of such a function can be found in Sect. 7.2. In particular, analyticity of the function f is needed for a rigorous backward error analysis to hold. Therefore, we will refer in this section to Assumption 5 (see Sect. 7.2). For the additive noise method presented in Conrad et al. (2017), the authors consider the SDE

$$\begin{aligned} \,\mathrm {d}Y = {\widetilde{f}}(Y) \,\mathrm {d}t + \sqrt{Q h^{2p}} \,\mathrm {d}W, \end{aligned}$$

(51)

where W is a d-dimensional standard Brownian motion. It is possible to show (Conrad et al. 2017, Theorem 2.4) that the solution of (51) satisfies

$$\begin{aligned} {|}{{\mathbb {E}}}\big (\Phi (Y_N) - \Phi (Y(T)) \mid Y_0 = y \big ){|} \le Ch^{2p}, \end{aligned}$$

where $T = Nh$ and $Y_N$ is the numerical solution given by the additive noise method after N steps. Here, we present a similar construction for the RTS-RK method. In particular, let us consider the modified SDE

$$\begin{aligned} \,\mathrm {d}{\widetilde{Y}}= & {} \Big ({\widetilde{f}}({\widetilde{Y}}) + \frac{1}{2}Ch^{2p}\partial _{tt}\Psi _h({\widetilde{Y}})\Big ) \,\mathrm {d}t \nonumber \\&\quad +\, \sqrt{Ch^{2p}\partial _t \Psi _h({\widetilde{Y}})\partial _t \Psi _h({\widetilde{Y}})^\top } \,\mathrm {d}W, \end{aligned}$$

(52)

where C is given in Assumption 1(iii). Let us denote by ${\widetilde{{\mathcal {L}}}}$ the generator of (52), which can be written explicitly as

$$\begin{aligned} {\widetilde{{\mathcal {L}}}} = \Big ({\widetilde{f}} + \frac{1}{2}Ch^{2p}\partial _{tt}\Psi _h\Big ) \cdot \nabla + \frac{1}{2}Ch^{2p}\partial _t \Psi _h \partial _t \Psi _h^\top : \nabla ^2, \end{aligned}$$

and, adopting the semi-group notation, it satisfies

$$\begin{aligned} {{\mathbb {E}}}\big (\Phi ({\widetilde{Y}}(h))\mid {\widetilde{Y}}(0) = y\big ) = \hbox {e}^{h{\widetilde{{\mathcal {L}}}}}\Phi (y). \end{aligned}$$

In the following lemma, we consider the error over one step between the numerical solution given by the RTS-RK method and the solution of (52) in the weak sense. The proof is inspired by the calculations presented in Conrad et al. (2017, Section 2.4).

Lemma 9

Under the assumptions of Lemma 1 and if Assumption 5 holds, then

$$\begin{aligned} {|}{{\mathbb {E}}}\big (\Phi (Y_1)- \Phi ({\widetilde{Y}}(h))\mid Y_0 = y\big ){|} \le C h^{2p+1}, \end{aligned}$$

where C is a positive constant independent of h and of y, ${\widetilde{Y}}$ is the solution of (52) and $Y_1$ is the numerical solution given by the RTS-RK method after one step.

Proof

Let us consider the modified ODE

$$\begin{aligned} {\widehat{y}}'(t) = {\widetilde{f}}({\widehat{y}}), \end{aligned}$$

(53)

and denote its flow as ${\widehat{\varphi }}_t$. The generator ${\widehat{{\mathcal {L}}}} = {\widetilde{f}} \cdot \nabla$ satisfies, adopting the semi-group notation,

$$\begin{aligned} \Phi ({\widehat{\varphi }}_h(y)) = \hbox {e}^{h{\widehat{{\mathcal {L}}}}}\Phi (y). \end{aligned}$$

We can now compute the distance between the solution to (52) and (53) as

$$\begin{aligned} \begin{aligned}&\hbox {e}^{h {\widetilde{{\mathcal {L}}}}}\Phi (y)- \hbox {e}^{h {\widehat{{\mathcal {L}}}}}\Phi (y) = \hbox {e}^{h{\widetilde{f}} \cdot \nabla }\\&\quad \Big (\hbox {e}^{\frac{1}{2}Ch^{{2p+1}}\partial _{tt}\Psi _h \cdot \nabla + \frac{1}{2}Ch^{{2p+1}}\partial _t \Psi _h \partial _t \Psi _h^\top : \nabla ^2} - I\Big ) \Phi (y)\\&\quad = \big (1 + {\mathcal {O}}(h)\big )\Big (\frac{1}{2}Ch^{{2p+1}}\partial _{tt}\Psi _h \cdot \nabla + \frac{1}{2}Ch^{{2p+1}}\partial _t \Psi _h \partial _t \Psi _h^\top : \nabla ^2 \\&\qquad + {\mathcal {O}}\big (h^{{4p+1}}\big )\Big )\Phi (y) \\&\quad = \frac{1}{2}Ch^{2p+1}\partial _{tt}\Psi _h \cdot \nabla \Phi (y) + \frac{1}{2}Ch^{2p+1}\partial _t \Psi _h \partial _t \Psi _h^\top : \nabla ^2 \Phi (y) \\&\qquad + {\mathcal {O}}\big (h^{{4p+1}}\big ). \end{aligned} \end{aligned}$$

Let us recall that equation (10) gives

$$\begin{aligned} \begin{aligned}&\hbox {e}^{h{\mathcal {L}}_h}\Phi (y) - \Phi (\Psi _h(y)) = \frac{1}{2} Ch^{2p+1}\partial _{tt}\Psi _h(y) \cdot \nabla \Phi (y)\\&\quad +\frac{1}{2}Ch^{2p+1}\partial _t \Psi _h(y) \partial _t \Psi _h(y)^\top :\nabla ^2\Phi (y) + {\mathcal {O}}(h^{2p+1}), \end{aligned} \end{aligned}$$

which implies that

$$\begin{aligned} \hbox {e}^{h{\widetilde{{\mathcal {L}}}}}\Phi (y)- \hbox {e}^{h{\mathcal {L}}_h}\Phi (y) = \hbox {e}^{h{\widehat{{\mathcal {L}}}}}\Phi (y) - \Phi (\Psi _h(y)) + {\mathcal {O}}(h^{2p+1}). \end{aligned}$$

Now, the theory of backward error analysis (see Sect. 7.2 or e.g. Hairer et al. 2006, Chapter IX) guarantees that

$$\begin{aligned} \hbox {e}^{h{\widehat{{\mathcal {L}}}}}\Phi (y) - \Phi (\Psi _h(y)) = {\mathcal {O}}(h^{q+l+2}). \end{aligned}$$

Choosing $l = 2p - q - 1$, we have therefore

$$\begin{aligned} \hbox {e}^{h{\widetilde{{\mathcal {L}}}}}\Phi (y)- \hbox {e}^{h{\mathcal {L}}_h}\Phi (y) = {\mathcal {O}}(h^{2p+1}), \end{aligned}$$

which is the desired result. $\square$

The error can be then propagated to final time as in Theorem 1, as presented in the following theorem.

Theorem 7

Under the assumptions of Lemma 9 and Theorem 1, and if there exists a constant $L > 0$ independent of h such that for all $\Phi \in {\mathcal {C}}^\infty _b({\mathbb {R}}^d, {\mathbb {R}})$

$$\begin{aligned} \sup _{u\in {\mathbb {R}}^d} {|}\hbox {e}^{h{\widetilde{{\mathcal {L}}}}}\Phi (u){|} \le (1 + Lh)\sup _{u\in {\mathbb {R}}^d}{|}\Phi (u){|}, \end{aligned}$$

then it holds

$$\begin{aligned} {|}{{\mathbb {E}}}\big (\Phi (Y_N)- \Phi ({\widetilde{Y}}(T))\mid Y_0 = y\big ){|} \le Ch^{2p}, \end{aligned}$$

where $T = Nh$ and C is a positive constant independent of h and of y, ${\widetilde{Y}}$ is the solution of (52) and $Y_N$ is the numerical solution given by the RTS-RK method after N steps.

Proof

The proof follows by replacing ${\mathcal {L}}$ with $\widetilde{{\mathcal {L}}}$ and Lemma 1 with Lemma 9 in the proof of Theorem 1. $\square$

1.2 Proof of Lemma 6

In the following, we denote by $\llbracket a, b \rrbracket$ the interval $\llbracket a, b \rrbracket = [a, b]$ if $a < b$ and $\llbracket a, b \rrbracket = [b, a]$ if $a \ge b$. Let us first consider $r \ge 2$ and the function $\gamma _r(x) = x^r \hbox {e}^{-r\kappa /x}$, whose first derivative is given by

$$\begin{aligned} \gamma _r'(x) = rx^{r-2}(x + \kappa ) \hbox {e}^{-r\kappa /x}. \end{aligned}$$

Under Assumption 6, we have that $H_j \le Mh$ almost surely, and hence for any $t \in \llbracket h, H_j\rrbracket$

$$\begin{aligned} {|}\gamma _r'(t){|} \le r (Mh)^{r-2} (Mh + \kappa )\hbox {e}^{-r\kappa /(Mh)}, \end{aligned}$$

where we exploited that $\hbox {e}^{-r\kappa /x}$ is a growing function of x. The fundamental theorem of calculus gives

$$\begin{aligned} \begin{aligned} {|}\gamma _r(H_j){|}&= \Big |\gamma _r(h) + \int _{h}^{H_j} \gamma _r'(t) \,\mathrm {d}t \,\Big |\\&\le \gamma _r(h) + r (Mh)^{r-2} (Mh + \kappa )\hbox {e}^{-r\kappa /(Mh)} {|}H_j - h{|}, \quad \text {almost surely}. \end{aligned} \end{aligned}$$

Taking expectation on both sides and since by (33) it holds ${|}\eta _j{|}^r \le C\gamma _r(H_j)$, we obtain

$$\begin{aligned} {{\mathbb {E}}}{|}\eta _j{|}^r \le C\left( \gamma _r(h) + r M^{r-2} h^{p+r-3/2} (Mh + \kappa )\hbox {e}^{-r\kappa /(Mh)}\right) , \end{aligned}$$

which proves the desired inequality. This is because Assumption 6 and Assumption 1(ii) imply that $M \ge 1$, and because Mh can be bounded by M. Let us now consider $r = 1$. In this case, we have for $t \in \llbracket h, H_j\rrbracket$

$$\begin{aligned} {|}\gamma _1'(t){|} \le (mh)^{-1} (Mh + \kappa )\hbox {e}^{-\kappa /(Mh)}, \quad \text {almost surely.} \end{aligned}$$

Hence, we apply the same reasoning as above and obtain almost surely

$$\begin{aligned} {|}\gamma _1(H_j){|} \le \gamma _1(h) + (mh)^{-1} (Mh + \kappa ) \hbox {e}^{-\kappa /(Mh)} {|}H_j - h{|}, \end{aligned}$$

which implies the desired result by proceeding as above. $\square$

1.3 Proof of Lemma 7

We first expand the square as

$$\begin{aligned} \begin{aligned}&\left( \sum _{j=0}^{n-1} \left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) \right) ^2 = \sum _{j=0}^{n-1} \left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) ^2 \\&\quad + 2 \sum _{j=1}^{n-1} \sum _{i=0}^{j-1}\left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) \left( \sum _{k=q}^{N-1} a_{ik} + b_i\right) . \end{aligned} \end{aligned}$$

(54)

Then, we expand the square in the first sum and obtain

$$\begin{aligned} \begin{aligned}&\left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) ^2 = \left( \sum _{k=q}^{N-1} a_{jk}\right) ^2 + b_j^2 + 2 b_j \sum _{k=q}^{N-1} a_{jk}\\&\qquad = \sum _{k=q}^{N-1} a_{jk}^2 + 2\sum _{k=q+1}^{N-1}\sum _{l=q}^{k-1} a_{jk} a_{jl} + b_j^2 + 2 b_j \sum _{k=q}^{N-1} a_{jk}\\&\qquad = a_{jq}^2 + \sum _{k=q+1}^{N-1} a_{jk}^2 + 2\sum _{k=q+1}^{N-1}\sum _{l=q}^{k-1} a_{jk} a_{jl} + b_j^2 + 2 b_j \sum _{k=q}^{N-1} a_{jk}. \end{aligned} \end{aligned}$$

(55)

We then rewrite the term appearing in the double sum in (54) as

$$\begin{aligned} \begin{aligned}&\left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) \left( \sum _{k=q}^{N-1} a_{ik} + b_i\right) = a_{jq}a_{iq} + \sum _{k=q}^{N-1}\sum _{\begin{array}{c} l=q \\ l+k > 2q \end{array}}^{N-1} a_{jk}a_{il} \\&\qquad + b_j \sum _{k=q}^{N-1} a_{ik} + b_i \sum _{k=q}^{N-1} a_{jk} + b_i b_j. \end{aligned} \end{aligned}$$

(56)

Substituting expressions (55) and (56) in (54), we finally get

$$\begin{aligned}&\left( \sum _{j=0}^{n-1} \left( \sum _{k=q}^{N-1} a_{jk} + b_j\right) \right) ^2 = \sum _{j=0}^{n-1} a_{jq}^2\\&\qquad + 2 \sum _{j=1}^{n-1} \sum _{i=0}^{j-1} a_{jq}a_{iq} + R(a) + S(a, b), \end{aligned}$$

where the remainder R(a) can be written as $R = R_1 + R_2 + R_3$ where

$$\begin{aligned} \begin{aligned}&R_1(a) = \sum _{j=0}^{n-1}\sum _{k=q+1}^{N-1} a_{jk}^2, \\&R_2(a) = 2 \sum _{j=0}^{n-1}\sum _{k=q+1}^{N-1}\sum _{l=q}^{k-1} a_{jk} a_{jl},\\&R_3(a) = 2 \sum _{j=1}^{n-1} \sum _{i=0}^{j-1} \sum _{k=q}^{N-1}\sum _{\begin{array}{c} l=q \\ l+k > 2q \end{array}}^{N-1} a_{jk}a_{il}, \end{aligned} \end{aligned}$$

and the remainder S(a, b) can be written as $S = S_1 + S_2 + S_3 + S_4$ where

$$\begin{aligned}&S_1(a, b) = \sum _{j=0}^{n-1} b_j^2, \\&S_2(a, b) = 2\sum _{j=1}^{n-1} \sum _{i=0}^{j-1} b_i b_j, \\&S_3(a, b) = 2\sum _{j=1}^{n-1} \sum _{k=q}^{N-1} b_j a_{jk}, \\&S_4(a, b) = 2\sum _{j=1}^{n-1}\sum _{i=0}^{n-1}\left( b_j \sum _{k=q}^{N-1} a_{ik} + b_i \sum _{k=q}^{N-1} a_{jk}\right) .\\ \end{aligned}$$

which proves the desired result. $\square$

1.4 Proof of Lemma 8

In the following, all the constants are independent of h and n, but can depend on N and q. Moreover, since $h < 1$, we often apply $h^r \le h^s$ for $r \ge s$. We first notice that, under Assumption 2 and Assumption 5, we get for all $j = 0, \ldots , n-1$ and $k = q, \ldots , N-1$

$$\begin{aligned} \begin{aligned} {|}\Delta _{j,k}{|}&= {|}Q_{k+1}(Y_j) - Q_{k+1}(Y_{j+1}){|}\\&\le C {{\Vert }\Psi _0(Y_j) - \Psi _{H_j}(Y_j){\Vert }}\\&\le C_\Delta {|}H_j{|}, \end{aligned} \end{aligned}$$

(57)

almost surely and where $C_\Delta$ is independent of h. Above, we exploited that $Q_{k+1}$ is Lipschitz continuous for all $k=q, \ldots , N+1$ due to Assumption 5. Let us now consider $R(\Delta )$. Due to (57) and to Assumption 6, we have

$$\begin{aligned} \begin{aligned} {{\mathbb {E}}}(H_j^k - h^k)^2 \Delta _{j,k}^2&\le {C_\Delta ^2}{{\mathbb {E}}}(H_j^{k+1} - H_j h^k)^2\\&= {C_\Delta ^2}\big (h^{2(k+1)} + C_{2(k+1)}h^{2p+2(k+1)-1} \\&+ h^{2(k+1)} + C_2 h^{2p+2k+1} \\&\quad - 2h^{2k+2} - 2C_{k+2}h^{2p+2k+1}\big )\\&= {C_\Delta ^2}\big ((C_{2(k+1)} + C_2 - 2C_{k+2})h^{2p+2k+1}\big )\\&\le C h^{2p+2k+1}, \end{aligned} \end{aligned}$$

(58)

where $C > 0$ is a positive constant. Now, since $k \ge q+1$, we get

$$\begin{aligned} {{\mathbb {E}}}(H_j^k - h^k)^2 \Delta _{j,k}^2 \le C h^{2(p+q+1)}. \end{aligned}$$

Hence, for $R_1(\Delta )$ there exists a constant ${\widetilde{C}}_1$ such that

$$\begin{aligned} {{\mathbb {E}}}R_1(\Delta ) \le {\widetilde{C}}_1 n h^{2(p+q+1)}. \end{aligned}$$

We now proceed to the second remainder $R_2(\Delta )$. Applying the Cauchy–Schwarz inequality and (58), we get

$$\begin{aligned} \begin{aligned}&{{\mathbb {E}}}\left( (H_j^k - h^k)\Delta _{j,k}(H_j^l - h^l) \Delta _{j,l}\right) \le \Big ({{\mathbb {E}}}\big ((H_j^k - h^k)^2\Delta _{j,k}^2\Big )^{1/2}\\&\qquad \Big ({{\mathbb {E}}}\big ((H_j^l - h^l)^2\Delta _{j,l}^2\Big )^{1/2}\\&\quad \le C h^{2p+k+l+1}, \end{aligned} \end{aligned}$$

where $C > 0$ is a positive constant. Now, since in the definition of $R_2(a)$ in (57) we have $k \ge q+1$ and $l \ge q$, we have here $k+l \ge 2q+1$. Therefore, there exists a constant ${\widetilde{C}}_2$ such that

$$\begin{aligned} {{\mathbb {E}}}R_2(\Delta ) \le {\widetilde{C}}_2 n h^{2(p+q+1)}. \end{aligned}$$

We now consider the term $R_3(\Delta )$. Since $H_i$ and $H_j$ are independent for $i \ne j$, we have

$$\begin{aligned} \begin{aligned}&{{\mathbb {E}}}\big ((H_j^k - h^k)\Delta _{j,k}(H_i^l - h^l) \Delta _{i,l}\big ) \\&\quad = {{\mathbb {E}}}(H_j^k - h^k)\Delta _{j,k} {{\mathbb {E}}}(H_i^l - h^l) \Delta _{i,l}. \end{aligned} \end{aligned}$$

Computing the two factors singularly, we have due to (57) and to Assumption 6

$$\begin{aligned} \begin{aligned} {{\mathbb {E}}}(H_j^k - h^k)\Delta _{j,k}&\le C_\Delta {{\mathbb {E}}}(H_j^{k+1} - H_jh^k) \\&= C_\Delta C_{k+1} h^{2p+k}, \end{aligned} \end{aligned}$$

(59)

and analogously for ${{\mathbb {E}}}(H_i^l - h^l) \Delta _{i,l}$. Then, since $k+l \ge 2q+1$

$$\begin{aligned}&{{\mathbb {E}}}\big ((H_j^k - h^k)\Delta _{j,k}(H_i^l - h^l) \Delta _{i,l}\big )\nonumber \\&\quad \le C_\Delta ^2 C_{k+1} C_{l+1} h^{2(2p+q+1/2)}. \end{aligned}$$

(60)

Hence, we have for a constant ${\widetilde{C}}_3 > 0$

$$\begin{aligned} {{\mathbb {E}}}R_3(\Delta ) \le {\widetilde{C}}_3 n^2 h^{2(2p+q+1/2)}. \end{aligned}$$

Finally, replacing $t_n = nh$, we can write for a constant $C > 0$

$$\begin{aligned} \begin{aligned} {{\mathbb {E}}}R(\Delta )&\le ({\widetilde{C}}_1 + {\widetilde{C}}_2) n h^{2(p+q+1)} + {\widetilde{C}}_3 n^2 h^{2(2p+q+1/2)}\\&= ({\widetilde{C}}_1 + {\widetilde{C}}_2) t_n h^{2(p + q + 1/2)} + {\widetilde{C}}_3 t_n^2 h^{2(2p + q - 1/2)}. \end{aligned} \end{aligned}$$

Let us now consider $S(\Delta , \eta )$. First, we notice that under the assumption $p \ge 3/2$ we have for any $r \ge 1$, $\min \{r, p+r-3/2\} = r$, and therefore Lemma 6 simplifies to

$$\begin{aligned} {{\mathbb {E}}}{|}\eta _j{|}^r \le Ch^r \hbox {e}^{-r\kappa /(Mh)}. \end{aligned}$$

We first consider $S_1(\Delta , \eta )$. Applying Lemma 6 with $r = 2$, we obtain for a constant ${\widehat{C}}_1 > 0$

$$\begin{aligned} {{\mathbb {E}}}S_1(\Delta , \eta ) \le {\widehat{C}}_1 n h^2 \hbox {e}^{-2\kappa /(Mh)}. \end{aligned}$$

For the second term $S_2(\Delta , \eta )$, we have by (33) that ${|}\eta _i{|}\le CH^i \hbox {e}^{-\kappa /H_i}$ and $\eta _j\le CH^j \hbox {e}^{-\kappa /H_j}$ almost surely. These two bounds are independent for $i \ne j$, and therefore, applying Lemma 6 with $r = 1$, we have for a constant ${\widehat{C}}_2 > 0$

$$\begin{aligned} {{\mathbb {E}}}S_2(\Delta , \eta ) \le {\widehat{C}}_2 n^2 h^2 \hbox {e}^{-2\kappa /(Mh)}. \end{aligned}$$

We now consider the third remainder $S_3(\Delta , \eta )$. Applying the Cauchy–Schwarz inequality, we obtain

$$\begin{aligned} {{\mathbb {E}}}\eta _j (H_j^k - h^k) \Delta _{j,k} \le ({{\mathbb {E}}}\eta _j^2)^{1/2} ({{\mathbb {E}}}(H_j^k - h^k)^2 \Delta _{j,k}^2)^{1/2}. \end{aligned}$$

Applying Lemma 6 with $r = 2$ to the first factor and (58) to the second, we get

$$\begin{aligned} \begin{aligned} {{\mathbb {E}}}\eta _j (H_j^k - h^k) \Delta _{j,k}&\le C h \hbox {e}^{-\kappa /(Mh)}h^{p+k+1/2}\\&= C h^{p+k+3/2}\hbox {e}^{-\kappa /(Mh)} \end{aligned} \end{aligned}$$

Now, since $k \ge q$, we have for a constant ${\widehat{C}}_3 > 0$

$$\begin{aligned} {{\mathbb {E}}}S_3(\Delta , \eta ) \le {\widehat{C}}_3 n h^{p+q+3/2}\hbox {e}^{-\kappa /(Mh)}. \end{aligned}$$

Finally, we consider the last term $S_4(\Delta , \eta )$. Since by (33) it holds ${|}\eta _j{|} \le CH_j \hbox {e}^{-\kappa /H_j}$ almost surely, and this bound is independent of $H_i$ for $i \ne j$, applying (59) and Lemma 6 we have

$$\begin{aligned} \begin{aligned} {{\mathbb {E}}}\eta _j(H_i^k - h^k)\Delta _{i,k}&= {{\mathbb {E}}}\eta _j {{\mathbb {E}}}(H_i^k - h^k)\Delta _{i,k}\\&\le Ch \hbox {e}^{-\kappa /(Mh)}h^{2p+k}, \end{aligned} \end{aligned}$$

which, since $k \ge q$, implies that there exists a constant ${\widehat{C}}_4 > 0$ such that

$$\begin{aligned} {{\mathbb {E}}}S_4(\Delta , \eta ) \le {\widehat{C}}_4 n^2 h^{2p+q+1}\hbox {e}^{-\kappa /(Mh)}. \end{aligned}$$

Finally, replacing $t_n = nh$, we can write

$$\begin{aligned} \begin{aligned}&{{\mathbb {E}}}S(\Delta , \eta ) \le ({\widehat{C}}_1 n h^2 + {\widehat{C}}_2 n^2 h^2) \hbox {e}^{-2\kappa /(Mh)} \\&\qquad + ({\widehat{C}}_3 n h^{p+q+3/2} + {\widehat{C}}_4 n^2 h^{2p+q+1})\hbox {e}^{-\kappa /(Mh)}\\&\quad = ({\widehat{C}}_1 t_n h + {\widehat{C}}_2 t_n^2) \hbox {e}^{-2\kappa /(Mh)} \\&\qquad + ({\widehat{C}}_3 t_n h^{p+q+1/2} + {\widehat{C}}_4 t_n^2 h^{2p+q-1})\hbox {e}^{-\kappa /(Mh)}, \end{aligned} \end{aligned}$$

which completes the proof. $\square$