Appendix A—Proof of Theorem 2
This appendix states the proof of Theorem 2. For sake of a brief notation, a certain trajectory of the Markov chain \(\mathscr {M}\) defined by \((\theta \llbracket 0\rrbracket = \theta _0, \dots , \theta \llbracket j\rrbracket = \theta _j)\) is denoted by \(({\theta _0,\dots ,\theta _j})\) in this proof. The corresponding realization probability is given by:
$$\begin{aligned} p_{(\theta _0, \dots , \theta _j)}:= \mathbf {P}\left( \theta \llbracket 0\rrbracket = \theta _0, \dots , \theta \llbracket j\rrbracket = \theta _j\right) = \mu _{\theta _0}\!(k) \cdot \prod _{l=0}^{j-1} p_{\theta _{l+1},\theta _l} . \end{aligned}$$
(49)
Let \(\Lambda _j\) denote the set of all possible Markov state trajectories with j transitions.
By applying the system dynamic (1) j times recursively and consecutive expansion of the products, it follows for the expected cost at time step \(k+j\):
$$\begin{aligned}&\text {E}\left[ x^{\intercal }\llbracket j\rrbracket \,Q_{\theta _j} \,x\llbracket j\rrbracket \right] \nonumber \\&\quad = \text {E}\left[ \left( A_{\theta _{j-1}}\,x\llbracket j-1 \rrbracket + B_{\theta _{j-1}}\, u\llbracket j-1 \rrbracket + G_{\theta _{j-1}}\, w\llbracket j-1 \rrbracket \right) ^{\intercal }\,Q_{\theta _j}\cdot \ldots \right. \nonumber \\&\qquad \quad \left. \ldots \cdot \left( A_{\theta _{j-1}}\,x\llbracket j-1 \rrbracket + B_{\theta _{j-1}}\, u\llbracket j-1 \rrbracket + G_{\theta _{j-1}}\, w\llbracket j-1 \rrbracket \right) \right]&\\&\qquad \qquad \qquad \qquad \qquad \qquad \vdots&\nonumber \\&\quad = \sum _{\Lambda _j} p_{(\theta _0,\dots ,\theta _{j})}\bigg ( 2\sum _{l=0}^{j-1} x^{\intercal }(k)\prod _{c=0}^{j-1} A^{\intercal }_{\theta _{c}}\,Q_{\theta _j}\,\prod _{c=1}^{j-l-1} A_{\theta _{j-c}} \, B_{\theta _{l}} \,u\llbracket l\rrbracket&\nonumber \\&\qquad +2\sum _{l_1=0}^{j-1}\sum _{l_2=0}^{j-1} \bar{w}^{\intercal }\llbracket l_1\rrbracket \,G^{\intercal }_{\theta _{l_1}}\,\prod _{c=l_1+1}^{j-1} A^{\intercal }_{\theta _{c}}\,Q_{\theta _j}\,\prod _{c=1}^{j-l_2-1} A_{\theta _{j-c}} \, B_{\theta _{l_2}} \,u\llbracket l_2\rrbracket \nonumber \\&\qquad + \sum _{l_1=0}^{j-1}\sum _{l_2=0}^{j-1} u^{\intercal }\llbracket l_1\rrbracket \,B^{\intercal }_{\theta _{l_1}}\,\prod _{c=l_1+1}^{j-1} A^{\intercal }_{\theta _{c}}\,Q_{\theta _j}\,\prod _{c=1}^{j-l_2-1} A_{\theta _{j-c}} \, B_{\theta _{l_2}} \,u\llbracket l_2\rrbracket \bigg ) + \varPsi \nonumber \\&\quad = \sum _{\Lambda _{j-1}} p_{(\theta _0,\dots ,\theta _{j-1})}\Bigg ( 2\sum _{l=0}^{j-1} x^{\intercal }(k)\prod _{c=0}^{j-1} A^{\intercal }_{\theta _{c}}\left( \sum _{\theta _j = 1}^{n_{\theta }} p_{\theta _{j},\theta _{j-1}} Q_{\theta _j}\right) \prod _{c=1}^{j-l-1} A_{\theta _{j-c}} \, B_{\theta _{l}} \,u\llbracket l\rrbracket \nonumber \\&\qquad + 2\sum _{l_1=0}^{j-1}\sum _{l_2=0}^{j-1} \bar{w}^{\intercal }\llbracket l_1\rrbracket \,G^{\intercal }_{\theta _{l_1}}\,\prod _{c=l_1+1}^{j-1} A^{\intercal }_{\theta _{c}}\left( \sum _{\theta _j = 1}^{n_{\theta }} p_{\theta _{j},\theta _{j-1}} Q_{\theta _j}\right) \prod _{c=1}^{j-l_2-1} A_{\theta _{j-c}} \, B_{\theta _{l_2}} \,u\llbracket l_2\rrbracket&\nonumber \\&\qquad \left. +\!\sum _{l_1=0}^{j-1}\sum _{l_2=0}^{j-1} u^{\intercal }\llbracket l_1\rrbracket \,B^{\intercal }_{\theta _{l_1}}\!\prod _{c=l_1+1}^{j-1}\! A^{\intercal }_{\theta _{c}}\! \left( \!\sum _{\theta _j = 1}^{n_{\theta }} p_{\theta _{j},\theta _{j-1}} Q_{\theta _j}\!\right) \!\prod _{c=1}^{j-l_2-1}\! A_{\theta _{j-c}} \, B_{\theta _{l_2}} \,u\llbracket l_2\rrbracket \! \right) + \varPsi . \nonumber \end{aligned}$$
(50)
Here, the variable \(\varPsi \) contains all costs that cannot be influenced by the inputs. The sums over the cost matrices \(Q_{\theta _j}\) can be replaced by \(\mathscr {T}_{\theta _{j-1}}(Q)\), like in (23). To express the costs as a function of \(\mathbf {u}(k)\), the sums over \(l,l_1\) and \(l_2\) are reformulated as matrix multiplications:
$$\begin{aligned}&\text {E}\left( x^{\intercal }\llbracket j\rrbracket \,Q_{\theta _j} \,x\llbracket j\rrbracket \right) - \varPsi&\nonumber \\&=\!\!\!\mathop \sum _{\Lambda _{j-1}} p_{(\theta _0,\dots ,\theta _{j-1})}\!\left( 2\, x^{\intercal }(k)\, A^{\intercal }_{\theta _0}\cdot \ldots \cdot A^{\intercal }_{\theta _{j-1}} \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix}\! \begin{bmatrix} A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{1}}\cdot B_{\theta _{0}} u\llbracket 0\rrbracket \\ A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{2}}\cdot B_{\theta _{1}} u\llbracket 1\rrbracket \\ \vdots \\ B_{\theta _{j-1}} u\llbracket j-1\rrbracket \end{bmatrix}\right. \nonumber \\&\,\, + 2\begin{bmatrix} A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{1}}\cdot G_{\theta _{0}} \bar{w}\llbracket 0\rrbracket \\ A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{2}}\cdot G_{\theta _{1}} \bar{w}\llbracket 1\rrbracket \\ \vdots \\ G_{\theta _{j-1}} \bar{w}\llbracket j-1\rrbracket \end{bmatrix}^{\intercal } \!\! \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \\ \vdots&\ddots&\vdots \\ \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix} \begin{bmatrix} A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{1}}\cdot B_{\theta _{0}} u\llbracket 0\rrbracket \\ A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{2}}\cdot B_{\theta _{1}} u\llbracket 1\rrbracket \\ \vdots \\ B_{\theta _{j-1}} u\llbracket j-1\rrbracket \end{bmatrix} \nonumber \\&\,\, \left. + \begin{bmatrix} A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{1}}\cdot B_{\theta _{0}} u\llbracket 0\rrbracket \\ A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{2}}\cdot B_{\theta _{1}} u\llbracket 1\rrbracket \\ \vdots \\ B_{\theta _{j-1}} u\llbracket j-1\rrbracket \end{bmatrix}^{\intercal } \!\! \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \\ \vdots&\ddots&\vdots \\ \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix} \begin{bmatrix} A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{1}}\cdot B_{\theta _{0}} u\llbracket 0\rrbracket \\ A_{\theta _{j-1}}\cdot \ldots \cdot A_{\theta _{2}}\cdot B_{\theta _{1}} u\llbracket 1\rrbracket \\ \vdots \\ B_{\theta _{j-1}} u\llbracket j-1\rrbracket \end{bmatrix}\right) \qquad \end{aligned}$$
(51)
$$\begin{aligned}&{=\!\mathop \sum \limits _{\Lambda _{j-1}} p_{(\theta _0,\dots ,\theta _{j-1})}\left( 2\, x^{\intercal }(k)\,\,A^{\intercal }_{\theta _0}\cdot \ldots \cdot A^{\intercal }_{\theta _{j-1}}\,\, \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix} \begin{bmatrix} A_{\theta _{j-1}}&\mathbf {0}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&A_{\theta _{j-1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&B_{\theta _{j-1}} \end{bmatrix}\cdot \cdots \right. }\nonumber \\&\qquad \quad \, {\ldots \cdot \begin{bmatrix} A_{\theta _{j-2}}&\mathbf {0}&\mathbf {0}&\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0}&\mathbf {0}\\ \mathbf {0}&\mathbf {0}&A_{\theta _{j-2}}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&B_{\theta _{j-2}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&\mathbf {0}&I_{ n_{\text {u}}} \end{bmatrix} \cdot \ldots \cdot \begin{bmatrix} A_{\theta _{1}}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&B_{\theta _{1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&I_{(j-2)\cdot n_{\text {u}}} \end{bmatrix} \begin{bmatrix} B_{\theta _{0}}&\mathbf {0} \\ \mathbf {0}&I_{(j-1)\cdot n_{\text {u}}} \end{bmatrix} \begin{bmatrix} u\llbracket 0\rrbracket \\ \vdots \\ u\llbracket j-1\rrbracket \end{bmatrix}} \nonumber \\&{\qquad \quad +2\begin{bmatrix} \bar{w}^{\intercal }\llbracket 0\rrbracket&\ldots&\bar{w}^{\intercal }\llbracket j-1\rrbracket \end{bmatrix} \begin{bmatrix} G^{\intercal }_{\theta _{0}}&\mathbf {0} \\ \mathbf {0}&I_{(j-1)\cdot n_{\text {w}}} \end{bmatrix} \begin{bmatrix} A_{\theta _{1}}^{\intercal }&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&G_{\theta _{1}}^{\intercal }&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&I_{(j-2)\cdot n_{\text {w}}} \end{bmatrix} \cdot \ldots \cdot \begin{bmatrix} A^{\intercal }_{\theta _{j-1}}&\mathbf {0}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&A^{\intercal }_{\theta _{j-1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&G_{\theta _{j-1}} \end{bmatrix} \ \ \ } \nonumber \\&{\qquad \quad \cdot \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \\ \vdots&\ddots&\vdots \\ \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix}\!\! \begin{bmatrix} A_{\theta _{j-1}}&\mathbf {0}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&A_{\theta _{j-1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&B_{\theta _{j-1}} \end{bmatrix} \cdot \ldots \cdot \begin{bmatrix} B^{\intercal }_{\theta _{0}}&\mathbf {0} \\ \mathbf {0}&I_{(j-1)\cdot n_{\text {u}}} \end{bmatrix}\!\! \begin{bmatrix} u\llbracket 0\rrbracket \\ \vdots \\ u\llbracket j-1\rrbracket \end{bmatrix}} \nonumber \\&{\qquad \quad +\begin{bmatrix} u^{\intercal }\llbracket 0\rrbracket&\ldots&u^{\intercal }\llbracket j-1\rrbracket \end{bmatrix} \begin{bmatrix} B^{\intercal }_{\theta _{0}}&\mathbf {0} \\ \mathbf {0}&I_{(j-1)\cdot n_{\text {u}}} \end{bmatrix} \begin{bmatrix} A_{\theta _{1}}^{\intercal }&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&B_{\theta _{1}}^{\intercal }&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&I_{(j-2)\cdot n_{\text {u}}} \end{bmatrix} \cdot \ldots \cdot \begin{bmatrix} A^{\intercal }_{\theta _{j-1}}&\mathbf {0}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&A^{\intercal }_{\theta _{j-1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&B_{\theta _{j-1}} \end{bmatrix} \ \ \ } \nonumber \\&{\qquad \quad \left. \cdot \begin{bmatrix} \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \\ \vdots&\ddots&\vdots \\ \mathscr {T}_{\theta _{j-1}}(Q)&\ldots&\mathscr {T}_{\theta _{j-1}}(Q) \end{bmatrix}\!\! \begin{bmatrix} A_{\theta _{j-1}}&\mathbf {0}&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\ddots&\mathbf {0}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&A_{\theta _{j-1}}&\mathbf {0} \\ \mathbf {0}&\mathbf {0}&\mathbf {0}&B_{\theta _{j-1}} \end{bmatrix} \cdot \ldots \cdot \begin{bmatrix} B^{\intercal }_{\theta _{0}}&\mathbf {0} \\ \mathbf {0}&I_{(j-1)\cdot n_{\text {u}}} \end{bmatrix}\!\! \begin{bmatrix} u\llbracket 0\rrbracket \\ \vdots \\ u\llbracket j-1\rrbracket \end{bmatrix}\!\right) ,} \nonumber \end{aligned}$$
where I and \(\mathbf {0}\) denote identity and zero matrices of appropriate dimensions. With the matrices defined in (30), Eq. (51) can be written as a function of \(\mathbf {u}(k)\):
$$\begin{aligned}&\text {E}\left( x^{\intercal }\llbracket j\rrbracket \,Q_{\theta _j} \,x\llbracket j\rrbracket \right) - \varPsi \nonumber \\&= \sum _{\Lambda _{j-1}} p_{(\theta _0,\ldots \theta _{j-1})}\Big ( 2x^{\intercal }(k)\,A^{\intercal }_{\theta _0} \cdot \ldots \cdot A^{\intercal }_{\theta _{j-1}}\,\, \hat{Q}_{\text {q}_{\text {x}},\theta _{j-1}}[j]\,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]\, \mathbf {u}(k) \nonumber \\&\qquad \qquad \quad + 2\bar{\mathbf {w}}^{\intercal }(k)\, \hat{G}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{G}^{\intercal }_{\theta _{j-1}}[j]\,\, \hat{Q}_{\text {q}_{\text {x}},\theta _{j-1}}[j]\,\,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]\, \mathbf {u}(k) \nonumber \\&\qquad \qquad \quad + \mathbf {u}^{\intercal }(k) \, \hat{B}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{B}^{\intercal }_{\theta _{j-1}}[j]\,\, \hat{Q}_{\text {W},\theta _{j-1}}[j] \,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]\, \mathbf {u}(k)\Big ).&\end{aligned}$$
(52)
Thus, one obtains for the cost prediction matrices:
$$\begin{aligned} q_{\text {x}}\llbracket j\rrbracket&:= {2x^{\intercal }(k)\!\!\mathop \sum _{\Lambda _{j-1}} p_{(\theta _0,\ldots ,\theta _{j-1})}\,\,A^{\intercal }_{\theta _0} \cdot \ldots \cdot A^{\intercal }_{\theta _{j-1}}\,\, \hat{Q}_{\text {q}_{\text {x}},\theta _{j-1}}[j]\,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1],}&\end{aligned}$$
(53)
$$\begin{aligned} q_{\text {w}}\llbracket j\rrbracket&:= {2\bar{\mathbf {w}}^{\intercal }(k)\mathop \sum _{\Lambda _{j-1}} p_{(\theta _0,\ldots \theta _{j-1})}\,\, \hat{G}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{G}^{\intercal }_{\theta _{j-1}}[j]\,\, \hat{Q}_{\text {q}_{\text {w}},\theta _{j-1}}[j] \,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1],}&\end{aligned}$$
(54)
$$\begin{aligned} W'\llbracket j\rrbracket&:= {\mathop \sum _{\Lambda _{j-1}} p_{(\theta _0,\ldots \theta _{j-1})}\,\, \hat{B}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{B}^{\intercal }_{\theta _{j-1}}[j]\,\, \hat{Q}_{\text {W},\theta _{j-1}}[j] \,\, \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1].} \end{aligned}$$
(55)
These equation describe a way to calculate \(q_{\text {x}}\llbracket j\rrbracket \), \(q_{\text {w}}\llbracket j\rrbracket \), and \(W'\llbracket j\rrbracket \). However, in this form the summation over all possible Markov trajectories is still employed. The computational effort still depends exponentially on \(n_{\theta }\) and N. To reduce the computational effort, the sums are reduced to only the parts that depend on the summation variable. Thus, a nested sum is formed which can be calculated recursively:
$$\begin{aligned}&q_{\text {x}}\llbracket j\rrbracket = 2x^{\intercal }(k)\sum _{\theta _0=1}^{n_{\theta }}\ldots \sum _{\theta _{j-1}=1}^{n_{\theta }}p_{(\theta _0,\ldots ,\theta _{j-1})} A^{\intercal }_{\theta _0} \cdot \ldots \cdot A^{\intercal }_{\theta _{j-1}} \hat{Q}_{q _{x }}[j] \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]&\nonumber \\&\qquad \quad = 2x^{\intercal }\!(k)\!\sum _{\theta _0=1}^{n_{\theta }}\cdots \sum _{\theta _{j-2}=1}^{n_{\theta }}p_{(\theta _0,\ldots ,\theta _{j-2})} \cdot A^{\intercal }_{\theta _0} \cdot \ldots \cdot A^{\intercal }_{\theta _{j-2}}&\nonumber \\&\qquad \qquad \quad \cdot \Bigg (\sum _{\theta _{j-1}=1}^{n_{\theta }}p_{\theta _{j-1},\theta _{j-2}} \underbrace{A^{\intercal }_{\theta _{j-1}} \hat{Q}_{q _{x }}[j] \hat{B}_{\theta _{j-1}}[j]}_{=: \chi _{\theta _{j-1}}^{(1)}}\Bigg ) \hat{B}_{\theta _{j-2}}[j-1]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]&\nonumber \\&\qquad \quad {= 2x^{\intercal }(k) \!\! \mathop \sum _{\theta _0=1}^{n_{\theta }}\!\!\cdots \!\!\!\!\mathop \sum _{\theta _{j-2}=1}^{n_{\theta }}\!p_{(\theta _0,\ldots ,\theta _{j-2})} \cdot A^{\intercal }_{\theta _0} \cdot \ldots \cdot \underbrace{A^{\intercal }_{\theta _{j-2}} \mathscr {T}_{\theta _{j-2}}\!\left( \chi ^{(1)}\!\right) \hat{B}_{\theta _{j-2}}[j-1]}_{=: \chi _{\theta _{j-2}}^{(2)}}\cdot \ldots \cdot \hat{B}_{\theta _0}[1]}&\nonumber \\&\qquad \quad {= 2x^{\intercal }(k) \!\! \mathop \sum _{\theta _0=1}^{n_{\theta }}\!\!\cdots \!\!\!\!\mathop \sum _{\theta _{j-3}=1}^{n_{\theta }}\!p_{(\theta _0,\ldots ,\theta _{j-3})} \cdot A^{\intercal }_{\theta _0} \cdot \ldots \cdot \underbrace{A^{\intercal }_{\theta _{j-3}} \mathscr {T}_{\theta _{j-3}}\!\left( \chi ^{(2)}\!\right) \hat{B}_{\theta _{j-3}}[j-2]}_{=: \chi _{\theta _{j-3}}^{(3)}}\cdot \ldots \cdot \hat{B}_{\theta _0}[1]}&\nonumber \\&\qquad \qquad \qquad \qquad \qquad \quad \qquad \qquad \vdots&\nonumber \\&\qquad \quad = 2x^{\intercal }(k) \sum _{\theta _0=1}^{n_{\theta }}\mu _{\theta _0}\!(k) \underbrace{A^{\intercal }_{\theta _{0}} \mathscr {T}_{\theta _{0}}\left( \chi ^{(j-1)} \right) \hat{B}_{\theta _{0}}[1]}_{=: \chi _{\theta _{0}}^{(j)}} = 2x^{\intercal }(k) \sum _{\theta _0=1}^{n_{\theta }}\mu _{\theta _0}\!(k) \chi _{\theta _{0}}^{(j)}. \end{aligned}$$
(56)
These transformations correspond to the steps defined in Theorem 2. An analogous procedure for \(W'\llbracket j\rrbracket \) leads to:
$$\begin{aligned}&W'\llbracket j\rrbracket = \sum _{\theta _0=1}^{n_{\theta }}\cdots \sum _{\theta _{j-1}=1}^{n_{\theta }}p_{(\theta _0,\ldots ,\theta _{j-1})} \cdot \hat{B}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{B}^{\intercal }_{\theta _{j-1}}[j] \hat{Q}_{W }[j] \hat{B}_{\theta _{j-1}}[j]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]&\nonumber \\&\qquad \quad = \sum _{\theta _0=1}^{n_{\theta }}\cdots \sum _{\theta _{j-2}=1}^{n_{\theta }}p_{(\theta _0,\ldots ,\theta _{j-2})} \cdot \hat{B}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \hat{B}^{\intercal }_{\theta _{j-2}}[j-1]\cdot \ldots&\nonumber \\&\qquad \qquad \ldots \cdot \bigg (\sum _{\theta _{j-1}=1}^{n_{\theta }}p_{\theta _{j-1},\theta _{j-2}} \underbrace{\hat{B}^{\intercal }_{\theta _{j-1}}[j] \hat{Q}_{W }[j] \hat{B}_{\theta _{j-1}}[j]}_{=: \kappa _{\theta _{j-1}}^{(1)}}\bigg ) \hat{B}_{\theta _{j-2}}[j-1]\cdot \ldots \cdot \hat{B}_{\theta _0}[1]&\nonumber \\&\qquad \quad {= \!\! \mathop \sum _{\theta _0=1}^{n_{\theta }}\!\!\cdots \!\!\!\!\mathop \sum _{\theta _{j-2}=1}^{n_{\theta }}\!p_{(\theta _0,\ldots ,\theta _{j-2})} \cdot \hat{B}^{\intercal }_{\theta _0}[1] \cdot \ldots \cdot \underbrace{\hat{B}^{\intercal }_{\theta _{j-2}}[j-1] \mathscr {T}_{\theta _{j-2}}\left( \kappa ^{(1)}\right) \hat{B}_{\theta _{j-2}}[j-1]}_{=: \kappa _{\theta _{j-2}}^{(2)}}\cdot \ldots \cdot \hat{B}_{\theta _0}[1]}&\nonumber \\&\qquad \qquad \quad \qquad \qquad \vdots&\nonumber \\&\qquad \quad = \sum _{\theta _0=1}^{n_{\theta }}\mu _{\theta _0}\!(k) \underbrace{\hat{B}^{\intercal }_{\theta _{0}}[1] \mathscr {T}_{\theta _{0}}\left( \kappa ^{(j-1)} \right) \hat{B}_{\theta _{0}}[1]}_{=: \kappa _{\theta _{0}}^{(j)}} = \sum _{\theta _0=1}^{n_{\theta }}\mu _{\theta _0}\!(k) \kappa _{\theta _{0}}^{(j)}.&\end{aligned}$$
(57)
For \( q_{\text {w}}\llbracket j\rrbracket \), the same procedure is used to obtain a recursive algorithm. With these derivations, it is proven that the algorithm in Theorem 2 calculates the cost prediction matrices \(q_{\text {x}}\llbracket j\rrbracket \), \(q_{\text {w}}\llbracket j\rrbracket \), and \(W'\llbracket j\rrbracket \). \(\blacksquare \)
Appendix B—MPC Approach Proposed in [24]
This section gives a very brief description of the MPC approach given in [24]. The following optimization problem is solved to determine the input trajectory:
$$\begin{aligned} \min \limits _{\mathbf {u}(k)} \quad&\mathop \sum _{j = 1}^N\Big ( \bar{x}^{\intercal }\llbracket j \rrbracket \, Q_j \, \bar{x}\llbracket j \rrbracket + u^{\intercal }\llbracket j-1 \rrbracket \, R_{j-1}\, u\llbracket j -1 \rrbracket \Big ) \nonumber \\ \text {s.t.} \quad&x_{\text {min},j}\ \le \ \bar{x}\llbracket j \rrbracket \ \le \ x_{\text {max,j}}, \ \ \ \ \ u_{\text {min},j}\ \le \ u\llbracket j -1\rrbracket \ \le \ u_{\text {max},j} \ \ \ \ \ \ \forall \ j \in \{1,\dots ,N\}. \end{aligned}$$
(58)
Here, the cost matrices do not depend on the Markov state, but on the prediction step. With (35) and results from Sect. 3 (58) can be formulated as a QP [24].
