Nonparametric Bayesian Inference with Kernel Mean Embedding

Abstract

Kernel methods have been successfully used in many machine learning problems with favorable performance in extracting nonlinear structure of high-dimensional data. Recently, nonparametric inference methods with positive definite kernels have been developed, employing the kernel mean expression of distributions. In this approach, the distribution of a variable is represented by the kernel mean, which is the mean element of the random feature vector defined by the kernel function, and relation among variables is expressed by covariance operators. This article gives an introduction to this new approach called kernel Bayesian inference, in which the Bayes’ rule is realized with the computation of kernel means and covariance expressions to estimate the kernel mean of posterior [11]. This approach provides a novel nonparametric way of Bayesian inference, expressing a distribution with weighted sample, and computing posterior with simple matrix calculation. As an example of problems for which this kernel Bayesian inference is applied effectively, nonparametric state-space model is discussed, in which it is assumed that the state transition and observation model are neither known nor estimable with a simple parametric model. This article gives detailed explanations on intuitions, derivations, and implementation issues of kernel Bayesian inference.

Appendix: Proof of Theorem 1.3.2

First, we show a lemma to derive a convergence rate of conditional kernel mean.

Lemma 1.5.1

Assume that the kernels are measurable and bounded. Let $N(\varepsilon ):=\mathrm {Tr}[C_{YY}(C_{YY}+\varepsilon I)^{-1}]$ and $\varepsilon _n$ be a constant such that $\varepsilon _n\rightarrow 0$ as $n\rightarrow \infty$. Then,

$$ \left\| (\widehat{C}^{(n)}_{YY}-C_{YY})(C_{YY}+\varepsilon _n I)^{-1}\right\| = O_p\left( \frac{1}{\varepsilon _n n} + \sqrt{\frac{N(\varepsilon _n)}{\varepsilon _n n}}\right) $$

and

$$ \left\| (\widehat{C}^{(n)}_{XY}-C_{XY})(C_{YY}+\varepsilon _n I)^{-1}\right\| = O_p\left( \frac{1}{\varepsilon _n n} + \sqrt{\frac{N(\varepsilon _n)}{\varepsilon _n n}}\right) $$

as $n\rightarrow \infty$.

Proof

The first result is shown in [4] (page 349). While the proof of the second one is similar, it is shown below for completeness.

Let $\xi _{yx}$ be an element in ${\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {X}}$ defined by

$$ \xi _{yx}:=\bigl \{(C_{YY}+\varepsilon _n I)^{-1} k(\cdot ,y)\bigr \}\otimes k(\cdot ,x). $$

With identification between $Hy\otimes {\mathscr {H}_\mathscr {X}}$ and the Hilbert–Schmidt operators from ${\mathscr {H}_\mathscr {X}}$ to ${\mathscr {H}_\mathscr {Y}}$,

$$ E[\xi _{YX}]=(C_{YY}+\varepsilon _n I)^{-1}C_{YX}. $$

Take $a>0$ such that $k(x,x)\le a^2$ and $k(y,y)\le a^2$. It follows from $\Vert f\otimes g\Vert =\Vert f\Vert \,\Vert g\Vert$ and $\Vert (C_{YY}+\varepsilon _n I)^{-1}\Vert \le 1/\varepsilon _n$ that

$$ \Vert \xi _{yx}\Vert = \bigl \Vert (C_{YY}+\varepsilon _n I)^{-1} k(\cdot ,y) \bigr \Vert \bigl \Vert k(\cdot ,x)\bigr \Vert \le \frac{1}{\varepsilon _n} \Vert k(\cdot ,y)\Vert \,\Vert k(\cdot ,x)\Vert \le \frac{a^2}{\varepsilon _n}, $$

and

$$\begin{aligned} E\Vert \xi _{YX}\Vert ^2&= E\bigl \Vert \{(C_{YY}+\varepsilon _n I)^{-1} k(\cdot ,Y)\}\otimes k(\cdot ,X)\bigr \Vert ^2 \\&= E\Vert k(\cdot ,X)\Vert ^2 \,\bigl \Vert (C_{YY}+\varepsilon _n I)^{-1} k(\cdot ,Y)\bigr \Vert ^2 \\&\le a^2 E\bigl \Vert (C_{YY}+\varepsilon _n I)^{-1} k(\cdot ,Y)\bigr \Vert ^2 \\&= a^2 E\bigl \langle (C_{YY}+\varepsilon _n I)^{-2}k(\cdot ,Y),k(\cdot ,Y)\bigr \rangle \\&= a^2 E \mathrm {Tr}\bigl [ (C_{YY}+\varepsilon _n I)^{-2}(k(\cdot ,Y)\otimes k(\cdot ,Y)^*)\bigr ] \\&=a^2 \mathrm {Tr}\bigl [ (C_{YY}+\varepsilon _n I)^{-2}C_{YY}\bigr ] \\&\le \frac{a^2}{\varepsilon _n }\mathrm {Tr}\bigl [ (C_{YY}+\varepsilon _n I)^{-1}C_{YY}\bigr ] = \frac{a^2}{\varepsilon _n } N(\varepsilon _n). \end{aligned}$$

Here $k(\cdot ,Y)^*$ is the dual element of $k(\cdot ,Y)$ and $k(\cdot ,Y)\otimes k(\cdot ,Y)^*$ is regarded as an operator on ${\mathscr {H}_\mathscr {Y}}$. In the last inequality, $(C_{YY}+\varepsilon _n I)^{-1}$ in the trace is replaced by its upper bound $\varepsilon _n^{-1} I$. Since $\frac{1}{n}\sum _{i=1}^n (C_{YY}+\varepsilon _n I)^{-1}\xi _{Y_i X_i} = (C_{YY}+\varepsilon _n I)^{-1}\widehat{C}^{(n)}_{YX}$, it follows from Proposition 2 in [4] that for all $n\in {\mathbb {N}}$ and $0<\eta <1$

$$\begin{aligned} \Pr \biggl ( \Bigg \Vert (C_{YY}+\varepsilon _n I)^{-1}\widehat{C}^{(n)}_{YX} - (C_{YY}+\varepsilon _n I)^{-1}&C_{YX} \Bigg \Vert \\&\ge 2\biggl (\frac{2a^2}{n\varepsilon _n} + \sqrt{\frac{a^2 N(\varepsilon _n)}{\varepsilon _n n}}\biggr )\log \frac{2}{\eta } \bigg ) \le \eta , \end{aligned}$$

which proves the assertion.$\square$

Proof of Theorem 1.3.2 First, we have

$$\begin{aligned}&\bigl \Vert \widehat{C}^{(n)}_{XY}(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) - E[k_\mathscr {X}(\cdot ,X)|Y=y_0] \bigr \Vert _{\mathscr {H}_\mathscr {X}}\nonumber \\&\le \bigl \Vert \widehat{C}^{(n)}_{XY}(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) - C_{XY}(C_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) \Vert _{\mathscr {H}_\mathscr {X}}\end{aligned}$$

(1.19)

$$\begin{aligned}&\qquad + \bigl \Vert C_{XY}(C_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)-E[k_\mathscr {X}(\cdot ,X)|Y=y_0] \bigr \Vert _{\mathscr {H}_\mathscr {X}}. \end{aligned}$$

(1.20)

Using the general formula $A^{-1}-B^{-1}=A^{-1}(B-A)B^{-1}$ for any invertible operators A, B, the first term in the right-hand side of the above inequality is upper bounded by

$$\begin{aligned}&\bigl \Vert (\widehat{C}^{(n)}_{XY}-C_{XY})(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)\bigr \Vert _{\mathscr {H}_\mathscr {X}}\nonumber \\&\qquad + \bigl \Vert C_{XY}(C_{YY}+\varepsilon _n I)^{-1}(C_{YY}-\widehat{C}^{(n)}_{YY})(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)\bigr \Vert _{\mathscr {H}_\mathscr {X}}\\ \le&\bigl \Vert (\widehat{C}^{(n)}_{XY}-C_{XY})(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}\bigr \Vert \,\bigl \Vert k_\mathscr {Y}(\cdot ,y_0)\bigr \Vert _{\mathscr {H}_\mathscr {Y}}\\&\qquad + \frac{1}{\sqrt{\varepsilon _n}}\Vert C_{XX}\Vert ^{1/2} \bigl \Vert (\widehat{C}^{(n)}_{YY}-C_{YY})(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}\bigr \Vert \, \bigl \Vert k_\mathscr {Y}(\cdot ,y_0)\bigr \Vert _{\mathscr {H}_\mathscr {Y}}, \end{aligned}$$

where in the second inequality the decomposition $C_{XY}=C_{XX}^{1/2}W_{XY}C_{YY}^{1/2}$ with some $W_{XY}:{\mathscr {H}_\mathscr {Y}}\rightarrow {\mathscr {H}_\mathscr {X}}$ ($\Vert W_{XY}\Vert \le 1$) [2] is used. It follows from Lemma 1.5.1 that

$$\begin{aligned} \bigl \Vert \widehat{C}^{(n)}_{XY}(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) - C_{XY}(C_{YY}&+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) \Vert _{\mathscr {H}_\mathscr {X}}\\&=O_p\left( \varepsilon _n^{-1/2}\left\{ \frac{1}{\varepsilon _n n}+\sqrt{\frac{N(\varepsilon _n)}{\varepsilon _n n}}\right\} \right) , \end{aligned}$$

as $n\rightarrow \infty$. It is known (Proposition 3, [4]) that, under the assumption on the decay rate of the eigenvalues, $N(\varepsilon )\le \frac{b\beta }{b-1}\varepsilon ^{-1/b}$ holds with some $\beta \ge 0$. Since $\varepsilon _n^{-3/2}n^{-1} \ll \varepsilon _n^{-1-\frac{1}{2b}}n^{-1/2}$ for $b>1$ and $n\varepsilon _n \rightarrow \infty$, we have

$$\begin{aligned} \bigl \Vert \widehat{C}^{(n)}_{XY}(\widehat{C}^{(n)}_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0) - C_{XY}(C_{YY}+\varepsilon _n I)^{-1}&k_\mathscr {Y}(\cdot ,y_0) \Vert _{\mathscr {H}_\mathscr {X}}\nonumber \\&=O_p\left( \varepsilon _n^{-1-\frac{1}{2b}}n^{-1/2}\right) , \end{aligned}$$

(1.21)

as $n\rightarrow \infty$.

For the second term of Eq. (1.19), let $\varTheta :=E[k(X,\tilde{X})|Y=\cdot ,\tilde{Y}=*]\in \mathscr {R}(C_{YY}\otimes C_{YY})$. Note that for any $\varphi \in {\mathscr {H}_\mathscr {Y}}$ we have

$$\begin{aligned} \langle C_{XY}\varphi ,&C_{XY}\varphi \rangle =E[k(X,\tilde{X})\varphi (Y)\varphi (\tilde{Y})]\\&\quad =E\bigl [ E[k(X,\tilde{X})|Y,\tilde{Y}]\varphi (Y)\varphi (\tilde{Y})\bigr ] =\langle (C_{YY}\otimes C_{YY})\varTheta ,\varphi \otimes \varphi \rangle _{{\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {Y}}}. \end{aligned}$$

Similarly,

$$\begin{aligned} \langle C_{XY}\varphi , E[k(\cdot ,X)|Y=y_0]\rangle _{\mathscr {H}_\mathscr {X}}=\langle E[k&(X,\tilde{X})|Y=y_0,\tilde{Y}=*], C_{YY} \varphi \rangle _{{\mathscr {H}_\mathscr {Y}}} \\&=\langle (I\otimes C_{YY})\varTheta ,k(\cdot ,y_0)\otimes \varphi \rangle _{{\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {Y}}}. \end{aligned}$$

It follows form these equalities with $\varphi =(C_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)$ that

$$\begin{aligned}&\bigl \Vert C_{XY}(C_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)-E[k_\mathscr {X}(\cdot ,X)|Y=y_0] \bigr \Vert _{\mathscr {H}_\mathscr {X}}^2 \\&=\bigl \langle \bigl \{ (C_{YY}+\varepsilon _n I)^{-1}C_{YY}\otimes (C_{YY}+\varepsilon _n I)^{-1}C_{YY} -I\otimes (C_{YY}+\varepsilon _n I)^{-1}C_{YY}\\&\qquad -(C_{YY}+\varepsilon _n I)^{-1}C_{YY}\otimes I + I\otimes I\bigr \}\varTheta , k_\mathscr {Y}(\cdot ,y_0)\otimes k_\mathscr {Y}(*,y_0)\bigr \rangle _{{\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {Y}}}. \end{aligned}$$

From the assumption $\varTheta \in \mathscr {R}(\mathbb {C}_{YY}\otimes C_{YY})$, there is $\varPsi \in {\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {Y}}$ such that $\varTheta = (C_{YY}\otimes C_{YY}) \varPsi$. Let $\{\phi _i\}$ be the eigenvectors of $C_{YY}$ with eigenvalues $\lambda _1\,{\ge }\, \lambda _2\,{\ge }\, \cdots 0$. Since the eigenvectors and eigenvalues of $C_{YY}\otimes C_{YY}$ are given by $\{\phi _i\otimes \phi _j\}_{ij}$ and $\lambda _i\lambda _j$, respectively, with the fact $(C_{YY}+\varepsilon _n I)^{-1}C_{YY}^2\phi _i=(\lambda _i^2/(1+\lambda _i))\phi _i$ and Parseval’s theorem we have

$$\begin{aligned}&\bigl \Vert \bigl \{(C_{YY}+\varepsilon _n I)^{-1}C_{YY}\otimes (C_{YY}+\varepsilon _n I)^{-1}C_{YY} -I\otimes (C_{YY}+\varepsilon _n I)^{-1}C_{YY}\\&\qquad -(C_{YY}+\varepsilon _n I)^{-1}C_{YY}\otimes I + I\otimes I\bigr \}\varTheta \bigr \Vert _{{\mathscr {H}_\mathscr {Y}}\otimes {\mathscr {H}_\mathscr {Y}}}^2 \\&= \sum _{i,j}\Bigl \{ \frac{\lambda _i^2}{\lambda _i+\varepsilon _n}\frac{\lambda _j^2}{\lambda _j+\varepsilon _n}- \frac{\lambda _i^2\lambda _j}{\lambda _i+\varepsilon _n}-\frac{\lambda _i\lambda _j^2}{\lambda _j+\varepsilon _n}+\lambda _i\lambda _j\Bigr \}^2 \langle \phi _i\otimes \phi _j,\varPsi \rangle _{{\mathscr {H}_\mathscr {X}}\otimes {\mathscr {H}_\mathscr {X}}}^2 \\&= \varepsilon _n^4 \sum _{i,j}\Bigl \{\frac{\lambda _i\lambda _j}{(\lambda _i+\varepsilon _n)(\lambda _j+\varepsilon _n)}\Bigr \}^2 \langle \phi _i\otimes \phi _j,\varPsi \rangle _{{\mathscr {H}_\mathscr {X}}\otimes {\mathscr {H}_\mathscr {X}}}^2 \le \varepsilon _n^4 \Vert \varPsi \Vert _{{\mathscr {H}_\mathscr {X}}\otimes {\mathscr {H}_\mathscr {X}}}^2, \end{aligned}$$

which shows

$$\begin{aligned} \bigl \Vert C_{XY}(C_{YY}+\varepsilon _n I)^{-1}k_\mathscr {Y}(\cdot ,y_0)-E[k_\mathscr {X}(\cdot ,X)|Y=y_0] \bigr \Vert _{\mathscr {H}_\mathscr {X}}= O(\varepsilon _n). \end{aligned}$$

(1.22)

By balancing Eqs. (1.21) and (1.22), the assertion is obtained with $\varepsilon _n=n^{-b/(4b+1)}$.

${}\square$