Incorporating Nonparametric Knowledge to the Least Mean Square Adaptive Filter

Abstract

In the framework of the maximum a posteriori estimation, the present study proposes the nonparametric probabilistic least mean square (NPLMS) adaptive filter for the estimation of an unknown parameter vector from noisy data. The NPLMS combines parameter space and signal space by combining the prior knowledge of the probability distribution of the process with the evidence existing in the signal. Taking advantage of kernel density estimation to estimate the prior distribution, the NPLMS is robust against the Gaussian and non-Gaussian noises. To achieve this, some of the intermediate estimations are buffered and then used to estimate the prior distribution. Despite the bias-compensated algorithms, there is no need to estimate the input noise variance. Theoretical analysis of the NPLMS is derived. In addition, a variable step-size version of NPLMS is provided to reduce the steady-state error. Simulation results in the system identification and prediction show the acceptable performance of the NPLMS in the noisy stationary and non-stationary environments against the bias-compensated and normalized LMS algorithms.

Appendix

1.1 A

Considering Eqs. (1) and (3), one can rewrite (20) as

$$\begin{aligned} \varvec{w}_{n+1} =\,&\varvec{w}_{n} +\alpha R_{n} \left( w_{o} -\varvec{w}_{n} \right) { +}\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)-\alpha \varvec{w}_{n} +\alpha \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } \varvec{w}_{i} \end{aligned}$$

(51)

where $R_{n} =\varvec{x}_{n}^\mathrm{T} \varvec{x}_{n}$ is the correlation matrix. Subtracting both sides of (51) from $w_{o}$, we have

$$\begin{aligned} \begin{aligned} w_{o} -\varvec{w}_{n+1}&=w_{o} -\varvec{w}_{n} -\alpha R_{n} \left( w_{o} -\varvec{w}_{n} \right) -\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)\\&\quad +\alpha \varvec{w}_{n} -\alpha \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } \varvec{w}_{i} \pm \alpha w_{o} \pm \alpha \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } w_{o}. \end{aligned} \end{aligned}$$

(52)

Defining $\tilde{\varvec{w}}_{n} =w_{o} -\varvec{w}_{n}$, (52) can be rewritten as

$$\begin{aligned} \begin{aligned} \tilde{\varvec{w}}_{n+1}&=\tilde{\varvec{w}}_{n} -\alpha R_{n} \tilde{\varvec{w}}_{n} -\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)-\alpha \tilde{\varvec{w}}_{n} \\&\quad +\alpha \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } \tilde{\varvec{w}}_{i}+\alpha w_{o} -\alpha w_{o} \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } \end{aligned} \end{aligned}$$

(53)

where $\tilde{\varvec{w}}_{i} =w_{o} -\varvec{w}_{i}$. Since $\sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } =1$, (53) reduces to

$$\begin{aligned} \tilde{\varvec{w}}_{n+1} =\left( I-\alpha R_{n} -\alpha I\right) \tilde{\varvec{w}}_{n} -\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)+\alpha \sum _{i=1}^{K}\mu _{i,\varvec{w}_{n} } \tilde{\varvec{w}}_{i} . \end{aligned}$$

(54)

On the other hand, it is clear that $\left\| \varvec{w}_{n} -\varvec{w}_{i} \right\| =\left\| w_{o} -w_{o} +\varvec{w}_{n} -\varvec{w}_{i} \right\| =\left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\|$. Using the reverse triangle inequality, we have

$$\begin{aligned} \left\| \tilde{\varvec{w}}_{i} \right\| -\left\| \tilde{\varvec{w}}_{n} \right\| \le \left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\| \end{aligned}$$

(55)

Therefore,

$$\begin{aligned} \begin{array}{l} {\left( \left\| \tilde{\varvec{w}}_{i} \right\| -\left\| \tilde{\varvec{w}}_{n} \right\| \right) ^{2} \le \left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\| ^{2} } \\ {\left\| \tilde{\varvec{w}}_{i} \right\| ^{2} +\left\| \tilde{\varvec{w}}_{n} \right\| ^{2} -2\left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \le \left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\| ^{2} } \end{array} \end{aligned}$$

(56)

and

$$\begin{aligned} \begin{aligned}&\underbrace{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\| ^{2} \right) }_{a_1}\\&\quad \le \underbrace{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{n} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }_{a_2}. \end{aligned} \end{aligned}$$

(57)

The summation over (57) is

$$\begin{aligned} \begin{aligned}&\underbrace{\sum _{i=1}^{K}\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} -\tilde{\varvec{w}}_{n} \right\| ^{2} \right) }_{a_3}\\&\quad \le \underbrace{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{n} \right\| ^{2} \right) \sum _{i=1}^{K}\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }_{a_4}. \end{aligned} \end{aligned}$$

(58)

Since $a_1 \le a_3$ and $a_2 \le a_4$, one can write

$$\begin{aligned} \begin{aligned} a_1a_4 \le a_3a_4\\ a_2a_3 \le a_3a_4. \end{aligned} \end{aligned}$$

(59)

Subtracting first inequality of (59) from the second one and vise versa leads to conclude that $a_1a_4=a_2a_3$. In other words, $\frac{a_1}{a_3}=\frac{a_2}{a_4}$. Therefore, considering (57), (58), and inequalities (59)$, \mu _{i,\varvec{w}_{i} }$ can be rewritten as

$$\begin{aligned} \begin{aligned} \mu _{i,\tilde{\varvec{w}}_{i} }&=\frac{\exp \left( -\frac{1}{2} \left\| \varvec{w}-\varvec{w}_{i} \right\| ^{2} \right) }{\sum _{j=1}^{K}\exp \left( -\frac{1}{2} \left\| \varvec{w}-\varvec{w}_{i} \right\| ^{2} \right) } \\&=\frac{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{n} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{n} \right\| ^{2} \right) \sum _{j=1}^{K}\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{j} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{j} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) } \\&=\frac{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }{\sum _{j=1}^{K}\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{j} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{j} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }. \end{aligned} \end{aligned}$$

(60)

Therefore, one can write (54) as

$$\begin{aligned} \tilde{\varvec{w}}_{n+1} =\left( I-\alpha R_{n} -\alpha I\right) \tilde{\varvec{w}}_{n} -\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)+\alpha \sum _{i=1}^{K}\mu _{i,\tilde{\varvec{w}}_{i} } \tilde{\varvec{w}}_{i} . \end{aligned}$$

(61)

Remark 2

If we assume $\tilde{\varvec{w}}_{i} \simeq \tilde{\varvec{w}}_{n}$, then $\tilde{\varvec{w}}_{n+1}$ reduces to $\tilde{\varvec{w}}_{n+1} =\left( I-\alpha R_{n} \right) \tilde{\varvec{w}}_{n} -\alpha \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)$.

1.2 B

Lemma 1

[2]: Assume $x_{i} ,i=1,\ldots ,m$ are possibly dependent identically distributed random variables with zero mean and finite pth absolute moment assumed, without loss of generality, to be equal to 1. Then

$$\begin{aligned} \begin{aligned} \mathbb E\left\{ \left\| x\right\| _{p} \right\}&=\mathbb E\left\{ \left[ \sum _{i=1}^{m}\left| x_{i} \right| ^{p} \right] ^{1/p} \right\} \\&\le \left\{ \mathbb E\left[ \sum _{i=1}^{m}\left| x_{i} \right| ^{p} \right] \right\} ^{1/p}=m^{1/p}. \end{aligned} \end{aligned}$$

(62)

Considering (60), one can write

$$\begin{aligned} \begin{aligned} {\mathbb E}\left\{ \sum _{i=1}^{K}\mu _{i,\tilde{\varvec{w}}_{i} } \tilde{\varvec{w}}_{i} \right\} = {{\mathbb E}}\left\{ \sum _{i=1}^{K}\frac{\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }{\sum _{j=1}^{K}\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{j} \right\| ^{2} \right) \exp \left( \left\| \tilde{\varvec{w}}_{j} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) } \tilde{\varvec{w}}_{i} \right\} . \end{aligned} \end{aligned}$$

(63)

Using linear approximation of Maclaurin series of $\exp \left( -\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right)$ and $\exp \left( \left\| \tilde{\varvec{w}}_{j} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right)$, one can approximate $\mu _{i,\tilde{\varvec{w}}_{i} }$ as

$$\begin{aligned} \mu _{i,\tilde{\varvec{w}}_{i} } \approx \frac{\left( 1-\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \left( 1-\left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }{\sum _{j=1}^{K}\left( 1-\frac{1}{2} \left\| \tilde{\varvec{w}}_{j} \right\| ^{2} \right) \left( 1-\left\| \tilde{\varvec{w}}_{j} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) } . \end{aligned}$$

(64)

Using Lemma 1,

$$\begin{aligned} \begin{aligned}&{\mathbb E}\left\{ \frac{\left( 1-\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \left( 1-\left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) }{\sum _{i=1}^{K}\left( 1-\frac{1}{2} \left\| \tilde{\varvec{w}}_{i} \right\| ^{2} \right) \left( 1-\left\| \tilde{\varvec{w}}_{i} \right\| \left\| \tilde{\varvec{w}}_{n} \right\| \right) } \right\} \\&\quad \le \frac{\left( 1-\frac{1}{2} m\right) \left( 1-m\right) }{\sum _{i=1}^{K}\left( 1-\frac{1}{2} m\right) \left( 1-m\right) } =\frac{1}{K} \end{aligned} \end{aligned}$$

(65)

therefore (63) changes to

$$\begin{aligned} {\mathbb E}\left\{ \sum _{i=1}^{K}\mu _{i,\tilde{\varvec{w}}_{i} } \tilde{\varvec{w}}_{i} \right\} \le \frac{1}{K} \sum _{i=1}^{K}\left\{ {{\mathbb E}}\left\{ \tilde{\varvec{w}}_{i} \right\} \right\} . \end{aligned}$$

(66)

Taking expectation of both sides of (61), considering (66) and $\mathrm{\mathrm{E}}\left( \varvec{x}_{n}^\mathrm{T} \varvec{v}(n)\right) =0$, because samples $\varvec{x}_{n}^\mathrm{T}$ and noise $\varvec{v}(n)$ are independent, one has

$$\begin{aligned} \begin{aligned} {\mathbb E}\left( \tilde{\varvec{w}}_{n+1} \right)&={{\mathbb E}}\left( I-\alpha R_{n} -\alpha \right) {{\mathbb E}}\left( \tilde{\varvec{w}}_{n} \right) +\alpha {{\mathbb E}}\left( \sum _{i=1}^{K}\mu _{i,\tilde{\varvec{w}}_{i} } \tilde{\varvec{w}}_{i} \right) \\&\le \left( I-\alpha R-\alpha \right) {{\mathbb E}}\left( \tilde{\varvec{w}}_{n} \right) +\alpha \frac{1}{K} \sum _{i=1}^{K}{\mathbb E}\left( \tilde{\varvec{w}}_{i} \right) . \end{aligned} \end{aligned}$$

(67)

If assumed ${\mathbb E}\left( \tilde{\varvec{w}}_{i} \right) =\beta _{i} {{\mathbb E}}\left( \tilde{\varvec{w}}_{n} \right)$, then (67) can be rewritten as

$$\begin{aligned} {\mathbb E}\left( \tilde{\varvec{w}}_{n+1} \right) \le \left( I-\alpha R-\alpha +\alpha \frac{1}{K} \sum _{i=1}^{K}\beta _{i} \right) {{\mathbb E}}\left( \tilde{\varvec{w}}_{n} \right) . \end{aligned}$$

(68)

Without loss of generality, it is assumed that

$$\begin{aligned} {\mathbb E}\left( \tilde{\varvec{w}}_{n+1} \right) \approx \left( I-\alpha R-\alpha +\alpha \frac{1}{K} \sum _{i=1}^{K}\beta _{i} \right) {{\mathbb E}}\left( \tilde{\varvec{w}}_{n} \right) . \end{aligned}$$

(69)