Multi-view local linear KNN classification: theoretical and experimental studies on image classification

Abstract

When handling special multi-view scenarios where data from each view keep the same features, we may perhaps encounter two serious challenges: (1) samples from different views of the same class are less similar than those from the same view but different class, which sometimes happen in local way in both training and/or testing phases; (2) training an explicit prediction model becomes unreliable and even infeasible for test samples in multi-view scenarios. In this study, we prefer the philosophy of the k nearest neighbor method (KNN) to circumvent the second challenge. Without an explicit prediction model trained directly from the above multi-view data, a new multi-view local linear k nearest neighbor method (MV-LLKNN) is then developed to circumvent the two challenges so as to predict the label of each test sample. MV-LLKNN has its two reliable assumptions. One is the theoretically and experimentally provable assumption that any test sample can be well approximated by a linear combination of its neighbors in the multi-view training dataset. The other assumes that these neighbors should demonstrate their clustering property according to certain commonality-based similarity measure between the multi-view test sample and these multi-view neighbors so as to avoid the first challenge. MV-LLKNN can realize its effective prediction for a test multi-view sample by cheaply using both on-hand fast iterative shrinkage thresholding algorithm (FISTA) and KNN. Our theoretical analysis and experimental results about real multi-view face datasets indicate the effectiveness of MV-LLKNN.

Appendices

Appendix 1

Proof of Theorem 1.

Proof:

If ${\mathbf{w}}^{k}$ is the representation obtained by MV-LLKNN on the multi-view data, then $J\left( {{\mathbf{w}}^{K} } \right) \le J\left( {\mathbf{0}} \right)$,

$$\begin{aligned} \left\| {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\mathbf{w}}^{k} } \right\|_{2}^{2} + \alpha \left\| {{\mathbf{w}}^{k} } \right\|_{1} + \beta \left\| {{\mathbf{w}}^{k} - \eta {\mathbf{s}}} \right\|_{2}^{2} \hfill \\ \le \left\| {{\mathbf{x}}^{k} } \right\|^{2} + \beta \left\| {\eta {\mathbf{s}}} \right\|^{2} \times \hfill \\ \end{aligned}$$

(39)

Since

$$\left\| {{\mathbf{w}}^{k} - \beta {\mathbf{s}}} \right\|_{2} \le \left( {\frac{1}{\beta } + \left\| {\eta {\mathbf{s}}} \right\|^{2} } \right)^{{\frac{1}{2}}}$$

(40)

we know that $\left\| {{\mathbf{w}}^{k} - \eta {\mathbf{s}}} \right\|_{2}$ is actually bounded by a small positive constant. That is to say, ${\mathbf{w}}^{k} \approx \eta {\mathbf{s}} + {\mathbf{const}}$.

The transformations in Eqs. (12) and (13) guarantees that each term in ${\mathbf{w}}^{k}$ satisfies that $0 \le w_{i}^{k} \le 1$ and $\sum\nolimits_{i = 1}^{m} {w_{i}^{k} } = 1$. It is worth noting that the transformations do not affect the classification results. Based on the similarity measure between the test sample and the training sample, MV-LLKNN₊ and MV-LLKNN_* are designed.

For MV-LLKNN₊, it can be approximated as follows:

$$\begin{aligned} c^{ * } & = \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \\ & \approx \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\eta s_{i} + \text{const}} } \\ & \propto \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\left( {1 - \frac{{\sum\limits_{l = 1}^{K} {\gamma_{l} \left\| {{\mathbf{x}}^{l} - {\mathbf{a}}_{i}^{l} } \right\|^{2} } }}{{2\sigma^{2} }}} \right)} } \\ \end{aligned}$$

(41)

In this study, let us consider the Epanechnikov kernel [61] : $h\left( u \right) = \frac{3}{4}\left( {1 - u^{2} } \right)$. Then

$$\begin{aligned} c^{ * } = \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \hfill \\ \, \propto \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\sum\limits_{l = 1}^{K} {h\left( {\frac{{{\mathbf{x}}^{l} - {\mathbf{a}}_{i}^{l} }}{\sigma }} \right)} } } \hfill \\ \end{aligned}$$

(42)

where $\sum\nolimits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\sum\nolimits_{l = 1}^{K} {h\left( {\frac{{{\mathbf{x}}^{l} - {\mathbf{a}}_{i}^{l} }}{\sigma }} \right)} }$ becomes the kernel density estimation of the conditional probability $p\left( {{\mathbf{x}}^{k} \left| c \right.} \right)\left( {k = 1,2, \ldots ,K} \right)$.

Since each view is assumed to be classified separately and independently. Therefore, if the prior probability $p\left( c \right)$ is the same for all the classes, then

$$\begin{aligned} c^{ * } & = \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \\ & \approx \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {p\left( {{\mathbf{x}}^{k} \left| c \right.} \right)} \\ & \propto \mathop {\arg \hbox{max} }\limits_{c} \sum\limits_{k = 1}^{K} {p\left( {c\left| {{\mathbf{x}}^{k} } \right.} \right)} \left( {{\text{i}} . {\text{e}} . , {\text{ Bayes classifier}}} \right) \\ \end{aligned}$$

(43)

Similarly, we have the following derivations for MV-LLKNN_*:

$$\begin{aligned} c^{ * } & = \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \\ & \approx \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\eta s_{i} + \text{const}} } \\ & \propto \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\prod\limits_{l = 1}^{K} {\left( {1 - \frac{{\left\| {{\mathbf{x}}^{l} - {\mathbf{a}}_{i}^{l} } \right\|^{2} }}{{\sigma^{2} }}} \right)^{{\gamma_{l} }} } } } \\ \end{aligned}$$

(44)

Then, we also consider another Epanechnikov kernel

$$\begin{aligned} c^{ * } = \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \hfill \\ \, \propto \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {\prod\limits_{l = 1}^{K} {h\left( {\frac{{{\mathbf{x}}^{l} - {\mathbf{a}}_{i}^{l} }}{\sigma }} \right)} } } \hfill \\ \end{aligned}$$

(45)

Therefore, if the prior probability $p\left( c \right)$ is the same for all the classes, then

$$\begin{aligned} c^{ * } & = \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {\sum\limits_{{{\mathbf{a}}_{i}^{k} \in {\mathbf{A}}_{c}^{k} }} {w_{i}^{k} } } \\ & \approx \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {p\left( {{\mathbf{x}}^{k} \left| c \right.} \right)} \\ & \propto \mathop {\arg \hbox{max} }\limits_{c} \prod\limits_{k = 1}^{K} {p\left( {c\left| {{\mathbf{x}}^{k} } \right.} \right)} \left( {{\text{i}} . {\text{e}} . , {\text{ Bayes classifier}}} \right) \\ \end{aligned}$$

(46)

In summary, from the perspective of density estimation, MV-LLKNN₊ and MV-LLKNN_* approximate the Bayes decision rule for minimum error and the approximation error mainly comes from ${\mathbf{w}}^{k} \approx \eta {\mathbf{s}} + {\mathbf{const}}$ and the kernel density estimation error.

Appendix 2

Proof of Theorem 2.

Proof

Let us observe Eq. (3) which is equivalent to

$$\begin{aligned} \mathop {arg\hbox{min} }\limits_{{\left[ {w_{1}^{k} ,w_{2}^{k} , \ldots ,w_{m}^{k} } \right]^{T} }} J\left( {\left[ {w_{1}^{k} ,w_{2}^{k} , \ldots ,w_{m}^{k} } \right]^{T} } \right) & = \sum\limits_{k = 1}^{K} {\left\| {{\mathbf{x}}^{k} - \sum\limits_{i = 1}^{m} {w_{i}^{k} {\mathbf{a}}_{i}^{k} } } \right\|_{2}^{2} } \\ & \quad + \alpha \sum\limits_{k = 1}^{K} {\sum\limits_{i = 1}^{m} {\left| {w_{i}^{k} } \right|} } + \beta \sum\limits_{k = 1}^{K} {\sum\limits_{i = 1}^{m} {\left( {w_{i}^{k} - \eta s_{i} } \right)^{2} } } \\ \end{aligned}$$

(47)

Let ${\tilde{\mathbf{w}}}^{k} \varvec{ = }\left[ {\tilde{w}_{1}^{k} ,\tilde{w}_{2}^{k} , \ldots ,\tilde{w}_{m}^{k} } \right]^{T}$ is the representation obtained by MV-LLKNN on the multi-view data, then we take the derivatives with respective $\tilde{w}_{i}^{k}$ and $\tilde{w}_{j}^{k}$:

$$\begin{aligned} \frac{\partial J}{{\partial \tilde{w}_{i}^{k} }} & = - 2\left( {{\mathbf{a}}_{i}^{k} } \right)^{T} \left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) + \alpha sign\left( {\tilde{w}_{i}^{k} } \right) \\ & \quad + 2\beta \left( {\tilde{w}_{i}^{k * } - \eta s_{i} } \right) \\ \end{aligned}$$

(48)

$$\begin{aligned} \frac{\partial J}{{\partial \tilde{w}_{j}^{k} }} & = - 2\left( {{\mathbf{a}}_{j}^{k} } \right)^{T} \left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) + \alpha sign\left( {\tilde{w}_{j}^{k} } \right) \\ & \quad + 2\beta \left( {\tilde{w}_{j}^{k} - \eta s_{j} } \right) \\ \end{aligned}$$

(49)

Let the above two derivatives to be zero. Since ${\text{sign}}(\tilde{w}_{i}^{k} ) = {\text{sign}}(\tilde{w}_{j}^{k} )$, then, $\frac{\partial J}{{\partial \tilde{w}_{i}^{k} }} - \frac{\partial J}{{\partial \tilde{w}_{j}^{k} }}$ is:

$$\begin{aligned} \beta \left( {\tilde{w}_{i}^{k} - \tilde{w}_{j}^{k} } \right) & = \left( {\left( {{\mathbf{a}}_{i}^{k} } \right)^{T} - \left( {{\mathbf{a}}_{j}^{k} } \right)^{T} } \right)\left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) \\ & \quad + \beta \eta \left( {s_{i} - s_{j} } \right) \\ \end{aligned}$$

(50)

By $J\left( {{\tilde{\mathbf{w}}}^{k} } \right) \le J\left( {\mathbf{0}} \right), \, \left\| {{\mathbf{x}}^{k} } \right\|^{2} = 1$, we can get:

$$\begin{aligned} \left| {\tilde{w}_{i}^{k} - \tilde{w}_{j}^{k} } \right| & = \frac{1}{\beta }\left| {\left( {\left( {{\mathbf{a}}_{i}^{k} } \right)^{T} - \left( {{\mathbf{a}}_{j}^{k} } \right)^{T} } \right)\left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) + \beta \eta \left( {s_{i} - s_{j} } \right)} \right| \\ & \le \frac{1}{\beta }\left| {\left( {\left( {{\mathbf{a}}_{i}^{k} } \right)^{T} - \left( {{\mathbf{a}}_{j}^{k} } \right)^{T} } \right)\left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right)} \right| + \eta \left| {s_{i} - s_{j} } \right| \\ & \le \frac{1}{\beta }\left( {\left( {{\mathbf{a}}_{i}^{k} } \right)^{T} - \left( {{\mathbf{a}}_{j}^{k} } \right)^{T} } \right)\left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) + \eta \left| {s_{i} - s_{j} } \right| \\ & = \frac{1}{\beta }\sqrt {2\left( {1 - \delta_{k} } \right)} \left( {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right) + \eta \left| {s_{i} - s_{j} } \right| \\ \end{aligned}$$

(51)

Then, since

$$\left\| {{\mathbf{x}}^{k} - {\mathbf{A}}^{k} {\tilde{\mathbf{w}}}^{k} } \right\|_{2}^{2} \le \left\| {{\mathbf{x}}^{k} } \right\|^{2} + \beta \left\| {\eta {\mathbf{s}}} \right\|^{2}$$

(52)

so we have

$$\begin{aligned} \left| {\tilde{w}_{i}^{k} - \tilde{w}_{j}^{k} } \right| & \le \frac{1}{\beta }\sqrt {2\left( {1 - \delta_{k} } \right)\left( {\left\| {{\mathbf{x}}^{k} } \right\|^{2} + \beta \eta^{2} \left\| {\mathbf{s}} \right\|^{2} } \right)} + \eta \left| {s_{i} - s_{j} } \right| \\ & = \frac{G}{\beta }\sqrt {2\left( {1 - \delta_{k} } \right)} + \eta \left| {s_{i} - s_{j} } \right| \\ \end{aligned}$$

(53)

where $G = \sqrt {1 + \beta \eta^{2} \left\| {\mathbf{s}} \right\|^{2} }$.