Robust Naive Bayes Combination of Multiple Classifications

Abstract

When we face new complex classification tasks, since it is difficult to design a good feature set for observed raw data, we often obtain an unsatisfactorily biased classifier. Namely, the trained classifier can only successfully classify certain classes of samples owing to its poor feature set. To tackle the problem, we propose a robust naive Bayes combination scheme in which we effectively combine classifier predictions that we obtained from different classifiers and/or different feature sets. Since we assume that the multiple classifier predictions are given, any type of classifier and any feature set are available in our scheme. In our combination scheme each prediction is regarded as an independent realization of a categorical random variable (i.e., class label) and a naive Bayes model is trained by using a set of the predictions within a supervised learning framework. The key feature of our scheme is the introduction of a class-specific variable selection mechanism to avoid overfitting to poor classifier predictions. We demonstrate the practical benefit of our simple combination scheme with both synthetic and real data sets, and show that it can achieve much higher classification accuracy than conventional ensemble classifiers.

Appendix

Derivations of Eqs. (3) and (4) are as follows. Equation (3) can be derived as follows:

$$\begin{aligned} P(\mathrm C |\mathrm R ;\mathbf{\alpha },\mathbf{\beta })&= \int P(\mathrm C | \mathrm R ,\mathbf{\phi },\varTheta ) p(\mathbf{\phi };\mathbf{\alpha })p(\varTheta ;\mathbf{\beta }) d\mathbf{\phi }d\varTheta \\&= \left( \prod _{k=1}^K \prod _{j=1}^J \frac{\varGamma (\sum _{l} \beta _{k,j,l})}{\prod _{l'} \varGamma (\beta _{k,j,l'})} \int \prod _{l=1}^K (\theta _{k,j,l})^{r_{k,j} n_{k,j,l}+\beta _{k,j,l}-1} d\theta _{k,j,l}\right) \\&\quad \times \, \frac{\varGamma (\sum _{l'}\alpha _{l'})}{\prod _{l'}\varGamma (\alpha _{l'})} \int \prod _{l=1}^K \phi _l^{\sum _k \sum _j (1-r_{k,j})n_{k,j,l}+\alpha _l-1}d\phi _l \\&= \left( \frac{\varGamma (\alpha _{\bullet })}{\prod _l\varGamma (\alpha _l)} \frac{\prod _l\varGamma (\sum _{k,j}\delta (r_{k,j},0)n_{k,j,l}+\alpha _l)}{\varGamma (\sum _{k,j}\delta (r_{k,j},0)N_k+\alpha _{\bullet })}\right) \\&\quad \times \, \left( \prod _{k=1}^K \prod _{j=1}^J \frac{\varGamma (\beta _{k,j,\bullet })}{\prod _l\varGamma (\beta _{k,j,l})} \frac{\prod _l \varGamma (r_{k,j}n_{k,j,l}+\beta _{k,j,l})}{\varGamma (r_{k,j}N_k+\beta _{j,\bullet })}\right) . \end{aligned}$$

Here, $B(x,y)$ is the beta function. We used the definition $B(x,y)=\varGamma (x)\varGamma (y)/ \varGamma (x+y)$. In a similar manner, Eq. (4) can be derived as follows:

$$\begin{aligned} P(\mathrm R ;a,b)&= \prod _{k=1}^K \prod _{j=1}^J \int P(r_{k,j};\lambda )p(\lambda ;a,b)d\lambda \\&= \int \lambda ^{\sum _k \sum _j r_{k,j} + a - 1} (1-\lambda )^{\sum _k \sum _j (1-r_{k,j})+b-1} d\lambda / B(a,b)\\&= \frac{B\left( \sum _k \sum _j r_{k,j}+a, \sum _k \sum _j (1-r_{k,j})+b\right) }{B(a,b)}\\&= \frac{\varGamma (\sum _k \sum _j r_{k,j}+a) \varGamma (\sum _k \sum _j (1-r_{k,j})+b)}{\varGamma (KJ+a+b)} \cdot \frac{\varGamma (a,b)}{\varGamma (a)\varGamma (b)} \\&= \frac{\varGamma (\sum _k \sum _j r_{k,j}+a) \varGamma (\sum _k \sum _j (1-r_{k,j})+b)\varGamma (a+b)}{\varGamma (KJ+a+b)\varGamma (a)\varGamma (b)}. \end{aligned}$$

Here, we used another definition of the beta function: $B(s,t)=\int x^{s-1}(1-x)^{t-1}dx$.