Classification with the pot–pot plot

Abstract

We propose a procedure for supervised classification that is based on potential functions. The potential of a class is defined as a kernel density estimate multiplied by the class’s prior probability. The method transforms the data to a potential–potential (pot–pot) plot, where each data point is mapped to a vector of potentials. Separation of the classes, as well as classification of new data points, is performed on this plot. For this, either the $\alpha$-procedure ($\alpha$-P) or k-nearest neighbors (k-NN) are employed. For data that are generated from continuous distributions, these classifiers prove to be strongly Bayes-consistent. The potentials depend on the kernel and its bandwidth used in the density estimate. We investigate several variants of bandwidth selection, including joint and separate pre-scaling and a bandwidth regression approach. The new method is applied to benchmark data from the literature, including simulated data sets as well as 50 sets of real data. It compares favorably to known classification methods such as LDA, QDA, max kernel density estimates, k-NN, and DD-plot classification using depth functions.

Appendix

Fig. 6

Examples of bandwidths-to-error plots (simulated data). (Color figure online)

See detailed description under Fig. 7.

Fig. 7

Examples of bandwidths-to-errors plots (real data). (Color figure online)

See the legend under Fig. 6.

The bandwidths-to-errors plots illustrate the dependencies between the selected kernel bandwidths and the classification errors for particular data sets using diverse DD-classifiers using joint (left column of panels) and separate (other panels) scaling.

In the joint scaling case the abscissa represents the logarithm of the bandwidth parameter $\log _{10}h^2\,$, and the ordinate the error rate. The rule of thumb (ROT) errors are shown with bold points.

For the separate scaling case the axes present the $\log _{10}h_i^2$ bandwidth of the first and the second classes kernels, respectively. The colors correspond to the classification errors achieved with these bandwidth combinations, where red corresponds to the highest error rate, violet to the lowest, and the colors in between are in the rainbow order. The black points represent the rule of thumb (ROT) bandwidths.

We search for the relationship between the bandwidth parameters using regressions. At first we calculate the classification error rate at 25 bandwidth points divided into five sets orthogonal to the main diagonal. Then the minimum is found over each set and a regression is computed to find the proper relation between the bandwidths. We use the found relationship to estimate error rates along the regression line, iterating one bandwidth parameter and calculating the other. For comparing the performance of this approach (pot-pot regressive separate), to joint and separate scaling; see Fig. 11.

Fig. 8

Efficiency of the methods. For the DD- and pot–pot classifiers the errors of the a-classifier are given

Fig. 9

Efficiency of the pot–pot classifiers

Fig. 10

Efficiency of DD-classifiers

Fig. 11

Efficiency of separate scaling with different bandwidth regressions over joint scaling

The minimum errors are found in Table 2 for simulated and in Table 4 for real data.

The index is the relation of the error rates of the chosen classifier and the reference classifier. Here and in the following figures we take the Bayes risk as the reference for the simulated data and LDA – for the real data. The index measures the relative efficiency of a classifier compared to the reference for a particular data set (the more efficient classifier has smaller index). For each classifier a boxplot is built that illustrates the distribution of the efficiency index over all data sets.

Table 1 Error rates (in %) of different classifiers for simulated data sets. Columns (6)–(10) and (13)–(15): $\alpha$-classifier in the DD- and pot–pot plots

Table 2 Error rates (in %) of the pot–pot classifiers for simulated data sets

The boxplots in Fig. 11 show minimal errors using the same bandwidth regressions as in the description of Fig. 6. The first and the last rows show the global minima for resp. separate and joint scaling. The classifier using joint scaling is taken as the reference.

Tables 1 and 3 show errors from using the different methods. The methods are grouped by type, and the best values within each group are marked black. The best classifiers for the particular data set are additionally underlined.

In the tables we use the following abbreviations for the data depths: HS for halfspace, Mah for Mahalanobis, Proj for projection and Spat for spatial. Dknn abbreviates the depth-based k-NN of Paindaveine and Van Bever (2015).

Table 3 Error rates (in %) of different classifiers for real data sets. Columns (7)–(11) and (14)–(16): $\alpha$-classifier in the DD- and pot–pot plots

Table 4 Error rates (in %) of the pot–pot classifiers for real data sets

Fig. 12

Calculation time of a single data point for various depth functions and potential, on the logarithmic time scale

Fig. 13

Performance of the KDE and the pot-pot classifiers in the multidimensional space