A Theoretical Analysis of the Peaking Phenomenon in Classification
- 10 Downloads
In this work, we analytically study the peaking phenomenon in the context of linear discriminant analysis in the multivariate Gaussian model under the assumption of a common known covariance matrix. The focus is finite-sample setting where the sample size and observation dimension are comparable. Therefore, in order to study the phenomenon in such a setting, we use an asymptotic technique whereby the number of sample points is kept comparable in magnitude to the dimensionality of observations. The analysis provides a more thorough picture of the phenomenon. In particular, the analysis shows that as long as the Relative Cumulative Efficacy of an additional Feature set (RCEF) is greater (less) than the size of this set, the expected error of the classifier constructed using these additional features will be less (greater) than the expected error of the classifier constructed without them. Our result highlights underlying factors of the peaking phenomenon relative to the classifier used in this study and, at the same time, calls into question the classical wisdom around the peaking phenomenon.
KeywordsPeaking phenomenon Linear discriminant analysis Classification error rate Multiple asymptotic analysis
This material is based in part upon work supported by the Nazarbayev University Faculty Development Competitive Research Grant, under award number SOE2018008.
- Bowker, A., & Sitgreaves, R. (1961). An asymptotic expansion for the distribution function of the w-classification statistic. In Solomon, H (Ed.) Studies in item analysis and prediction (pp. 292–310): Stanford University Press.Google Scholar
- Couplet, R., & Debbah, M. (2013). Signal processing in large systems, a new paradigm. IEEE Signal Processing Magazine, 24–39.Google Scholar
- Duda, R.O., Hart, P.E., Stork, D.G. (2000). Pattern classification. Wiley.Google Scholar
- Raudys, S. (1967). On determining training sample size of a linear classifier. Computer Systems, 28, 79–87. In Russian.Google Scholar
- Serdobolskii, V.I. (1983). On minimum error probability in discriminant analysis. Soviet. Math. Dokl., 27, 720–725.Google Scholar
- Serdobolskii, V.I. (2000). Multivariate statistical analysis: a high-dimensional approach. Kluwer Academic Publishers.Google Scholar
- Serdobolskii, V. (2008). Multiparametric statistics. Elsevier.Google Scholar
- Sitgreaves, R. (1961). Some results on the distribution of the W-classification statistics. In Solomon, H. (Ed.) Studies in item analysis and prediction (pp. 241–251). Stanford: Stanford University Press.Google Scholar