Abstract
This paper discusses three methods to test the significance of classification efficiency (i.e. the fraction of correctly classified patterns of a test set), which can be applied to multiple class problems: the exact probability test, the Monte-Carlo test and the x2 test. First, a short theoretical description of the three methods is given. The methods have been applied to different classification problems. A comparison is made in terms of the following criteria: required assumptions, power and time behaviour. To conclude, the paper describes a set of criteria for the selection of the appropriate classification efficiency testing method.
Chapter PDF
Keywords
- Stochastic Simulation
- Exact Probability
- Classification Efficiency
- Pattern Recognition Letter
- Random Classification
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Robert Jennrich and Paul Sampson. Stepwise discriminant analysis. In Wilfrid Joseph Dixon, editor, BMDP Statistical Software Manual. University of California Press, 1988.
Edgard Nyssen. Evaluation of pattern classifiers — testing the significance of classification efficiency using an exact probability technique. Pattern Recognition Letters, 17(11):1125–1129, September 1996.
Edgard Nyssen. Interpretation of pattern classification results, obtained from a test set. In Proceedings 1st IAPR TO Intl. Workshop on Statistical Techniques in Pattern Recognition, STIPR'97, pages 103–105, Prague, June 1997.
Edgard Nyssen. Evaluation of pattern classifiers — applying a Monte-Carlo significance test to the classification efficiency. Pattern Recognition Letters, 1998. Accepted for publication.
William H. Press, Saul A. Teukolsky, and William T. Vetterling. Numerical recipes in C: the art of scientific computing. University Press: Cambridge, 1995.
Brian D. Ripley. Stochastic Simulation. John Wiley & Sons, 1987.
Sheldon M. Ross. Introduction to Probability and Statistics for Engineers and Scientists. John Wiley & Sons, 1987.
Sidney Siegel. Nonparametric Statistics for the Behavioural Sciences. McGraw Hill, 1956.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1998 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nyssen, E. (1998). Comparison of different methods for testing the significance of classification efficiency. In: Amin, A., Dori, D., Pudil, P., Freeman, H. (eds) Advances in Pattern Recognition. SSPR /SPR 1998. Lecture Notes in Computer Science, vol 1451. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0033316
Download citation
DOI: https://doi.org/10.1007/BFb0033316
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-64858-1
Online ISBN: 978-3-540-68526-5
eBook Packages: Springer Book Archive