Comparison of different methods for testing the significance of classification efficiency

  • Edgard Nyssen
Poster Papers
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1451)


This paper discusses three methods to test the significance of classification efficiency (i.e. the fraction of correctly classified patterns of a test set), which can be applied to multiple class problems: the exact probability test, the Monte-Carlo test and the x2 test. First, a short theoretical description of the three methods is given. The methods have been applied to different classification problems. A comparison is made in terms of the following criteria: required assumptions, power and time behaviour. To conclude, the paper describes a set of criteria for the selection of the appropriate classification efficiency testing method.


Stochastic Simulation Exact Probability Classification Efficiency Pattern Recognition Letter Random Classification 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Robert Jennrich and Paul Sampson. Stepwise discriminant analysis. In Wilfrid Joseph Dixon, editor, BMDP Statistical Software Manual. University of California Press, 1988.Google Scholar
  2. 2.
    Edgard Nyssen. Evaluation of pattern classifiers — testing the significance of classification efficiency using an exact probability technique. Pattern Recognition Letters, 17(11):1125–1129, September 1996.CrossRefGoogle Scholar
  3. 3.
    Edgard Nyssen. Interpretation of pattern classification results, obtained from a test set. In Proceedings 1st IAPR TO Intl. Workshop on Statistical Techniques in Pattern Recognition, STIPR'97, pages 103–105, Prague, June 1997.Google Scholar
  4. 4.
    Edgard Nyssen. Evaluation of pattern classifiers — applying a Monte-Carlo significance test to the classification efficiency. Pattern Recognition Letters, 1998. Accepted for publication.Google Scholar
  5. 5.
    William H. Press, Saul A. Teukolsky, and William T. Vetterling. Numerical recipes in C: the art of scientific computing. University Press: Cambridge, 1995.Google Scholar
  6. 6.
    Brian D. Ripley. Stochastic Simulation. John Wiley & Sons, 1987.Google Scholar
  7. 7.
    Sheldon M. Ross. Introduction to Probability and Statistics for Engineers and Scientists. John Wiley & Sons, 1987.Google Scholar
  8. 8.
    Sidney Siegel. Nonparametric Statistics for the Behavioural Sciences. McGraw Hill, 1956.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  • Edgard Nyssen
    • 1
  1. 1.Department of ElectronicsBrussels University (VUB)BrusselsBelgium

Personalised recommendations