Lower Bounds for Empirical Classifier Selection

Abstract

In Chapter 12 a classifier was selected by minimizing the empirical error over a class of classifiers C. With the help of the Vapnik-Chervonenkis theory we have been able to obtain distribution-free performance guarantees for the selected rule. For example, it was shown that the difference between the expected error probability of the selected rule and the best error probability in the class behaves at least as well as $$ where V _C is the Vapnik-Chervonenkis dimension of C, and n is the size of the training data D _n . (This upper bound is obtained from Theorem 12.5. Corollary 12.5 may be used to replace the log n term with log V _C .) Two questions arise immediately: Are these upper bounds (at least up to the order of magnitude) tight? Is there a much better way of selecting a classifier than minimizing the empirical error? This chapter attempts to answer these questions. As it turns out, the answer is essentially affirmative for the first question, and negative for the second.