Abstract
In this paper we present new bounds on the generalization error of a classifier f constructed as a convex combination of base classifiers from the class H. The algorithms of combining simple classifiers into a complex one, such as boosting and bagging, have attracted a lot of attention. We obtain new sharper bounds on the generalization error of combined classifiers that take into account both the empirical distribution of “classification margins” and the “approximate dimension” of the classifier, which is defined in terms of weights assigned to base classifiers by a voting algorithm. We study the performance of these bounds in several experiments with learning algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anthony, M. and Bartlett, P. (1999) Neural Network Learning: Theoretical Foundations. Cambridge University Press.
Bartlett, P. (1998) The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network. IEEE Transactions on Information Theory, 44, 525–536.
Blake, C., Merz, C. (1998) UCI repository of machine learning databases. URL: http://www.ics.uci.edu/ mlearn/MLRepository.html.
Breiman, L. (1998) Arcing Classifiers. The Annals of Statistics, 26(3).
Devroye, L., Györfi, L. and Lugosi, L. (1996) A Probabilistic Theory of Pattern Recognition. Springer-Verlag, New York.
Dudley, R.M. (1999) Uniform Central Limit Theorems. Cambridge University Press.
Kearns, M., Mansour, Y., Ng, A., Ron, D. (1997) An Experimental and Theoretical Comparison of Model Selection Methods. Machine Learning, 27(1)
Koltchinskii, V. and Panchenko, D. (2000) Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers. To appear in Ann. Statist.
Koltchinskii, V., Panchenko, D. and Lozano, F. (2000) Bounding the Generalization Error of Convex Combinations of Classifiers: Balancing the Dimensionality and the Margins. Preprint.
Koltchinskii, V., Panchenko, D. and Lozano, F. (2000) Some New Bounds on the Generalization Error of Combined Classifiers. Advances in Neural Information Processing Systems 13: Proc. of NIPS’2000.
Massart, P. (2000) About the Constants in Talagrand’s Concentration Inequalities for Empirical Processes. Ann. Probab., 28(2).
Panchenko, D. (2001) A Note on Talagrand’s Concentration Inequality. To appear in Electron. J. Probab.
Schapire, R., Freund, Y., Bartlett, P. and Lee, W.S. (1998) Boosting the Margin: A New Explanation of Effectiveness of Voting Methods. Ann. Statist. 26, 1651–1687.
Talagrand, M. (1996a) A New Look at Independence. Ann. Probab., 24, 1–34.
Talagrand, M. (1996b) New Concentration Inequalities in Product Spaces. Invent. Math., 126, 505–563.
van der Vaart, A.W. and Wellner, J.A. (1996) Weak Convergence and Empirical Processes. With Applications to Statistics. Springer-Verlag, New York.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Koltchinskii, V., Panchenko, D., Lozano, F. (2001). Further Explanation of the Effectiveness of Voting Methods: The Game between Margins and Weights. In: Helmbold, D., Williamson, B. (eds) Computational Learning Theory. COLT 2001. Lecture Notes in Computer Science(), vol 2111. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44581-1_16
Download citation
DOI: https://doi.org/10.1007/3-540-44581-1_16
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42343-0
Online ISBN: 978-3-540-44581-4
eBook Packages: Springer Book Archive