Abstract
The aim of this paper is to propose a simple procedure that a priori determines a minimum number of classifiers to combine in order to obtain a prediction accuracy level similar to the one obtained with the combination of larger ensembles. The procedure is based on the McNemar non-parametric test of significance. Knowing a priori the minimum size of the classifier ensemble giving the best prediction accuracy, constitutes a gain for time and memory costs especially for huge data bases and real-time applications. Here we applied this procedure to four multiple classifier systems with C4.5 decision tree (Breiman’s Bagging, Ho’s Random subspaces, their combination we labeled ’Bagfs’, and Breiman’s Random forests) and five large benchmark data bases. It is worth noticing that the proposed procedure may easily be extended to other base learning algorithms than a decision tree as well. The experimental results showed that it is possible to limit significantly the number of trees. We also showed that the minimum number of trees required for obtaining the best prediction accuracy may vary from one classifier combination method to another.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Ali and Pazzani. Error reduction through learning multiple descriptions. Machine Learning, 24:173–202, 1996.
Stephen D. Bay. Nearest neighbor classification from multiple feature subsets. In Proceedings of the International Conference on Machine Learning, Madison, Wisc., 1998. Morgan Kaufmann Publishers.
C. Blake, E. Keogh, and C.J. Merz. Uci repository of machine learning databases. [http://www.ics.uci.edu/mlearn/MLRepository.html]. Irvine, CA: University of California, Department of Information and Computer Science, 1998.
Leo Breiman. Bagging predictors. Machine Learning, 24, 1996.
Leo Breiman. Arcing classifiers. Annals of statistics, 26:801–849, 1998.
Leo Breiman. Random forests–random features. Technical Report 567, Statistics Department, University of California, Berkeley, CA 94720, september 1999.
T.G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10:1895–1923, 1998.
T.G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting and randomization. Machine Learning, 40:139–157, 2000.
Giorgio Giacento and Fabio Roli. An approach to the automatic design of multiple classifier systems. Pattern recognition letters, 22:25–33, 2001.
T.K. Ho. the random subspace method for constructing decision forests. IEEE Trans. Pattern Analysis and Machine Intelligence, 20:832–844, 1998.
Ji and Ma. Combinations of weak classifiers. IEEE Trans. Neural Network, 7(1):32–42, 1997.
Ron Kohavi and Clayton Kunz. Option decision trees with majority votes. In Proceedings of the Fourtheeth International Conference on Machine Learning, pages 161–169, San Francisco, CA, 1997. Morgan Kaufmann.
Patrice Latinne, Olivier Debeir, and Christine Decaestecker. Different ways of weakening decision trees and their impact on classification accuracy. In Proc. of the 1st International Workshop of Multiple Classifier System, pages 200–210, Cagliari, Italy, 2000. Springer (Lecture Notes in Computer Sciences; Vol. 1857).
Patrice Latinne, Olivier Debeir, and Christine Decaestecker. Mixing bagging and multiple feature subsets to improve classification accuracy of decision tree combination. In Proc. of the Tenth Belgian-Dutch Conference on Machine Learning Benelearn’00, pages 15–22, Tilburg University, 2000. Ed. Ad Feelders.
J.R. Quinlan. C4.5: Programs For Machine Learning. Morgan Kaufmann Publishers, San Mateo, California, 1993.
J.R. Quinlan. Bagging, boosting, and c4.5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 725–730, 1996.
Bernard Rosner. Fundamentals of Biostatistics. Duxbury Press (ITP), Belmont, CA, USA, 4th edition, 1995.
Steven Salzberg. On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and knowledge discovery, 1:317–327, 1997.
R.E. Schapire. The strength of weak learnability. Machine Learning, 5:197–227, 1990.
S. Siegel and N.J. Castellan. Nonparametric Statistics for the behavioral sciences. McGraw-Hill, second edition, 1988.
K. Tumer and J. Ghosh. Classifier combining: analytical results and implications. In Proceedings of the National Conference on Artificial Intelligence, Portland, OR, 1996.
Zijian Zheng. Generating classifier committees by stochastically selecting both attributes and training examples. In Proceedings of the 5th Pacific Rim International Conferences on Artificial Intelligence (PRICAI’98), pages 12–23. Berlin: Springer-Verlag, 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Latinne, P., Debeir, O., Decaestecker, C. (2001). Limiting the Number of Trees in Random Forests. In: Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2001. Lecture Notes in Computer Science, vol 2096. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48219-9_18
Download citation
DOI: https://doi.org/10.1007/3-540-48219-9_18
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42284-6
Online ISBN: 978-3-540-48219-2
eBook Packages: Springer Book Archive