Abstract
Combining several classifiers has proved to be an efficient machine learning technique. We propose a new measure of the goodness of an ensemble of classifiers in an information theoretic framework. It measures a trade-off between diversty and individual classifier accuracy. This technique can be directly used for the selection of an ensemble in a pool of classifiers. We also propose a variant of AdaBoost for directly training the classifiers by taking into account this new information theoretic measure.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)
Freund, Y., Mansour, Y., Schapire, R.: Why averaging classifiers can protect against overfitting. In: Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics (2001)
Kuncheva, L.I.: Combining Pattern Classifiers Methods and Algorithms. John Wiley, New York (2004)
Kittler, J., et al.: On combining classifiers. IEEE Trans. Pattern Anal. Mach. Intell. 20(3), 226–239 (1998)
Kuncheva, L., Whitaker, C.: Measures of diversity in classifier ensembles. Machine Learning 51(2), 181–207 (2003)
Principe, J.C., Xu, D., Fisher, J.W.: Learning from examples with information theoretic criteria. J. VLSI Signal Process. Syst. 26, 61–77 (2000)
Fano, R.M.: Transmission of Information: A Statistical Theory of Communication. Wiley, Chichester (1961)
Cover, T., Thomas, J.: Elements of Information Theory. John Wiley and Sons, Inc., New York (1991)
Erdogmus, D., Principe, J.C.: Lower and upper bounds for misclassification probability based on renyi’s information. Journal of VLSI Signal Processing 37, 305–317 (2004)
Fisher III., J., Principe, J.: A methodology for information theoretic feature extraction. In: IEEE International Conference on Neural Networks (IJCNN’98), vol. 3, Anchorage, AK, pp. 1712–1716 (1998)
Hild II, K.E., et al.: Feature extraction using information-theoretic learning. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1385–1392 (2006)
Butz, T., Thiran, J.-P.: From error probability to information theoretic (multi-modal) signal processing. Signal Processing 85(5), 875–902 (2005)
Sindhwani, V., et al.: Feature selection in mlps and svms based on maximum output information. IEEE Transactions On Neural Networks 15, 937–949 (2004)
Raudys, S.: Trainable fusion rules. II Small sample-size effects. Neural Networks 19(10), 1517–1527 (2006)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)
Shapley, L., Grofman, B.: Optimizing group judgemental accuracy in the presence of interdependencies. Public Choice 43, 329–343 (1984)
Yule, G.: On the association of attributes in statistics. Biometrika 2, 121–134 (1903)
Giacinto, G., Roli, F.: Design of effective neural network ensembles for image classification purposes. Image Vision Comput. 19(9-10), 9–10 (2001)
Skalak, D.: The sources of increased accuracy for two proposed boosting algorithms. In: AAAI ’96 Workshop on Integrating Multiple Learned Models for Improving and Scaling Machine Learning Algorithms (1996)
Brown, G., et al.: Diversity creation methods: A survey and categorisation. Journal of Information Fusion 6(1), 5–20 (2005)
Duin, R.P.W., et al.: Prtools4, a matlab toolbox for pattern recognition. Delft University of Technology (2004)
Meynet, J., et al.: Combining svms for face class modeling. In: 13th European Signal Processing Conference - EUSIPCO, Antalya, Turkey (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Meynet, J., Thiran, JP. (2007). Information Theoretic Combination of Classifiers with Application to AdaBoost. In: Haindl, M., Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2007. Lecture Notes in Computer Science, vol 4472. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72523-7_18
Download citation
DOI: https://doi.org/10.1007/978-3-540-72523-7_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72481-0
Online ISBN: 978-3-540-72523-7
eBook Packages: Computer ScienceComputer Science (R0)