Abstract
Boosting methods while being among the best classification methods developed so far, are known to degrade performance in case of noisy data and overlapping classes. In this paper we propose a new upper generalization bound for weighted averages of hypotheses, which uses posterior estimates for training objects and is based on reduction of binary classification problem with overlapping classes to a deterministic problem. If we are given accurate posterior estimates, proposed bound is lower than existing bound by Schapire et al [25]. We design an AdaBoost-like algorithm which optimizes proposed generalization bound and show that incorporated with good posterior estimates it performs better than the standard AdaBoost on real-world data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Barinova, O., Vetrov, D.: ODDboost: Incorporating posterior estimates into AdaBoost, Technical Report 1422 (2009), http://graphics.cs.msu.ru/en/publications/text/oddboost.pdf
Boser, B., Guyon, I., Vapnik, V.: A training algorithm for optimal margin classifiers. In: Fifth Annual Workshop on Computational Learning Theory. ACM Press, Pittsburgh (1992)
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)
Burges, C.J.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge 2, 121–167 (1998)
Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1996)
Caruana, R., Niculescu-Mizil, A.: An Empirical Comparison of Supervised Learning Algorithms. In: Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)(2006)
Collins, M., Schapire, R., Singer, Y.: Logistic regression, AdaBoost and Bregman distances. Machine Learning 48(1/2/3) (2002)
Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1895–1923 (1998)
Dietterich, T.G.: An experimental comparison of three methods for constructing ensebles of decision trees: Bagging, boosting, and randomization. Machine Learning 40(2) (1999)
Domingos, P.: A Unified Bias-Variance Decomposition for Zero-One and Squared Loss. In: Proc. of the 17th National Conference on Artificial Intelligence (2000)
Domingo, C., Watanabe, O.: Madaboost: A modication of adaboost. In: 13th Annual Conference on Comp. Learning Theory (2000)
Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55 (1997)
Freund, Y., Schapire, R.: Discussion of the paper “Additive logistic regression: a statistical view of boosting” by J. Friedman, T. Hastie and R. Tibshirani. The Annals of Statistics 38(2), 391–393 (2000)
Friedman, J., Hastie, T., Tibshirani, R.: Additive Logistic Regression: a Statistical View of Boosting. The Annals of Statistics 28(2), 337–407 (2000)
Friedman, J.: Greedy function approximation: A gradient boosting machine. Annals of Statistics 29(5) (2001)
Grove, A.J., Schuurmans, D.: Boosting in the limit: Maximizing the margin of learned ensembles. In: Proceedings of the Fifteenth National Conference on Artifical Intelligence (1998)
Kearns, M., Vazirani, U.: An introduction to computational learning theory. MIT Press, Cambridge (1994)
Krause, N., Singer, Y.: Leveraging the Margin More Carefully. In: ACM International Conference Proceeding Series, vol. 69 (2004)
Mason, L., Baxter, J., Bartlett, P., Frean, M.: Boosting algorithms as gradient descent. In: Neural Information Processing Systems, vol. 12, pp. 512–518. MIT Press, Cambridge (2000)
Niculescu-Mizil, A., Caruana, R.: Obtaining Calibrated Probabilities from Boosting. In: Proc. of 21st Conference on Uncertainty in Artificial Intelligence (2005)
Perrone, M.: Improving regression estimation: Averaging methods for Variance reduction with extension to General Convex Measure Optimization, Ph.D. Thesis, Brown University (1993)
Ratsch, G.: Robust Boosting and Convex Optimization. Doctoral dissertation, University of Potsdam (2001)
Reyzin, L., Schapire, R.: How boosting the margin can also boost classifier complexity. In: Proceedings of the 23rd International Conference on Machine Learning (2006)
Rosset, S.: Robust Boosting and Its Relation to Bagging. In: KDD 2005 (2005)
Schapire, R., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: A new explanation for the effectiveness of voting methods. In: Machine Learning: Proceedings of the Fourteenth International Conference (1997)
Schapire, R., Singer, Y.: Improved boosting algorithm using confidence-rated predictions. Machine Learning 37(3), 297–336 (1999)
Schapire, R., Rochery, M., Rahim, M., Gupta, N.: Incorporating prior knowledge into boosting. In: Machine Learning: Proceedings of the Nineteenth International Conference, pp. 538–545 (2002); Takenouchi T., Eguchi S.: Robustifying AdaBoost by adding the naive error rate. Neural Computation 16 (2004)
Taniguchi, M., Tresp, V.: Averaging Regularized Estimators. Neural Computation (1997)
Vezhnevets, A., Barinova, O.: Avoiding boosting overfitting by removing confusing samples. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS, vol. 4701, pp. 430–441. Springer, Heidelberg (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Barinova, O., Vetrov, D. (2009). ODDboost: Incorporating Posterior Estimates into AdaBoost. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2009. Lecture Notes in Computer Science(), vol 5632. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03070-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-03070-3_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03069-7
Online ISBN: 978-3-642-03070-3
eBook Packages: Computer ScienceComputer Science (R0)