Advertisement

On Boosting Improvement: Error Reduction and Convergence Speed-Up

  • Marc Sebban
  • Henri-Maxime Suchier
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2837)

Abstract

Boosting is not only the most efficient ensemble learning method in practice, but also the one based on the most robust theoretical properties. The adaptive update of the sample distribution, which tends to increase the weight of the misclassified examples, allows to improve the performance of any learning algorithm. However, its ability to avoid overfitting has been challenged when boosting is applied to noisy data. This situation is frequent with the modern databases, built thanks to new data acquisition technologies, such as the Web. The convergence speed of boosting is also penalized on such databases, where there is a large overlap of probability density functions of the classes to learn (large Bayesian error). In this article, we propose a slight modification of the weight update rule of the algorithm Adaboost. We show that by exploiting an adaptive measure of a local entropy, computed from a neighborhood graph built on the examples, it is possible to identify not only the outliers but also the examples located in the Bayesian error region. Taking into account this information, we correct the weight of the examples to improve the boosting performances. A broad experimental study shows the interest of our new algorithm, called i Adaboost .

Keywords

Convergence Speed Noisy Data Error Reduction Neighborhood Graph Weak Hypothesis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Breiman, L.: Bagging predictors. Machine Learning 24, 123–140 (1996)zbMATHMathSciNetGoogle Scholar
  2. 2.
    Breiman, L.: Bias, variance, and arcing classifiers. Technical Report 460, Department of Statistics, University of California, Berkeley (1996)Google Scholar
  3. 3.
    Freund, Y.: Boosting a weak learning algorithms by majority. Information and Computation 121, 256–285 (1995)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Schapire, R.E.: The strength of weak learnability. Machine Learning (1990) Google Scholar
  5. 5.
    Schapire, R.E., Freund, Y., Bartlett, P.L., Lee, W.S.: Boosting the margin: a new explanation for the effectiveness of voting methods. Annals of Statistics 26, 1651–1686 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. In: Press, A. (ed.) Eleventh Annual Conference on Computational Learning Theory, pp. 80–91 (1998)Google Scholar
  7. 7.
    Freund, Y., Schapire, R.: A decision-theoretic generalization of online learning and an application to boosting. International Journal of Computer and System Sciences 55(1), 119–139 (1997)zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Kaufmann, M. (ed.) Thirteenth International Conference on Machine Learning, pp. 148–156 (1996)Google Scholar
  9. 9.
    Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Machine Learning, 1–22 (1999)Google Scholar
  10. 10.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Technical report (1998)Google Scholar
  11. 11.
    Freund, Y.: An adaptive version of the boost by majority algorithm. Machine Learning 43, 293–318 (2001)zbMATHCrossRefGoogle Scholar
  12. 12.
    Domingo, C., Watanabe, O.: Madaboost: a modification of adaboost. In: Press, A., ed.: Third Annual Confernce on Computational Learning Theory, pp. 180–189 (2000)Google Scholar
  13. 13.
    Nock, R., Sebban, M.: A bayesian boosting theorem. Pattern Recognition Letters 22 (3-4), 413–419 (2001)zbMATHCrossRefGoogle Scholar
  14. 14.
    Sebban, M., Nock, R., Lallich, S.: Stopping criterion for boosting-based data reduction techniques: from binary to multiclass problems. Journal of Machine Learning Research (2003) Google Scholar
  15. 15.
    Wilson, D., Martinez, T.: Reduction techniques for exemplar-based learning algorithms. Machine Learning (1998) Google Scholar
  16. 16.
    Ratsch, G., Onoda, T., Muller, K.R.: Regularizing adaboost. In: Kearns, M.S., Solla, S.A., Cohn, D.A. (eds.) Conference NIPS (1998)Google Scholar
  17. 17.
    Kwek, S., Nguyen, C.: iboost: Boosting using an instance-based exponential weighting scheme. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 245–257. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  18. 18.
    Nock, R., Lefaucheur, P.: A robust boosting algorithm. In: Thirteenth European Conference on Machine Learning (2002) Google Scholar
  19. 19.
    Maclin, R.: Boosting classifiers regionally. In: AAAI/IAAI, pp. 700–705 (1998)Google Scholar
  20. 20.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inform. Theory IT13, 21–27 (1967)CrossRefGoogle Scholar
  21. 21.
    Preparata, F., Shamos, M.: Pattern recognition and scene analysis. Springer, Heidelberg (1985)Google Scholar
  22. 22.
    Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and regression trees with misclassification costs. Chapman and Hall, Boca Raton (1984)Google Scholar
  23. 23.
    Merz, C.J., Murphy, P.M.: Uci repository of machine learning databases (1996), http://www.ics.uci.edu/mlearn/mlrepository.html

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Marc Sebban
    • 1
  • Henri-Maxime Suchier
    • 1
  1. 1.EURISEUniversité Jean Monnet de Saint-EtienneSaint-Etienne cedex 2France

Personalised recommendations