Tracking Concept Change with Incremental Boosting by Minimization of the Evolving Exponential Loss

  • Mihajlo Grbovic
  • Slobodan Vucetic
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6911)


Methods involving ensembles of classifiers, such as bagging and boosting, are popular due to the strong theoretical guarantees for their performance and their superior results. Ensemble methods are typically designed by assuming the training data set is static and completely available at training time. As such, they are not suitable for online and incremental learning. In this paper we propose IBoost, an extension of AdaBoost for incremental learning via optimization of an exponential cost function which changes over time as the training data changes. The resulting algorithm is flexible and allows a user to customize it based on the computational constraints of the particular application. The new algorithm was evaluated on stream learning in presence of concept change. Experimental results showed that IBoost achieves better performance than the original AdaBoost trained from scratch each time the data set changes, and that it also outperforms previously proposed Online Coordinate Boost, Online Boost and its non-stationary modifications, Fast and Light Boosting, ADWIN Online Bagging and DWM algorithms.


Ensemble Learning Incremental Learning Boosting Concept Change 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kolter, J.Z., Maloof, M.A.: Dynamic weighted majority: A new ensemble method for tracking concept drift. In: ICDM, pp. 123–130 (2003)Google Scholar
  2. 2.
    Scholz, M.: Knowledge-Based Sampling for Subgroup Discovery. In: Local Pattern Detection, pp. 171–189. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  3. 3.
    Bifet, A., Holmes, G., Pfahringer, B., Kirkby, R., Gavald, R.: New ensemble methods for evolving data streams. In: ACM SIGKDD (2009)Google Scholar
  4. 4.
    Wang, H., Fan, W., Yu, P.S., Han, J.: Mining concept-drifting data streams using ensemble classifiers. In: Proc. ACM SIGKDD, pp. 226–235 (2003)Google Scholar
  5. 5.
    Chu, F., Zaniolo, C.: Fast and light boosting for adaptive mining of data streams. In: Proc. PAKDD, pp. 282–292 (2004)Google Scholar
  6. 6.
    Oza, N., Russell, S.: Experimental comparisons of online and batch versions of bagging and boosting. In: ACM SIGKDD (2001)Google Scholar
  7. 7.
    Pocock, A., Yiapanis, P., Singer, J., Lujan, M., Brown, G.: Online Non-Stationary Boosting. In: Intl. Workshop on Multiple Classifier Systems (2010)Google Scholar
  8. 8.
    Attar, V., Sinha, P., Wankhade, K.: A fast and light classifier for data streams. Evolving Systems 1(4), 199–207 (2010)CrossRefGoogle Scholar
  9. 9.
    Pelossof, R., Jones, M., Vovsha, I., Rudin, C.: Online Coordinate Boosting. In: On-line Learning for Computer Vision Workshop, ICCV (2009)Google Scholar
  10. 10.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. The Annals of Statistics 28, 337–407 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine Learning: Proceedings of the Thirteenth International Conference, pp. 148–156 (1996)Google Scholar
  12. 12.
    Schapire, R.E., Singer, Y.: Improved Boosting Algorithms Using Confidence-rate Predictions. Machine Learning Journal 37, 297–336 (1999)CrossRefzbMATHGoogle Scholar
  13. 13.
    Schapire, R.E.: The convergence rate of adaboost. In: COLT (2010)Google Scholar
  14. 14.
    Zliobaite, I.: Learning under Concept Drift: an Overview, Technical Report, Vilnius University, Faculty of Mathematics and Informatics (2009)Google Scholar
  15. 15.
    Street, W., Kim, Y.: A streaming ensemble algorithm (SEA) for large-scale classification. In: ACM SIGKDD, pp. 377–382 (2001)Google Scholar
  16. 16.
    Weigend, A.S., Mangeas, M., Srivastava, A.N.: Nonlinear gated experts for time series: discovering regimes and avoiding overfitting. In: IJNS, vol. 6, pp. 373–399 (1995)Google Scholar
  17. 17.
    Bifet, A., Gavald, R.: Learning from time changing data with adaptive windowing. In: SIAM International Conference on Data Mining, pp. 443–448 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Mihajlo Grbovic
    • 1
  • Slobodan Vucetic
    • 1
  1. 1.Department of Computer and Information SciencesTemple UniversityPhiladelphiaUSA

Personalised recommendations