Skip to main content
SpringerLink
Log in
Book cover

Advances in Machine Learning I pp 209–225Cite as

Beyond Sequential Covering – Boosted Decision Rules

  • Chapter

Part of the book series: Studies in Computational Intelligence ((SCI,volume 262))

Abstract

From the beginning of machine learning, rule induction has been regarded as one of the most important issues in this research area. One of the first rule induction algorithms was AQ introduced by Michalski in early 80’s. AQ, as well as several other well-known algorithms, such as CN2 and Ripper, are all based on sequential covering. With the advancement of machine learning, some new techniques based on statistical learning were introduced. One of them, called boosting, or forward stagewise additive modeling, is a general induction procedure which appeared to be particularly efficient in binary classification and regression tasks. When boosting is applied to induction of decision rules, it can be treated as generalization of sequential covering, because it approximates the solution of the prediction task by sequentially adding new rules to the ensemble without adjusting those that have already entered the ensemble. Each rule is fitted by concentrating on examples which were the hardest to classify correctly by the rules already present in the ensemble. In this paper, we present a general scheme for learning an ensemble of decision rules in a boosting framework, using different loss functions and minimization techniques. This scheme, called ENDER, is covered by such algorithms as SLIPPER, LRI and MLRules. A computational experiment compares these algorithms on benchmark data.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Michalski, R.S.: A Theory and Methodology of Inductive Learning. In: Michalski, R.S., Carbonell, J.G., Mitchell, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, pp. 83–129. Tioga Publishing, Palo Alto (1983)

    Google Scholar 

  2. Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learning 3, 261–283 (1989)

    Google Scholar 

  3. Cohen, W.W.: Fast Effective Rule Induction. In: Proc. of International Conference of Machine Learning, pp. 115–123 (1995)

    Google Scholar 

  4. Fürnkranz, J.: Separate-and-Conquer Rule Learning. Artificial Intelligence Review 13(1), 3–54 (1996)

    Article  Google Scholar 

  5. Jovanoski, V., Lavrac, N.: Classification Rule Learning with APRIORI-C. In: Proc. of the 10th Portuguese Conference on Progress in Artificial Intelligence, Knowledge Extraction, Multi-agent Systems, Logic Programming and Constraint Solving, London, UK, pp. 44–51. Springer, Heidelberg (2001)

    Google Scholar 

  6. Stefanowski, J., Vanderpooten, D.: Induction of Decision Rules in Classification and Discovery-oriented Perspectives. International Journal on Intelligent Systems 16(1), 13–27 (2001)

    Article  MATH  Google Scholar 

  7. Bazan, J.G.: Discovery of Decision Rules by Matching New Objects Against Data Tables. In: Polkowski, L., Skowron, A. (eds.) RSCTC 1998. LNCS (LNAI), vol. 1424, pp. 521–528. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  8. Góra, G., Wojna, A.: RIONA: A New Classification System Combining Rule Induction and Instance-based Learning. Fundamenta Informaticae 54, 369–390 (2002)

    Google Scholar 

  9. Domingos, P.: Unifying Instance-based and Rule-based Induction. Machine Learning 24, 141–168 (1996)

    Google Scholar 

  10. Góra, G., Wojna, A.: Local Attribute Value Grouping for Lazy Rule Induction. In: Alpigini, J.J., Peters, J.F., Skowron, A., Zhong, N. (eds.) RSCTC 2002. LNCS (LNAI), vol. 2475, pp. 405–412. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  11. Boros, E., Hammer, P.L., Ibaraki, T., Kogan, A., Mayoraz, E., Muchnik, I.: An Implementation of Logical Analysis of Data. IEEE Transactions on Knowledge and Data Engineering 12, 292–306 (2000)

    Article  Google Scholar 

  12. Pawlak, Z.: Rough Sets. Theoretical Aspects of Reasoning about Data. Kluwer Academic Publishers, Dordrecht (1991)

    MATH  Google Scholar 

  13. Słowiński, R. (ed.): Intelligent Decision Support. In: Handbook of Applications and Advances of the Rough Set Theory. Kluwer Academic Publishers, Dordrecht (1992)

    Google Scholar 

  14. Grzymala-Busse, J.W.: LERS — A System for Learning from Examples based on Rough Sets. In: Słowiński, R. (ed.) Intelligent Decision Support, Handbook of Applications and Advances of the Rough Sets Theory, pp. 3–18. Kluwer Academic Publishers, Dordrecht (1992)

    Google Scholar 

  15. Skowron, A.: Extracting Laws from Decision Tables - A Rough Set Approach. Computational Intelligence 11, 371–388 (1995)

    Article  MathSciNet  Google Scholar 

  16. Stefanowski, J.: On Rough Set based Approach to Induction of Decision Rules. In: Skowron, A., Polkowski, L. (eds.) Rough Set in Knowledge Discovering, pp. 500–529. Physica Verlag, Heidelberg (1998)

    Google Scholar 

  17. Greco, S., Matarazzo, B., Słowiński, R., Stefanowski, J.: An algorithm for induction of decision rules consistent with the dominance principle. In: Ziarko, W.P., Yao, Y. (eds.) RSCTC 2000. LNCS (LNAI), vol. 2005, pp. 304–313. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  18. Freund, Y., Schapire, R.E.: A Decision-theoretic Generalization of On-line Learning and an Application to Boosting. Journal of Computer and System Sciences 55(1), 119–139 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  19. Hastie, T., Tibshirani, R., Friedman, J.H.: Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Heidelberg (2003)

    Google Scholar 

  20. Cohen, W.W., Singer, Y.: A Simple, Fast, and Effective Rule Learner. In: Proc. of National Conference on Artificial Intelligence, pp. 335–342 (1999)

    Google Scholar 

  21. Weiss, S.M., Indurkhya, N.: Lightweight Rule Induction. In: Proc. of International Conference on Machine Learning, pp. 1135–1142 (2000)

    Google Scholar 

  22. Friedman, J.H., Popescu, B.E.: Predictive Learning via Rule Ensembles. Annals of Applied Statistics 2(3), 916–954 (2008)

    Article  MATH  Google Scholar 

  23. Dembczyński, K., Kotłowski, W., Słowiński, R.: Maximum Likelihood Rule Ensembles. In: Proc. of International Conference on Machine Learning, pp. 224–231 (2008)

    Google Scholar 

  24. Błaszczyński, J., Dembczyński, K., Kotłowski, W., Słowiński, R., Szelag, M.: Ensembles of Decision Rules. Foundations of Computing and Decision Sciences 31(3-4), 221–232 (2006)

    Google Scholar 

  25. Friedman, J.H.: Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics 29(5), 1189–1232 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  26. Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1996)

    MATH  MathSciNet  Google Scholar 

  27. Mason, L., Baxter, J., Bartlett, P., Frean, M.: Functional Gradient Techniques for Combining Hypotheses. In: Bartlett, P., Schölkopf, B., Schuurmans, D., Smola, A.J. (eds.) Advances in Large Margin Classifiers, pp. 33–58. MIT Press, Cambridge (1999)

    Google Scholar 

  28. Fürnkranz, J.: Rule-based Classification. In: From Local Patterns to Global Models ECML/PKDD 2008 Workshop (2008)

    Google Scholar 

  29. Dembczyński, K., Kotłowski, W., Słowiński, R.: A General Framework for Learning an Ensemble of Decision Rules. In: Fürnkranz, J., Knobbe, A. (eds.) From Local Patterns to Global Models ECML/PKDD 2008 Workshop (2008)

    Google Scholar 

  30. Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. Annals of Statistics 26(5), 1651–1686 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  31. Friedman, J.H., Hastie, T., Tibshirani, R.: Additive Logistic Regression: A Statistical View of Boosting. Annals of Statistics (with discussion) 28(2), 337–407 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  32. Dietterich, T.G.: An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization. Machine Learning 40(2), 139–158 (2000)

    Article  Google Scholar 

  33. Friedman, J.H., Popescu, B.E.: Importance Sampled Learning Ensembles. Research report, Dept. of Statistics, Stanford University (2003)

    Google Scholar 

  34. Dembczyński, K., Kotłowski, W., Słowiński, R.: Solving Regression by Learning an Ensemble of Decision Rules. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 533–544. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  35. Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)

    MATH  Google Scholar 

  36. Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007)

    Google Scholar 

  37. Demšar, J.: Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7, 1–30 (2006)

    Google Scholar 

  38. Hilderman, R.J., Hamilton, H.J.: Knowledge Discovery and Measures of Interest. Kluwer Academic Publishers, Boston (2001)

    MATH  Google Scholar 

  39. Greco, S., Pawlak, Z., Słowiński, R.: Can Bayesian confirmation measures be useful for rough set decision rules? Engineering Applications of Artificial Intelligence 17, 345–361 (2004)

    Article  Google Scholar 

  40. Brzezińska, I., Greco, S., Słowiński, R.: Mining Pareto-optimal Rules with Respect to Support and Anti-support. Engineering Applications of Artificial Intelligence 20, 587–600 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Poznań University of Technology, 60-965, Poznań, Poland

    Krzysztof Dembczyński, Wojciech Kotłowski & Roman Słowiński

  2. Systems Research Institute, Polish Academy of Sciences, 01-447, Warsaw, Poland

    Roman Słowiński

Authors
  1. Krzysztof Dembczyński
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wojciech Kotłowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roman Słowiński
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Computer Science, Polish Academy of Sciences, ul.Ordona 21, 01-237, Warsaw, Poland

    Jacek Koronacki  & Sławomir T. Wierzchoń  & 

  2. Woodward Hall 430C University of North Carolina, 9201 University City Blvd., N.C. 28223, Charlotte, USA

    Zbigniew W. Raś

  3. Systems Research Institute, Polish Academy of Sciences, ul.Newelska 6, 01-447, Warsaw, 01-447

    Janusz Kacprzyk

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Dembczyński, K., Kotłowski, W., Słowiński, R. (2010). Beyond Sequential Covering – Boosted Decision Rules. In: Koronacki, J., Raś, Z.W., Wierzchoń, S.T., Kacprzyk, J. (eds) Advances in Machine Learning I. Studies in Computational Intelligence, vol 262. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-05177-7_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-05177-7_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-05176-0

  • Online ISBN: 978-3-642-05177-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation