Tight Combinatorial Generalization Bounds for Threshold Conjunction Rules

  • Konstantin Vorontsov
  • Andrey Ivahnenko
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6744)


We propose a combinatorial technique for obtaining tight data dependent generalization bounds based on a splitting and connectivity graph (SC-graph) of the set of classifiers. We apply this approach to a parametric set of conjunctive rules and propose an algorithm for effective SC-bound computation. Experiments on 6 data sets from the UCI ML Repository show that SC-bound helps to learn more reliable rule-based classifiers as compositions of less overfitted rules.


computational learning theory generalization bounds permutational probability splitting-connectivity bounds rule induction 


  1. 1.
    Boucheron, S., Bousquet, O., Lugosi, G.: Theory of classification: A survey of some recent advances. ESAIM: Probability and Statistics (9), 323–375 (2005)Google Scholar
  2. 2.
    Cohen, W.W.: Fast effective rule induction. In: Proc. of the 12th International Conference on Machine Learning, Tahoe City, CA, pp. 115–123. Morgan Kaufmann, San Francisco (1995)Google Scholar
  3. 3.
    Cohen, W.W., Singer, Y.: A simple, fast and effective rule learner. In: Proc. of the 16 National Conference on Artificial Intelligence, pp. 335–342 (1999)Google Scholar
  4. 4.
    Fürnkranz, J., Flach, P.A.: Roc ‘n’ rule learning-towards a better understanding of covering algorithms. Machine Learning 58(1), 39–77 (2005)CrossRefzbMATHGoogle Scholar
  5. 5.
    Martin, J.K.: An exact probability metric for decision tree splitting and stopping. Machine Learning 28(2-3), 257–291 (1997)CrossRefGoogle Scholar
  6. 6.
    Quinlan, J.R.: C4.5: Programs for machine learning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  7. 7.
    Quinlan, J.R.: Bagging, boosting, and C4.5. AAAI/IAAI 1, 725–730 (1996)Google Scholar
  8. 8.
    Vorontsov, K.V.: Combinatorial probability and the tightness of generalization bounds. Pattern Recognition and Image Analysis 18(2), 243–259 (2008)CrossRefGoogle Scholar
  9. 9.
    Vorontsov, K.V.: Splitting and similarity phenomena in the sets of classifiers and their effect on the probability of overfitting. Pattern Recognition and Image Analysis 19(3), 412–420 (2009)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Konstantin Vorontsov
    • 1
    • 2
  • Andrey Ivahnenko
    • 1
    • 2
  1. 1.Dorodnycin Computing Center RASMoscowRussia
  2. 2.Moscow Institute of Physics and TechnologyMoscowRussia

Personalised recommendations