Lower bounds on learning decision lists and trees

Abstract

k-decision lists and decision trees play important roles in learning theory as well as in practical learning systems, k-decision lists generalize classes such as monomials, k-DNF, and k-CNF and like these subclasses is polynomially PAC-learnable [19]. This leaves open the question of whether k-decision lists can be learned as efficiently as k-DNF. We answer this question negatively in a certain sense, thus disproving a claim in a popular textbook [2]. Decision trees, on the other hand, are not even known to be polynomially PAC-learnable, despite their widespread practical application. We will show that decision trees are not likely to be efficiently PAC-learnable. We summarize our specific results.

The following problems cannot be approximated in polynomial time within a factor of $2^{\log ^\delta n}$ for any δ<1, unless NP⊂DTIME[2^{polylog n}]: a generalized set cover, k-decision lists, k-decision lists by monotone decision lists, and decision trees. Decision lists cannot be approximated in polynomial time within a factor of n ^δ, for some constant δ>0, unless NP=P. Also, k-decision lists with l 0–1 alternations cannot be approximated within a factor log^l n unless NP⊂DTIME[n ^O(log log n)] (providing an interesting comparison to the upper bound recently obtained in [1]).

The research was supported by NSERC Research Grants OGP0046613 and OGP-0046506, an NSERC International Fellowship, and ITRC.