Teaching and Compressing for Low VC-Dimension

Abstract

In this work we study the quantitative relation between VC-dimension and two other basic parameters related to learning and teaching. Namely, the quality of sample compression schemes and of teaching sets for classes of low VC-dimension. Let C be a binary concept class of size m and VC-dimension d. Prior to this work, the best known upper bounds for both parameters were log(m), while the best lower bounds are linear in d. We present significantly better upper bounds on both as follows. Set k = O(d2^dloglog | C | ).

We show that there always exists a concept c in C with a teaching set (i.e. a list of c-labeled examples uniquely identifying c in C) of size k. This problem was studied by Kuhlmann (On teaching and learning intersection-closed concept classes. In: EuroCOLT, pp 168–182, 1999). Our construction implies that the recursive teaching (RT) dimension of C is at most k as well. The RT-dimension was suggested by Zilles et al. (J Mach Learn Res 12:349–384, 2011) and Doliwa et al. (Recursive teaching dimension, learning complexity, and maximum classes. In: ALT, pp 209–223, 2010). The same notion (under the name partial-ID width) was independently studied by Wigderson and Yehudayoff (Population recovery and partial identification. In: FOCS, pp 390–399, 2012). An upper bound on this parameter that depends only on d is known just for the very simple case d = 1, and is open even for d = 2. We also make small progress towards this seemingly modest goal.

We further construct sample compression schemes of size k for C, with additional information of klog(k) bits. Roughly speaking, given any list of C-labelled examples of arbitrary length, we can retain only k labeled examples in a way that allows to recover the labels of all others examples in the list, using additional klog(k) information bits. This problem was first suggested by Littlestone and Warmuth (Relating data compression and learnability. Unpublished, 1986).

Appendix: Double Sampling

Here we provide our version of the double sampling argument from [8] that upper bounds the sample complexity of PAC learning for classes of constant VC-dimension. We use the following simple general lemma.

Lemma A.1

Let $(\Omega,\mathcal{F},\mu )$ and $(\Omega ',\mathcal{F}',\mu ')$ be countable¹⁵ probability spaces. Let

$$\displaystyle{F_{1},F_{2},F_{3},\ldots \in \mathcal{F},\ F_{1}^{{\prime}},F_{ 2}^{{\prime}},F_{ 3}^{{\prime}},\ldots \in \mathcal{F}'}$$

be so that μ′(F _i ^′) ≥ 1∕2 for all i. Then

$$\displaystyle{\mu \times \mu '\left (\bigcup _{i}F_{i} \times F_{i}^{{\prime}}\right ) \geq \frac{1} {2}\mu \left (\bigcup _{i}F_{i}\right ),}$$

where μ ×μ′ is the product measure.

Proof

Let $F =\bigcup _{i}F_{i}$ . For every ω ∈ F, let $F'(\omega ) =\bigcup _{i:\omega \in F_{i}}F_{i}^{{\prime}}$. As there exists i such that ω ∈ F _i it holds that F _i ^′ ⊆ F′(ω) and hence μ′(F′(ω)) ≥ 1∕2. Thus,

$$\displaystyle{ \mu \times \mu '\left (\bigcup _{i}F_{i} \times F_{i}^{{\prime}}\right ) =\sum _{\omega \in F}\mu (\{\omega \}) \cdot \mu '(F'(\omega )) \geq \sum _{\omega \in F}\mu (\{\omega \})/2 =\mu (F)/2. }$$

□

We now give a proof of Theorem 1.2. To ease the reading we repeat the statement of the theorem.

Theorem

Let X be a set and C ⊆ {0, 1}^X be a concept class of VC-dimension d. Let μ be a distribution over X. Let ε, δ > 0 and m an integer satisfying 2(2m + 1)^d(1 −ε∕4)^m < δ. Let c ∈ C and Y = (x ₁, …, x _m ) be a multiset of m independent samples from μ. Then, the probability that there is c′ ∈ C so that c | _Y = c′ | _Y but μ({x: c(x) ≠ c′(x)}) > ε is at most δ.

Proof of Theorem 1.2

Let Y ′ = (x ₁ ^′, …, x _m ^′) be another m independent samples from μ, chosen independently of Y. Let

$$\displaystyle{H =\{ h \in C: \mathsf{dist}_{\mu }(h,c)>\epsilon \}.}$$

For h ∈ C, define the event

$$\displaystyle{F_{h} =\{ Y: c\vert _{Y } = h\vert _{Y }\},}$$

and let $F =\bigcup _{h\in H}F_{h}$. Our goal is thus to upper bound $\Pr (F)$. For that, we also define the independent event

$$\displaystyle{F_{h}^{{\prime}} =\{ Y ': \mathsf{dist}_{ Y '}(h,c)>\epsilon /2\}.}$$

We first claim that $\Pr (F_{h}^{{\prime}}) \geq 1/2$ for all h ∈ H. This follows from Chernoff’s bound, but even Chebyshev’s inequality suffices: For every i ∈ [m], let V _i be the indicator variables of the event h(x _i ^′) ≠ c(x _i ^′) (i.e., V _i = 1 if and only if h(x _i ^′) ≠ c(x _i ^′)). The event F _h ^′ is equivalent to V = ∑ _i V _i ∕m > ε∕2. Since h ∈ H, we have $p:= \mathbb{E}[V ]>\epsilon$. Since elements of Y ′ are chosen independently, it follows that Var(V ) = p(1 − p)∕m. Thus, the probability of the complement of F _h ^′ satisfies

$$\displaystyle{ \Pr ((F_{h}^{{\prime}})^{c}) \leq \Pr (\vert V - p\vert \geq p -\epsilon /2) \leq \frac{p(1 - p)} {(p -\epsilon /2)^{2}m} <\frac{4} {\epsilon m} \leq 1/2. }$$

We now give an upper bound on $\Pr (F)$. We note that

Let S = Y ∪ Y ′, where the union is as multisets. Conditioned on the value of S, the multiset Y is a uniform subset of half of the elements of S. Thus,

Notice that if dist _{Y ′}(h′, c) > ε∕2 then dist _S (h′, c) > ε∕4, hence the probability that we choose Y such that h′ | _Y = c | _Y is at most (1 −ε∕4)^m. Using Theorem 1.1 we get

$$\displaystyle{ \Pr (F) \leq 2\mathop{ \mathbb{E}} _{S}\left [\sum _{h'\in H\vert _{S}}(1 -\epsilon /4)^{m}\right ] \leq 2(2m + 1)^{d}(1 -\epsilon /4)^{m}. }$$

□