Foundations of Statistical Learning and Model Selection

Abstract

What the reader should know to understand this chapter $\bullet$ Basic notions of machine learning. $\bullet$ Notions of calculus. $\bullet$ Chapter 5.

Problems

7.1

Prove that the average error, in the case of regression, $\mathcal {E}[(f(\mathbf {x},\mathcal {D})-F(\mathbf {x}))^2]$ can be decomposed in the following way:

$$ \mathcal {E}[(f(\mathbf {x},\mathcal {D})-F(\mathbf {x}))^2]= (\mathcal {E}[f(\mathbf {x},\mathcal {D})-F(\mathbf {x})])^2 + \mathcal {E}[(f(\mathbf {x},\mathcal {D})-\mathcal {E}[f(\mathbf {x},\mathcal {D})])^2] $$

7.2

Consider the bias-variance decomposition for classification. Show that if the classification error $P(f(\mathbf {x}, \mathcal {D}=y)$ does not coincide with Bayes discriminant error, it is given by:

$$ P(f(\mathbf {x}, \mathcal {D}=y)= |2 \gamma (\mathbf {x})-1| + P(y_b(\mathbf {x})=y). $$

7.3

Prove that the class of functions $\textit{sin}(\alpha x)$ ($\alpha \in \mathbb {R}$) has infinite VC dimension (Theorem 4). You can compare your proof with the one reported in [25].

7.4

For any $\epsilon > 0$, prove that

$$\begin{aligned} P(|\mathcal {R}_{\textit{emp}}[f]-\mathcal {R}[f]| \ge \epsilon ) \le \exp (-2 \ell \epsilon ^2) \end{aligned}$$

(7.50)

7.5

Prove that the annealed entropy is an upper bound of VC Entropy. Hint: use Jensen’s inequality [25] which states that for a concave function $\psi$ the inequality

$$ \int \psi (\varPhi (x))dF(x) \le \psi \left( \int \varPhi (x) dF(x) \right) $$

holds.

7.6

Prove that if a class of function $\mathcal {F}$ can shatter any data set of $\ell$ samples the third milestone of VC theory is not fulfilled, that is the condition (7.32) does not hold.

7.7

Implement the AIC criterion. Consider spam data that can be dowloaded by ftp.ics.uci.edu/pub/machine-learning-databases/spam. Divide randomly spam data in two subsets with the same number of samples. Take the former and the latter sets respectively as the training and the test set. Select a learning algorithm for classification (e.g., K-Means or MLP) and train the algorithm with several parameter values. Use the AIC criterion for model selection. Compare their performances by means of the model assessment.

7.8

Implement the BIC criterion. Repeat Problem 7.7 and use the crossvalidation for model selection. Compare its performance with AIC.

7.9

Implement the crossvalidation criterion. Repeat Problem 7.7 and use 5-fold crossvalidation for model selection. Compare its performance with AIC and BIC.

7.10

Implement the leave-one-out method and test it on Iris Data  [12] which can be dowloaded by ftp.ics.uci.edu/pub/machine-learning-databases/iris.