Bayesian Theory of Decision

Abstract

What the reader should know to understand this chapter $\bullet$ Basic notions of statistics and probability theory (see Appendix A). $\bullet$ Calculus notions are an advantage.

Problems

5.1

Given a normal distribution $\mathcal {N}(\sigma ,\mu )$, show that the percentage of samples that assume values in $[-3\sigma , 3\sigma ]$ exceeds 99 %.

5.2

Consider the function $f(x)= \frac{a}{1+x^2}$ where $a \in \mathbb {R}$. Find the value a such that f(x) is a probability density. Besides, compute the expected value of x.

5.3

Consider the Geometric distribution [14] defined by:

$$ p(x)= \theta (1-\theta )^x \quad (x=0,1,2,\dots , 0\le \theta \le 1). $$

Prove that its mean is $\mathcal {E}[x]= \frac{1-\theta }{\theta }$.

5.4

Given a probability density f(x), the moment of fourth order [14] is defined by

$$ \frac{1}{\sigma ^4} \int _{-\infty }^{\infty } f(x) (x-\mu )^4 \textit{dx} $$

where $\mu$ and $\sigma ^2$ are, respectively, the mean and the variance.

Prove that the moment of fourth-order of a normal distribution $\mathcal {N}(\mu ,\sigma )$ is 3.

5.5

Let $x=(x_1,\dots ,x_{\ell })$ and $y=(y_1,\dots ,y_{\ell })$ be two variables. Prove that if they are statistically independent their covariance is null.

5.6

Suppose we have two classes $\mathcal {C}_1$ and $\mathcal {C}_2$ with a priori probabilities $p(\mathcal {C}_1)= \frac{1}{3}$ and $p(\mathcal {C}_2)= \frac{2}{3}$. Suppose that their likelihoods are $p(x|\mathcal {C}_1)= \mathcal {N}(1,1)$ and $p(x|\mathcal {C}_2)= \mathcal {N}(1,0)$. Find numerically the value of x such that the posterior probabilities $p(\mathcal {C}_1|x)$, $p(\mathcal {C}_2|x)$ are equal.

5.7

Suppose we have two classes $\mathcal {C}_1$ and $\mathcal {C}_2$ with a priori probabilities $p(\mathcal {C}_1)= \frac{2}{5}$ and $p(\mathcal {C}_2)= \frac{3}{5}$. Suppose that their likelihoods are $p(x|\mathcal {C}_1)= \mathcal {N}(1,0)$ and $p(x|\mathcal {C}_2)= \mathcal {N}(1,1)$. Compute the joint probability such that both points $x_1= -0.1$, $x_2= 0.2$ belong to $\mathcal {C}_1$.

5.8

Suppose we have two classes $\mathcal {C}_1$ and $\mathcal {C}_2$ with a priori probabilities $p(\mathcal {C}_1)= \frac{1}{4}$ and $p(\mathcal {C}_2)= \frac{3}{4}$. Suppose that their likelihoods are $p(x|\mathcal {C}_1)= \mathcal {N}(2,0)$ and $p(x|\mathcal {C}_2)= \mathcal {N}(0.5,1)$. Compute the likelihood ratio and write the discriminant function.

5.9

Suppose we have three classes $\mathcal {C}_1$, $\mathcal {C}_2$ and $\mathcal {C}_3$ with a priori probabilities $p(\mathcal {C}_1)= \frac{1}{6}$, $p(\mathcal {C}_2)= \frac{1}{3}$ and $p(\mathcal {C}_2)= \frac{1}{2}$. Suppose that their likelihoods are respectively $p(x|\mathcal {C}_1)= \mathcal {N}(0.25,0)$, $p(x|\mathcal {C}_2)= \frac{a}{1+x^2}$ and $p(x|\mathcal {C}_3)= \frac{1}{b+(x-1)^2}$. Find the values a and b such that likelihoods are density functions and write three discriminant functions.

5.10

Implement the whitening transform. Test your implementation transforming Iris Data [9], which can be downloaded by ftp.ics.uci.edu/pub/machine-learning-databases/iris. Verify that the covariance matrix of the transformed data is the identity matrix.

5.11

Suppose that the features are statistically independent and that they have the same variance $\sigma$. In this case where the discriminant function is a linear classifier. Given two adjacent decision regions $\mathcal {D}_1$ and $\mathcal {D}_2$, show that their separating hyperplane is orthogonal to the line connecting the means $\mu _1$ and $\mu _2$.

5.12

Suppose that the covariance matrix is the same for all the classes. The discriminant function is a linear classifier. Given two adjacent decision regions $\mathcal {D}_1$ and $\mathcal {D}_2$ show that their separating hyperplane is not orthogonal to the line connecting the means $\mu _1$ and $\mu _2$.