Variable Selection

Abstract

We have already remarked that multicollinearity, i.e., nearly linearly dependent columns in the design matrix, may increase the variance of the estimators $\hat{\beta }_{i}$. For simplicity of presentation, we will assume throughout this section that the response is centered and predictor variables are standardized. More formally, Zvára (2008, Theorem 11.1) observes in the linear model (8.1) that

$$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2} =\|\beta \| ^{2} +\sigma ^{2}\mathop{ \mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1}}$$

and

$$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{Y }\|^{2} =\| \mathcal{X}\beta \|^{2} +\sigma ^{2}\mathop{ \mathrm{\text{rank}}}\nolimits (\mathcal{X}).}$$

It follows that multicollinearity does not affect the fitted values $\hat{Y } = \mathcal{X}\hat{\beta }$ because the expectation of its squared length depends only on σ ² and the rank of the model matrix $\mathcal{X}$. On the other hand, the expectation of the squared length of the estimator $\hat{\beta }$ depends on the term $\mathop{\mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1} =\sum \lambda _{ i}^{-1}$, where λ _i are the eigenvalues of $\mathcal{X}^{\top }\mathcal{X}$. If the columns of $\mathcal{X}$ are nearly dependent, some of these eigenvalues may be very small and $\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2}$ then may become very large even if, technically, the design matrix still has full rank.