The General Linear Model IV

Abstract

In this chapter we take up the problems occasioned by the failure of the rank condition (for the matrix of explanatory variables). This problem arises as a matter of course in analysis of variance (or covariance) models where some of the variables are classificatory. In this case, we are led to the construction of “dummy” variables representing the classificatory schemes. Since all such classificatory schemes are exhaustive, it is not surprising that the “dummy” variables are linearly dependent and, thus, the rank condition for the data matrix fails.

Appendix: Modern Regression Analysis, the Case of Least Angle Regression

There are many modern approaches to identifying the best-subsets of variables for linear regression models. Weighted latent root regression model first applied in Bloch et al. [34], and later validated in Guerard, Rachev, and Shao [161] was introduced in Chap. 4. Tibshirayi [304] reported variable selection modeling using the LASSO technique. Hastie, Johnstone, and Tibshirayi [174] introduced the Least Angle Regression (LAR) technique for variable selection. LASSO and LAR techniques are widely used and will make reasonable and relevant benchmarks for analyzing composite model building techniques. Hastie, Johnstone, and Tibshirayi [174] introduce LAR to the reader by discussing automatic model-building algorithms, including forward selection, all subsets, and back elimination. One can measure the goodness of fit in terms of predictive accuracy, but we will use a different manner, how well model subsets perform in terms of portfolio geometric means using out-of-sample variable weights. LAR is a variation of forward selection; that is the technique selects the variable with the largest absolute correlation, x_j1, with the response variable, y and performs simple linear regression of y on x_j1. The regression produces a residual vector orthogonal to x_j1, now considered to be the response, or dependent variable. One projects the other predictor variables orthogonally to x_j1, and repeat the selection process. The application of K steps produces a set of predictor variables, x_j1, x_j2, x_j3, …, x_jk to construct a K-parameter linear model. Hastie et al. [174] state that forward selection is an aggressive fitting technique that can be overly greedy, eliminating at a second step useful prediction correlated with x_j1.

Forward Stagewise regression, is a cautious version of forward selection and is geometrically related to LASSO. All variables are standardized, to be zero mean and unit variance, or

$$ {\sum}_{i=1}^n{y}_i=0,{\sum}_{i=1}^n{x}_{ij}=0,{\sum}_{i=1}^n{x}_{ij}^z=1, $$

(A.1)

A prediction vector, m, can be written as an X_nx_m matrix.

$$ \widehat{\mu}={\sum}_{j=1}^m{x}_j{\widehat{\beta}}_j=X\widehat{\beta}, $$

(A.2)

The total squared error is

$$ S\left(\widehat{\beta}\right)={\left\Vert y-\widehat{\mu}\right\Vert}^2={\sum}_{i=1}^n{\left({y}_1-{\mu}_i\right)}^2 $$

(A.3)

If $\mathrm{T}\left(\widehat{\beta}\right)$ is the absoluter norm of $\widehat{\beta}$, then

$$ \mathrm{T}\left(\widehat{\beta}\right)={\sum}_{j=1}^{nm}\left|{\widehat{\beta}}_j\right| $$

(A.4)

The LASSO chooses $\widehat{\beta}$ by minimizing $\mathrm{S}\left(\widehat{\beta}\right)$ subject to a bound on $\mathrm{T}\left(\widehat{\beta}\right)$ The LASSO tends to shrink the OLS coefficients to zero, the shrinkage improves predictive accuracy, and LASSO is parsimonious, with only a subset of covariates have non-zero values of ${\widehat{\beta}}_i$ ${\widehat{\beta}}_j$. Stagewise, or forward Stagewise linear regression, is an iterative process that begins with $\widehat{\mu}=0$ and builds the regression function in successive small steps. If $c\left(\widehat{\mu}\right)$ is the vector of current correlations,

$$ \widehat{c}=c\left(\widehat{\mu}\right)={x}^1\left(y-\widehat{\mu}\right) $$

(A.5)

Then the next step of the Stagewise algorithm is taken in the direction of the greatest current correlation

$$ \widehat{J}=\arg \max \mid {\widehat{c}}_j\mid \mathrm{and}\;\widehat{\mu}\to \widehat{\mu}+\upvarepsilon \operatorname {sign}\;\left({\widehat{c}}_j\right)\cdot {x}_j. $$

(A.6)

The Stagewise procedure does not lead to the “big” choice $\upvarepsilon =\mid {\widehat{c}}_j\mid$ the Classic Forward Selection technique, but rather to a “small” choice, which is not overly greedy.

LAR is a version of the Stagewise procedure to accelerate calculations. All coefficients start as equal to zero. LAR is similar to Forward Selection in that it finds the predictor most correlated with the response.

LAR next proceeds to find the predictor with as much correlation with the current residual. LAR finds in an equiangular direction between the first two predictors and a third predictor that is equally-correlated with the “most correlated” set, along the “least angle direction.” LAR creates a regression model, one covariate at a step, such that after K steps, only K of the ${\widehat{\beta}}_j$s are non-zero.

$$ {\widehat{\mu}}_1={\widehat{\mu}}_0+{\widehat{\gamma}}_1{x}_1 $$

(A.7)

LAR uses ${\widehat{\gamma}}_1$ ${\widehat{\gamma}}_1$ such that ${\overline{y}}_2-\widehat{\mu}$ is equally correlated with x ₁ x ₁ and x ₂ x ₂. Efron et al. (2004) demonstrate that ${\overline{y}}_2-{\widehat{\mu}}_1$ ${\overline{y}}_2-$bisects the angle between x ₁ x ₁ and x ₂ such that ${c}_1\left({\widehat{\mu}}_1\right)={c}_2\left({\widehat{\mu}}_1\right)$.

$$ {\widehat{\mu}}_2={\widehat{\mu}}_1+{\widehat{\gamma}}_2{u}_2, $$

(A.8)

where u ₂ u ₂ is the unit vector lying along the bisector.The LAR subsequent steps are taken along equiangular vectors. The covariate vectors are linearly independent.

LAR begins at ${\widehat{\mu}}_0=0$ with the Stagewise procedure. Each step builds $\widehat{\mu}$. If the current ${\widehat{\mu}}_A$ is $\widehat{c}={x}^1\left(y-{\widehat{\mu}}_A\right)$ and is the current correlations vector. The active set A is the set of indices corresponding to

$$ \widehat{c}=\max \left\{|{\widehat{c}}_j|\right\}\mathrm{and}\;\mathrm{A}=\left\{j|{\widehat{c}}_j|=\widehat{c}\right\} $$

(A.9)

Let ${s}_j=\operatorname{sign}\left\{{\widehat{c}}_j\right\}$ for j ϵ A.The inner product vector, a, is:

$$ \mathrm{a}\equiv {x}^1{u}_A $$

The next LAR step update is:

$$ {\widehat{\mu}}_{A+}={\widehat{\mu}}_A+{\widehat{\gamma}}_{\mu A} $$

where

$$ \widehat{\gamma}={\min}_{\mathrm{j}\in \mathrm{A}}+\left\{\frac{c-{\widehat{c}}_j}{A_A-{a}_j},\frac{c-{\widehat{c}}_j}{A_A-{a}_j}\right\} $$

(A.10)

$$ {\displaystyle \begin{array}{lll}\upmu \left(\upgamma \right)& =& {\widehat{\mu}}_A+{\gamma}_1{\mu}_A\\ {}{c}_j\left(\gamma \right)& =& {x}_j\left(y-\mu \left(\gamma \right)\right)={\widehat{c}}_j-{\widehat{\gamma}}_{aj}\\ {}\mid {c}_j\left(\gamma \right)\mid & =& \widehat{c}-\gamma {A}_A\end{array}} $$

(A.11)

Thus, $\widehat{\gamma}$ is the smallest positive value of γ such that the new index j joins the active set, A_+. The new maximum absolute correlation is ${c}_{+}=\widehat{c}-\widehat{\gamma}{A}_A$.

$$ {\displaystyle \begin{array}{l}{\overline{y}}_k={\widehat{\mu}}_{k-1}+{Y}_k{J}_k^{-1}{X}_k^1\left(y-{\widehat{\mu}}_{k-1}\right)+\frac{{\widehat{c}}_k}{A_k}{U}_k\\ {}{X}_k^1\left(y-{\mu}_{k-1}^a\right)={\widehat{C}}_k{I}_A\\ {}{\widehat{\gamma}}_k=\frac{{\widehat{c}}_k}{A_k}\\ {}{\widehat{\mu}}_k-{\widehat{\mu}}_{k-1}=\frac{{\widehat{\gamma}}_k}{{\overline{\gamma}}_k}\left({\overline{y}}_k-{\widehat{\mu}}_{k-1}\right)\end{array}} $$

Because ${\widehat{\gamma}}_k$ is less than ${\overline{\gamma}}_k$, ${\overline{\mu}}_k$ lies closer to ${\overline{y}}_k$ to ${\widehat{\mu}}_{k-1}$. Successive LAR steps approach but do not reach the OLS estimate ${\overline{y}}_k$. LAR is computationally thrifty.