Shrinkage Ridge Regression Estimators in High-Dimensional Linear Models

Abstract

In this paper, we suggest shrinkage ridge regression estimators for a multiple linear regression model, and compared their performance with some penalty estimators which are lasso, adaptive lasso and SCAD. Monte Carlo studies were conducted to compare the estimators and a real data example is presented to illustrate the usefulness of the suggested methods.

Appendix

The distributions of $\varvec{\widehat{\beta }}_{1}^{RFM}$ and $\varvec{\widehat{\beta }}_{1}^{RSM}$ are given by:

$$\begin{aligned} \vartheta _{1}= & {} \sqrt{n}\left( \varvec{\widehat{\beta }}_{1}^{RFM}- \varvec{\beta }_{1}\right) \overset{d}{\rightarrow }N _{p_{1}}\left( -\varvec{\mu }_{11.2},\sigma ^{2}\varvec{Q} _{11.2}^{-1}\right) , \\ \text { and}~~ \vartheta _{2}= & {} \sqrt{n}\left( \varvec{\widehat{\beta }}_{1}^{RSM}- \varvec{\beta }_{1}\right) \overset{d}{\rightarrow }N _{p_{1}}\left( -\varvec{\varvec{\gamma }},\sigma ^{2}\varvec{Q} _{11}^{-1}\right) , \end{aligned}$$

where “$\overset{d}{\rightarrow }$” denotes convergence in distribution, $\varvec{\varvec{\gamma }}=\varvec{\mu }_{11.2}+\varvec{\delta }$ and $\varvec{\delta }$ $=\varvec{Q}_{11}^{-1}\varvec{Q}_{12} \varvec{\omega } .$

To obtain the relationship between sub-model and full model estimators of $\varvec{\beta }_{1}$, we use following equation by using $\varvec{ \widetilde{y}}=\varvec{y}-\varvec{X}_{2}\varvec{\widehat{\beta }} _{2}^{RFM}.$

$$\begin{aligned} \varvec{\widehat{\beta }}_{1}^{RFM}= & {} \underset{\varvec{\varvec{\beta }_{1}}}{\arg \min }\left\{ \left\| \varvec{\widetilde{y}}-\varvec{X}_{1} \varvec{\beta }_{1}\right\| +\lambda ^{R}\left\| \varvec{\beta }_{1}\right\| ^{2}\right\} \nonumber \\= & {} \left( \varvec{X}_{1}^{\top }\varvec{X}_{1}+\lambda ^{R} \varvec{I}_{p_{1}}\right) ^{-1}\varvec{X}_{1}^{\top }\varvec{ \widetilde{y}} \nonumber \\= & {} \left( \varvec{X}_{1}^{\top }\varvec{X}_{1}+\lambda ^{R} \varvec{I}_{p_{1}}\right) ^{-1}\varvec{X}_{1}^{\top }\varvec{y} -\left( \varvec{X}_{1}^{\top }\varvec{X}_{1}+\lambda ^{R}\varvec{ I}_{p_{1}}\right) ^{-1}\varvec{X}_{1}^{\top }\varvec{X}_{2} \varvec{\widehat{\beta }}_{2}^{RFM} \nonumber \\= & {} \varvec{\widehat{\beta }}_{1}^{RSM}-\left( \varvec{X}_{1}^{\top } \varvec{X}_{1}+\lambda ^{R}\varvec{I}_{p_{1}}\right) ^{-1} \varvec{X}_{1}^{\top }\varvec{X}_{2}\varvec{\widehat{\beta }} _{2}^{RFM}. \end{aligned}$$

(6)

Proof (Proof of Theorem 2 )

From the definition of ADB,

$$\begin{aligned} ADB\left( \varvec{\widehat{\beta }}_{1}^{RFM}\right)= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{\widehat{ \beta }}_{1}^{RFM}-\varvec{\beta }_{1}\right) \right\} \\= & {} -\varvec{\mu }_{11.2} \text {.} \end{aligned}$$

To verify the asymptotic bias of $\varvec{\widehat{\beta }}_{1}^{RSM}$, we use the Eq. (6). Hence, it can be written as follows:

$$\begin{aligned} ADB\left( \varvec{\widehat{\beta }}_{1}^{RSM} \right)= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{ \widehat{\beta }}_{1}^{RSM} -\varvec{\beta }_{1}\right) \right\} \\= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{ \widehat{\beta }}_{1}^{RFM} -\varvec{Q}_{11}^{-1}\varvec{Q}_{12} \varvec{\widehat{\beta }}_{2}^{RFM} -\varvec{\beta }_{1}\right) \right\} \\= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{ \widehat{\beta }}_{1}^{RFM} -\varvec{\beta }_{1}\right) \right\} -E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{ Q}_{11}^{-1}\varvec{Q}_{12}\varvec{\widehat{\beta }}_{2}^{RFM} \right) \right\} \\= & {} -\varvec{\mu }_{11.2}-\varvec{Q}_{11}^{-1}\varvec{Q} _{12}\varvec{\omega } \\= & {} -\left( \varvec{\mu }_{11.2}+\varvec{\delta }\right) \\= & {} -\varvec{\gamma } . \end{aligned}$$

Proof (Proof of Theorem 3 )

Firstly, the asymptotic covariance of $\varvec{\widehat{\beta }}_{1}^{RFM}$ is given by:

$$\begin{aligned} \varGamma \left( \varvec{\widehat{\beta }}_{1}^{RFM} \right)= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{\widehat{ \beta }}_{1}^{RFM} -\varvec{\beta }_{1}\right) \sqrt{n}\left( \varvec{\widehat{ \beta }}_{1}^{RFM} -\varvec{\beta }_{1}\right) ^{\top }\right\} \\= & {} E\left( \vartheta _{1}\vartheta _{1}^{\top }\right) \\= & {} Cov\left( \vartheta _{1}\vartheta _{1}^{\top }\right) +E\left( \vartheta _{1}\right) E\left( \vartheta _{1}^{\top }\right) \\= & {} \sigma ^{2}\varvec{Q}_{11.2}^{-1}+\varvec{\mu }_{11.2}\varvec{ \mu }_{11.2}^{\top }. \end{aligned}$$

The asymptotic covariance of $\varvec{\widehat{\beta }}_{1}^{RSM}$ is given by:

$$\begin{aligned} \varGamma \left( \varvec{\widehat{\beta }}_{1}^{RSM} \right)= & {} E\left\{ \underset{n\rightarrow \infty }{\lim }\sqrt{n}\left( \varvec{\widehat{\beta }} _{1}^{RSM} -\varvec{\beta }_{1}\right) \sqrt{n}\left( \varvec{\widehat{\beta }} _{1}^{RSM} -\varvec{\beta }_{1}\right) ^{\top }\right\} \\= & {} E\left( \vartheta _{2}\vartheta _{2}^{\top }\right) \\= & {} Cov\left( \vartheta _{2}\vartheta _{2}^{\top }\right) +E\left( \vartheta _{2}\right) E\left( \vartheta _{2}^{\top }\right) \\= & {} \sigma ^{2}\varvec{Q}_{11}^{-1}+\varvec{\gamma \gamma }^{\top }, \end{aligned}$$

By using the Eq. (3),

$$\begin{aligned} R\left( \varvec{\widehat{\beta }}_{1}^{RFM}\right)= & {} \sigma ^{2}tr\left( \varvec{WQ}_{11.2}^{-1}\right) +\varvec{\mu } _{11.2}^{\top }\varvec{W\mu }_{11.2}, \\ R\left( \varvec{\widehat{\beta }}_{1}^{RSM}\right)= & {} \sigma ^{2}\text {tr }\left( \varvec{WQ}_{11}^{-1}\right) +\varvec{\varvec{\gamma }} ^{\top }\varvec{W\varvec{\gamma }}. \end{aligned}$$