A Problem in Model Selection of LASSO and Introduction of Scaling

Abstract

In this article, we considered to assign a single scaling parameter to LASSO estimators for investigating and improving a problem of excessive shrinkage at a sparse representation. This problem is important because it directly affects a quality of model selection in LASSO. We derived a prediction risk for LASSO with scaling and obtained an optimal scaling parameter value that minimizes the risk. We then showed the risk is improved by assigning the optimal scaling value. In a numerical example, we found that an estimate of the optimal scaling value is larger than one especially at a sparse representation; i.e. excessive shrinkage is relaxed by expansion via scaling. Additionally, we observed that a risk for LASSO is high at a sparse representation and it is minimized at a relatively large model while this is improved by the introduction of an estimate of the optimal scaling value. We here constructed a fully empirical risk estimate that approximates the actual risk well. We then observed that, by applying the risk estimate as a model selection criterion, LASSO with scaling tends to obtain a model with low risk and high sparsity compared to LASSO without scaling.