Bounds on the Prediction Error of Penalized Least Squares Estimators with Convex Penalty
This paper considers the penalized least squares estimator with arbitrary convex penalty. When the observation noise is Gaussian, we show that the prediction error is a subgaussian random variable concentrated around its median. We apply this concentration property to derive sharp oracle inequalities for the prediction error of the LASSO, the group LASSO, and the SLOPE estimators, both in probability and in expectation. In contrast to the previous work on the LASSO-type methods, our oracle inequalities in probability are obtained at any confidence level for estimators with tuning parameters that do not depend on the confidence level. This is also the reason why we are able to establish sparsity oracle bounds in expectation for the LASSO-type estimators, while the previously known techniques did not allow for the control of the expected risk. In addition, we show that the concentration rate in the oracle inequalities is better than it was commonly understood before.
KeywordsPenalized least squares Oracle inequality LASSO estimator SLOPE estimator Group LASSO
This work was supported by GENES and by the French National Research Agency (ANR) under the grants IPANEMA (ANR-13-BSH1-0004-02) and Labex Ecodec (ANR-11-LABEX-0047). It was also supported by the “Chaire Economie et Gestion des Nouvelles Donné es”, under the auspices of Institut Louis Bachelier, Havas-Media and Paris-Dauphine.
- 2.Bellec, P.C., Lecué, G., Tsybakov, A.B.: Slope Meets Lasso: Improved Oracle Bounds and Optimality (2016). arXiv:1605.08651
- 6.Dalalyan, A.S., Hebiri, M., Lederer, J.: On the Prediction Performance of the Lasso (2014). arXiv:1402.1700
- 7.Giraud. C.: Introduction to High-dimensional Statistics, vol. 138. CRC Press, Boca Raton (2014)Google Scholar
- 8.Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms I: Fundamentals. Springer, Berlin (1993)Google Scholar
- 10.Lifshits, M.: Lectures on Gaussian Processes. Springer, Berlin (2012)Google Scholar
- 13.Micchelli, C.A., Morales, J.M., Pontil, M.: A family of penalty functions for structured sparsity. Adv. Neural. Inf. Process. Syst. NIPS 23, 2010 (2010)Google Scholar
- 17.van de Geer, S., Wainwright, M.: On Concentration for (Regularized) Empirical Risk Minimization (2015). arXiv:1512.00677