Asymptotics for high dimensional regression M-estimates: fixed design results



We investigate the asymptotic distributions of coordinates of regression M-estimates in the moderate p / n regime, where the number of covariates p grows proportionally with the sample size n. Under appropriate regularity conditions, we establish the coordinate-wise asymptotic normality of regression M-estimates assuming a fixed-design matrix. Our proof is based on the second-order Poincaré inequality  and leave-one-out analysis. Some relevant examples are indicated to show that our regularity conditions are satisfied by a broad class of design matrices. We also show a counterexample, namely an ANOVA-type design, to emphasize that the technical assumptions are not just artifacts of the proof. Finally, numerical experiments confirm and complement our theoretical results.


M-estimation Robust regression High-dimensional statistics Second order Poincaré inequality Leave-one-out analysis 

Mathematics Subject Classification

Primary 62J99 Secondary 62E20 

Supplementary material


  1. 1.
    Anderson, T.W.: An Introduction to Multivariate Statistical Analysis. Wiley, New York (1962)Google Scholar
  2. 2.
    Bai, Z., Silverstein, J.W.: Spectral Analysis of Large Dimensional Random Matrices, vol. 20. Springer, Berlin (2010)CrossRefMATHGoogle Scholar
  3. 3.
    Bai, Z., Yin, Y.: Limit of the smallest eigenvalue of a large dimensional sample covariance matrix. Ann. Probab. 21(3), 1275–1294 (1993)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Baranchik, A.: Inadmissibility of maximum likelihood estimators in some multiple regression problems with three or more independent variables. Ann. Stat. 1(2), 312–321 (1973)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Bean, D., Bickel, P.J., El Karoui, N., Lim, C., Yu, B.: Penalized robust regression in high-dimension. Technical Report 813, Department of Statistics, UC Berkeley (2012)Google Scholar
  6. 6.
    Bean, D., Bickel, P.J., El Karoui, N., Yu, B.: Optimal M-estimation in high-dimensional regression. Proc. Natl. Acad. Sci. 110(36), 14563–14568 (2013)CrossRefGoogle Scholar
  7. 7.
    Bickel, P.J., Doksum, K.A.: Mathematical Statistics: Basic Ideas and Selected Topics, Volume I, vol. 117. CRC Press, Boca Raton (2015)MATHGoogle Scholar
  8. 8.
    Bickel, P.J., Freedman, D.A.: Some asymptotic theory for the bootstrap. Ann. Stat. 9(6), 1196–1217 (1981)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Bickel, P.J., Freedman, D.A.: Bootstrapping regression models with many parameters. Festschrift for Erich L. Lehmann pp. 28–48 (1983)Google Scholar
  10. 10.
    Chatterjee, S.: Fluctuations of eigenvalues and second order Poincaré inequalities. Probab. Theory Relat. Fields 143(1–2), 1–40 (2009)CrossRefMATHGoogle Scholar
  11. 11.
    Chernoff, H.: A note on an inequality involving the normal distribution. Ann. Probab. 9(3), 533–535 (1981)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Cizek, P., Härdle, W.K., Weron, R.: Statistical Tools for Finance and Insurance. Springer, Berlin (2005)MATHGoogle Scholar
  13. 13.
    Cochran, W.G.: Sampling Techniques. Wiley, Hoboken (1977)MATHGoogle Scholar
  14. 14.
    David, H.A., Nagaraja, H.N.: Order Statistics. Wiley Online Library, Hoboken (1981)MATHGoogle Scholar
  15. 15.
    Donoho, D., Montanari, A.: High dimensional robust M-estimation: asymptotic variance via approximate message passing. Probab. Theory Relat. Fields 166, 935–969 (2016)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Durrett, R.: Probability: Theory and Examples. Cambridge University Press, Cambridge (2010)CrossRefMATHGoogle Scholar
  17. 17.
    Efron, B.: The Jackknife, the Bootstrap and Other Resampling Plans, vol. 38. SIAM, Philadelphia (1982)CrossRefMATHGoogle Scholar
  18. 18.
    El Karoui, N.: Concentration of measure and spectra of random matrices: applications to correlation matrices, elliptical distributions and beyond. Ann. Appl. Probab. 19(6), 2362–2405 (2009)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    El Karoui, N.: High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: risk underestimation. Ann. Stat. 38(6), 3487–3566 (2010)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    El Karoui, N.: Asymptotic behavior of unregularized and ridge-regularized high-dimensional robust regression estimators: rigorous results. arXiv preprint arXiv:1311.2445 (2013)
  21. 21.
    El Karoui, N.: On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators. Probab. Theory Relat. Fields, pp. 1–81 (2015)Google Scholar
  22. 22.
    El Karoui, N., Bean, D., Bickel, P.J., Lim, C., Yu, B.: On Robust Regression with High-Dimensional Predictors. Technical Report 811, Department of Statistics, UC Berkeley (2011)Google Scholar
  23. 23.
    El Karoui, N., Bean, D., Bickel, P.J., Lim, C., Yu, B.: On robust regression with high-dimensional predictors. Proc. Natl. Acad. Sci. 110(36), 14557–14562 (2013)CrossRefMATHGoogle Scholar
  24. 24.
    El Karoui, N., Purdom, E.: Can We Trust the Bootstrap in High-Dimension? Technical Report 824. Department of Statistics, UC Berkeley (2015)Google Scholar
  25. 25.
    Esseen, C.G.: Fourier analysis of distribution functions. A mathematical study of the Laplace–Gaussian law. Acta Math. 77(1), 1–125 (1945)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Geman, S.: A limit theorem for the norm of random matrices. Ann. Probab. 8(2), 252–261 (1980)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Hanson, D.L., Wright, F.T.: A bound on tail probabilities for quadratic forms in independent random variables. Ann. Math. Stat. 42(3), 1079–1083 (1971)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (2012)CrossRefGoogle Scholar
  29. 29.
    Huber, P.J.: Robust estimation of a location parameter. Ann. Math. Stat. 35(1), 73–101 (1964)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Huber, P.J.: The 1972 wald lecture robust statistics: a review. Ann. Math. Stat. 43(4), 1041–1067 (1972)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Huber, P.J.: Robust regression: asymptotics, conjectures and Monte Carlo. Ann. Stat. 1(5), 799–821 (1973)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Huber, P.J.: Robust Statistics. Wiley, New York (1981)CrossRefMATHGoogle Scholar
  33. 33.
    Johnstone, I.M.: On the distribution of the largest eigenvalue in principal components analysis. Ann. Stat. 29(2), 295–327 (2001)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Jurečkovà, J., Klebanov, L.B.: Inadmissibility of robust estimators with respect to \(L_1\) norm. In: Dodge, Y. (ed.) \(L_1\)-Statistical Procedures and Related Topics. Lecture Notes–Monograph Series, vol. 31, pp. 71–78. Institute of Mathematical Statistics, Hayward (1997)CrossRefGoogle Scholar
  35. 35.
    Latała, R.: Some estimates of norms of random matrices. Proc. Am. Math. Soc. 133(5), 1273–1282 (2005)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Ledoux, M.: The Concentration of Measure Phenomenon, vol. 89. American Mathematical Society, Providence (2001)MATHGoogle Scholar
  37. 37.
    Litvak, A.E., Pajor, A., Rudelson, M., Tomczak-Jaegermann, N.: Smallest singular value of random matrices and geometry of random polytopes. Adv. Math. 195(2), 491–523 (2005)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Mallows, C.: A note on asymptotic joint normality. Ann. Math. Stat. 43(2), 508–515 (1972)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Mammen, E.: Asymptotics with increasing dimension for robust regression with applications to the bootstrap. Ann. Stat. 17(1), 382–400 (1989)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Marčenko, V.A., Pastur, L.A.: Distribution of eigenvalues for some sets of random matrices. Math. USSR Sbornik 1(4), 457 (1967)CrossRefGoogle Scholar
  41. 41.
    Muirhead, R.J.: Aspects of Multivariate Statistical Theory, vol. 197. Wiley, Hoboken (1982)CrossRefMATHGoogle Scholar
  42. 42.
    Portnoy, S.: Asymptotic behavior of M-estimators of \(p\) regression parameters when \(p^{2}/n\) is large. I. Consistency. Ann. Stat. 12(4), 1298–1309 (1984)CrossRefMATHGoogle Scholar
  43. 43.
    Portnoy, S.: Asymptotic behavior of M-estimators of \(p\) regression parameters when \(p^{2} / n\) is large. II. Normal approximation. Ann. Stat. 13(4), 1403–1417 (1985)MATHGoogle Scholar
  44. 44.
    Portnoy, S.: On the central limit theorem in \(\mathbb{R}^{p}\) when \(p\rightarrow \infty \). Probab. Theory Relat. Fields 73(4), 571–583 (1986)CrossRefMATHGoogle Scholar
  45. 45.
    Portnoy, S.: A central limit theorem applicable to robust regression estimators. J. Multivar. Anal. 22(1), 24–50 (1987)MathSciNetCrossRefMATHGoogle Scholar
  46. 46.
    Posekany, A., Felsenstein, K., Sykacek, P.: Biological assessment of robust noise models in microarray data analysis. Bioinformatics 27(6), 807–814 (2011)CrossRefGoogle Scholar
  47. 47.
    Relles, D.A.: Robust Regression by Modified Least-Squares. Technical reports, DTIC Document (1967)Google Scholar
  48. 48.
    Rosenthal, H.P.: On the subspaces of \(l^{p} (p > 2)\) spanned by sequences of independent random variables. Isr. J. Math. 8(3), 273–303 (1970)CrossRefMATHGoogle Scholar
  49. 49.
    Rudelson, M., Vershynin, R.: Smallest singular value of a random rectangular matrix. Commun. Pure Appl. Math. 62(12), 1707–1739 (2009)MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Rudelson, M., Vershynin, R.: Non-asymptotic theory of random matrices: extreme singular values. arXiv preprint arXiv:1003.2990 (2010)
  51. 51.
    Rudelson, M., Vershynin, R.: Hanson-Wright inequality and sub-gaussian concentration. Electron. Commun. Probab. 18(82), 1–9 (2013)MathSciNetMATHGoogle Scholar
  52. 52.
    Scheffe, H.: The Analysis of Variance, vol. 72. Wiley, Hoboken (1999)MATHGoogle Scholar
  53. 53.
    Silverstein, J.W.: The smallest eigenvalue of a large dimensional Wishart matrix. Ann. Probab. 13(4), 1364–1368 (1985)MathSciNetCrossRefMATHGoogle Scholar
  54. 54.
    Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B (Methodolog) 36(2), 111–147 (1974)MathSciNetMATHGoogle Scholar
  55. 55.
    Tyler, D.E.: A distribution-free M-estimator of multivariate scatter. Ann. Stat. 15(1), 234–251 (1987)MathSciNetCrossRefMATHGoogle Scholar
  56. 56.
    Van der Vaart, A.W.: Asymptotic Statistics. Cambridge University Press, Cambridge (1998)CrossRefMATHGoogle Scholar
  57. 57.
    Vershynin, R.: Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 (2010)
  58. 58.
    Wachter, K.W.: Probability plotting points for principal components. In: Ninth Interface Symposium Computer Science and Statistics, pp. 299–308. Prindle, Weber and Schmidt, Boston (1976)Google Scholar
  59. 59.
    Wachter, K.W.: The strong limits of random matrix spectra for sample matrices of independent elements. Ann. Probab. 6(1), 1–18 (1978)MathSciNetCrossRefMATHGoogle Scholar
  60. 60.
    Wasserman, L., Roeder, K.: High dimensional variable selection. Ann. Stat. 37(5A), 2178 (2009)MathSciNetCrossRefMATHGoogle Scholar
  61. 61.
    Yohai, V.J.: Robust M-Estimates for the General Linear Model. Universidad Nacional de la Plata. Departamento de Matematica (1972)Google Scholar
  62. 62.
    Yohai, V.J., Maronna, R.A.: Asymptotic behavior of M-estimators for the linear model. Ann. Stat. 7(2), 258–268 (1979)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of StatisticsUniversity of California, BerkeleyBerkeleyUSA
  2. 2.Criteo ResearchParisFrance

Personalised recommendations