Advertisement

Numerische Mathematik

, Volume 142, Issue 3, pp 667–711 | Cite as

Correcting for unknown errors in sparse high-dimensional function approximation

  • Ben Adcock
  • Anyi Bao
  • Simone BrugiapagliaEmail author
Article

Abstract

We consider sparsity-based techniques for the approximation of high-dimensional functions from random pointwise evaluations. To date, almost all the works published in this field contain some a priori assumptions about the error corrupting the samples that are hard to verify in practice. In this paper, we instead focus on the scenario where the error is unknown. We study the performance of four sparsity-promoting optimization problems: weighted quadratically-constrained basis pursuit, weighted LASSO, weighted square-root LASSO, and weighted LAD-LASSO. From the theoretical perspective, we prove uniform recovery guarantees for these decoders, deriving recipes for the optimal choice of the respective tuning parameters. On the numerical side, we compare them in the pure function approximation case and in applications to uncertainty quantification of ODEs and PDEs with random inputs. Our main conclusion is that the lesser-known square-root LASSO is better suited for high-dimensional approximation than the other procedures in the case of bounded noise, since it avoids (both theoretically and numerically) the need for parameter tuning.

Mathematics Subject Classification

65D15 41A10 94A20 

Notes

Acknowledgements

BA, AB and SB acknowledge the Natural Sciences and Engineering Research Council of Canada through Grant 611675 and the Alfred P. Sloan Foundation and the Pacific Institute for the Mathematical Sciences (PIMS) Collaborative Research Group “High-Dimensional Data Analysis”. SB acknowledges the support of the PIMS Post-doctoral Training Center in Stochastics. The authors are grateful to Claire Boyer, John Jakeman, Richard Lockhart, Akil Narayan, and Clayton G. Webster for interesting and fruitful discussions.

References

  1. 1.
    Adcock, B.: Infinite-dimensional compressed sensing and function interpolation. Found. Comput. Math. 18(3), 661–701 (2018)Google Scholar
  2. 2.
    Adcock, B.: Infinite-dimensional \(\ell ^1\) minimization and function approximation from pointwise data. Constr. Approx. 45(3), 345–390 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Adcock, B., Bao, A., Narayan, A., Author, U.: Compressed sensing with sparse corruptions: fault-tolerant sparse collocation approximations (2017). arXiv:1703.00135
  4. 4.
    Adcock, B., Brugiapaglia, S., Webster, C.G.: Compressed sensing approaches for polynomial approximation of high-dimensional functions (2017). arXiv:1703.06987
  5. 5.
    Adcock, B., Hansen, A.C., Poon, C., Roman, B.: Breaking the coherence barrier: a new theory for compressed sensing. Forum Math. Sigma 5, E4. (2017).  https://doi.org/10.1017/fms.2016.32
  6. 6.
    Arlot, S., Celisse, A.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Arslan, O.: Weighted LAD-LASSO method for robust parameter estimation and variable selection in regression. Comput. Stat. Data Anal. 56(6), 1952–1965 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Babu, P., Stoica, P.: Connection between spice and square-root lasso for sparse parameter estimation. Signal Process. 95, 10–14 (2014)CrossRefGoogle Scholar
  9. 9.
    Bäck, J., Nobile, F., Tamellini, L., Tempone, R.: Stochastic spectral galerkin and collocation methods for pdes with random coefficients: A numerical comparison. In: Hesthaven, J.S., Rønquist, E.M. (eds.) Spectral and High Order Methods for Partial Differential Equations: Selected Papers from the ICOSAHOM ’09 Conference. June 22–26, Trondheim, Norway, pp. 43–62. Springer, Berlin (2011)Google Scholar
  10. 10.
    Ballani, J., Grasedyck, L.: Hierarchical tensor approximation of output quantities of parameter-dependent pdes. SIAM/ASA J. Uncertain. Quantif. 3(1), 852–872 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Bastounis, A., Hansen, A.C.: On the absence of the RIP in real-world applications of compressed sensing and the RIP in levels (2014). arXiv:1411.4449
  12. 12.
    Belloni, A., Chernozhukov, V., Wang, L.: Square-root lasso: pivotal recovery of sparse signals via conic programming. Biometrika 98(4), 791–806 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Belloni, A., Chernozhukov, V., Wang, L.: Pivotal estimation via square-root lasso in nonparametric regression. Ann. Stat. 42(2), 757–788 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Bridges, P.G., Ferreira, K.B., Heroux, M.A., Hoemmen, M.: Fault-tolerant linear solvers via selective reliability (2012). arXiv:1206.1390
  15. 15.
    Brugiapaglia, S., Adcock, B.: Robustness to unknown error in sparse regularization (2017). arXiv:1705.10299
  16. 16.
    Brugiapaglia, S., Adcock, B., Archibald, R.K.: Recovery guarantees for compressed sensing with unknown errors. In: 2017 International Conference on Sampling Theory and Applications (SampTA). IEEE (2017)Google Scholar
  17. 17.
    Bunea, F., Lederer, J., She, Y.: The group square-root lasso: theoretical properties and fast algorithms. IEEE Trans. Inform. Theory 60(2), 1313–1325 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory 52(2), 489–509 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell ^1\) minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  20. 20.
    Chkifa, A., Cohen, A., Migliorati, G., Nobile, F., Tempone, R.: Discrete least squares polynomial approximation with random evaluations—application to parametric and stochastic elliptic pdes. ESAIM Math. Model. Numer. Anal. 49(3), 815–837 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Chkifa, A., Cohen, A., Schwab, C.: High-dimensional adaptive sparse polynomial interpolation and applications to parametric PDEs. Found. Comput. Math. 14(4), 601–633 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Chkifa, A., Cohen, A., Schwab, C.: Breaking the curse of dimensionality in sparse polynomial approximation of parametric PDEs. J. Math. Pures Appl. 103(2), 400–428 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Chkifa, A., Dexter, N., Tran, H., Webster, C.G.: Polynomial approximation via compressed sensing of high-dimensional functions on lower sets. Math. Comput. 87(311), 1415–1450 (2018)Google Scholar
  24. 24.
    Cohen, A., DeVore, R., Schwab, C.: Convergence rates of best N-term Galerkin approximations for a class of elliptic sPDEs. Found. Comput. Math. 10(6), 615–646 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Cohen, A., DeVore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s. Anal. Appl. 9(01), 11–47 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    de Boor, C., Ron, A.: On multivariate polynomial interpolation. Constr. Approx. 6(3), 287–302 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  27. 27.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inform. Theory 52(4), 1289–1306 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Donoho, D.L., Logan, B.F.: Signal recovery and the large sieve. SIAM J. Appl. Math. 52(2), 577–591 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  29. 29.
    Doostan, A., Owhadi, H.: A non-adapted sparse approximation of PDEs with stochastic inputs. J. Comput. Phys. 230(8), 3015–3034 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Dyn, N., Floater, M.S.: Multivariate polynomial interpolation on lower sets. J. Approx. Theory 177(Supplement C), 34–42 (2014)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing, Appl. Numer. Harmon. Anal. Springer, New York (2013)zbMATHCrossRefGoogle Scholar
  32. 32.
    Friedlander, M.P., Mansour, H., Saab, R., Yilmaz, O.: Recovering compressively sampled signals using partial support information. IEEE Trans. Inform. Theory 58(2), 1122–1134 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Gao, X.: Penalized methods for high-dimensional least absolute deviations regression. Ph.D. Thesis, The University of Iowa (2008)Google Scholar
  34. 34.
    Gao, X., Huang, J.: Asymptotic analysis of high-dimensional lad regression with lasso. Stat. Sin. 20(4), 1485–1506 (2010)Google Scholar
  35. 35.
    Grant, M., Boyd, S.: Graph implementations for nonsmooth convex programs. In: Blondel, V., Boyd, S., Kimura, H. (eds.) Recent Advances in Learning and Control. Lecture Notes in Control and Information Sciences, pp. 95–110. Springer-Verlag Limited (2008)Google Scholar
  36. 36.
    Grant, M., Boyd, S.: CVX: Matlab software for disciplined convex programming, version 2.1. http://cvxr.com/cvx, March (2014)
  37. 37.
    Hastie, T., Tibshirani, R., Wainwright, M.: Statistical Learning with Sparsity: the Lasso and Generalizations. CRC Press, Boca Raton (2015)zbMATHCrossRefGoogle Scholar
  38. 38.
    Hecht, F.: New development in FreeFem++. J. Numer. Math. 20(3–4), 251–265 (2012)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Jakeman, J.D., Eldred, M.S., Sargsyan, K.: Enhancing \(\ell ^1\)-minimization estimates of polynomial chaos expansions using basis selection. J. Comput. Phys. 289, 18–34 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Laska, J.N., Davenport, M.A., Baraniuk, R.G.: Exact signal recovery from sparsely corrupted measurements through the pursuit of justice. In: 2009 Conference Record of the 43rd Asilomar Conference on Signals, Systems and Computers, pp. 1556–1560. IEEE (2009)Google Scholar
  41. 41.
    Li, Q., Wang, L.: Robust change point detection method via adaptive lad-lasso. Stat. Pap. 1–13 (2017).  https://doi.org/10.1007/s00362-017-0927-3
  42. 42.
    Li, X.: Compressed sensing and matrix completion with constant proportion of corruptions. Constr. Approx. 37(1), 73–99 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Logan, B.F.: Properties of high-pass signals. Ph.D. Thesis, Columbia University (1965)Google Scholar
  44. 44.
    Lorentz, G.G., Lorentz, R.A.: Solvability problems of bivariate interpolation I. Constr. Approx. 2(1), 153–169 (1986)MathSciNetzbMATHCrossRefGoogle Scholar
  45. 45.
    Migliorati, G., Nobile, F., von Schwerin, E., Tempone, R.: Analysis of discrete \(L^2\) projection on polynomial spaces with random evaluations. Found. Comput. Math. 14(3), 419–456 (2014)MathSciNetzbMATHGoogle Scholar
  46. 46.
    Nguyen, N.H., Tran, T.D.: Exact recoverability from dense corrupted observations via \(\ell _1\)-minimization. IEEE Trans. Inform. Theory 59(4), 2017–2035 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  47. 47.
    Peng, J., Hampton, J., Doostan, A.: A weighted \(\ell _1\) minimization approach for sparse polynomial chaos expansions. J. Comput. Phys. 267, 92–111 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Pham, V., El Ghaoui, L.: Robust sketching for multiple square-root lasso problems. In: Artificial Intelligence Statistics, pp. 753–761 (2015)Google Scholar
  49. 49.
    Rauhut, H., Schwab, C.: Compressive sensing Petrov–Galerkin approximation of high-dimensional parametric operator equations. Math. Comput. 86(304), 661–700 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  50. 50.
    Rauhut, H., Ward, R.: Interpolation via weighted \(\ell _1\) minimization. Appl. Comput. Harmon. Anal. 40(2), 321–351 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  51. 51.
    Shin, Y., Xiu, D.: Correcting data corruption errors for multivariate function approximation. SIAM J. Sci. Comput. 38(4), A2492–A2511 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  52. 52.
    Stankovic, L., Stankovic, S., Amin, M.: Missing samples analysis in signals for applications to l-estimation and compressive sensing. Signal Process. 94, 401–408 (2014)CrossRefGoogle Scholar
  53. 53.
    Stucky, B., van de Geer, S.A.: Sharp oracle inequalities for square root regularization (2015). arXiv:1509.04093
  54. 54.
    Studer, C., Kuppinger, P., Pope, G., Bolcskei, H.: Recovery of sparsely corrupted signals. IEEE Trans. Inform. Theory 58(5), 3115–3130 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  55. 55.
    Su, D.: Compressed sensing with corrupted Fourier measurements (2016). arXiv:1607.04926
  56. 56.
    Su, D.: Data recovery from corrupted observations via l1 minimization (2016). arXiv:1601.06011
  57. 57.
    Sun, T., Zhang, C.-H.: Scaled sparse linear regression. Biometrika 99(4), 879–898 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Tian, X., Loftus, J.R., Taylor, J.E.: Selective inference with unknown variance via the square-root lasso (2015). arXiv:1504.08031
  59. 59.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Methodol. 58(1), 267–288 (1996)Google Scholar
  60. 60.
    van de Geer, S.A.: Estimation and Testing Under Sparsity. Springer, Berlin (2016)zbMATHCrossRefGoogle Scholar
  61. 61.
    Wagener, J., Dette, H.: The adaptive lasso in high-dimensional sparse heteroscedastic models. Math. Methods Stat. 22(2), 137–154 (2013)Google Scholar
  62. 62.
    Wang, H., Li, G., Jiang, G.: Robust regression shrinkage and consistent variable selection through the LAD-Lasso. J. Bus. Econo. Stat. 25(3), 347–355 (2007)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Wright, J., Ma, Y.: Dense error correction via \(\ell ^1\)-minimization. IEEE Trans. Inform. Theory 56(7), 3540–3560 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  64. 64.
    Xu, J.: Parameter estimation, model selection and inferences in L1-based linear regression. Ph.D. Thesis, Columbia University (2005)Google Scholar
  65. 65.
    Xu, J., Ying, Z.: Simultaneous estimation and variable selection in median regression using lasso-type penalty. Ann. Inst. Stat. Math. 62(3), 487–514 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  66. 66.
    Yan, L., Guo, L., Xiu, D.: Stochastic collocation algorithms using \(\ell _1\)-minimization. Int. J. Uncertain. Quantif. 2(3), 279–293 (2012)MathSciNetCrossRefGoogle Scholar
  67. 67.
    Yang, X., Karniadakis, G.E.: Reweighted \(\ell ^1\) minimization method for stochastic elliptic differential equations. J. Comput. Phys. 248, 87–108 (2013)zbMATHCrossRefGoogle Scholar
  68. 68.
    Yu, X., Baek, S.J.: Sufficient conditions on stable recovery of sparse signals with partial support information. IEEE Signal Process. Lett. 20(5), 539–542 (2013)CrossRefGoogle Scholar
  69. 69.
    Zou, H.: The adaptive lasso and its oracle properties. J. Am. Stat. Assoc. 101(476), 1418–1429 (2006)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Simon Fraser UniversityBurnabyCanada
  2. 2.University of British ColumbiaVancouverCanada

Personalised recommendations