Skip to main content
SpringerLink
Log in

Linear convergence of the randomized sparse Kaczmarz method

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

The randomized version of the Kaczmarz method for the solution of consistent linear systems is known to converge linearly in expectation. And even in the possibly inconsistent case, when only noisy data is given, the iterates are expected to reach an error threshold in the order of the noise-level with the same rate as in the noiseless case. In this work we show that the same also holds for the iterates of the recently proposed randomized sparse Kaczmarz method for recovery of sparse solutions. Furthermore we consider the more general setting of convex feasibility problems and their solution by the method of randomized Bregman projections. This is motivated by the observation that, similarly to the Kaczmarz method, the Sparse Kaczmarz method can also be interpreted as an iterative Bregman projection method to solve a convex feasibility problem. We obtain expected sublinear rates for Bregman projections with respect to a general strongly convex function. Moreover, even linear rates are expected for Bregman projections with respect to smooth or piecewise linear-quadratic functions, and also the regularized nuclear norm, which is used in the area of low rank matrix problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Because very general control sequences besides simple cyclic control fulfill this requirement, the corresponding method was also called method of random Bregman projections in [4]. But such control sequences are not necessarily stochastic objects, in contrast to the situation in the present work. Hence we use the word randomized in Algorithm 3 instead of random to distinguish between the cases.

References

  1. Agaskar, A., Wang, C., Lu, Y.M.: Randomized Kaczmarz algorithms: exact MSE analysis and optimal sampling probabilities. In: IEEE global conference on signal and information processing (GlobalSIP) (2015)

  2. Alber, Y., Butnariu, D.: Convergence of Bregman projection methods for solving consistent convex feasibility problems in reflexive Banach spaces. J. Optim. Theory Appl. 92(1), 33–61 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38(3), 367–426 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bauschke, H.H., Borwein, J.M.: Legendre functions and the method of random Bregman projections. J. Convex Anal. 4(1), 27–67 (1997)

    MathSciNet  MATH  Google Scholar 

  5. Bauschke, H.H., Borwein, J.M., Combettes, P.L.: Bregman monotone optimization algorithms. SIAM J. Control Optim. 42(2), 596–636 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bauschke, H.H., Borwein, J.M., Li, W.: Strong conical hull intersection property, bounded linear regularity, Jameson’s property (G), and error bounds in convex optimization. Math. Program. 86(1), 135–160 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  7. Brand, M.: Fast low-rank modifications of the thin singular value decomposition. Linear Algebra Appl. 415(1), 20–30 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bregman, L.M.: The relaxation method for finding common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7, 200–217 (1967)

    Article  MathSciNet  Google Scholar 

  9. Briskman, J., Needell, D.: Block Kaczmarz method with inequalities. J. Math. Imaging Vis. 52(3), 385–396 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  10. Burger, M.: Bregman distances in inverse problems and partial differential equations. In: Hiriart-Urruty, J.-B., Korytowski, A., Maurer, H., Szymkat, M. (eds.) Advances in Mathematical Modeling, Optimization and Optimal Control, pp. 3–33. Springer (2016)

  11. Byrne, C.: Iterative oblique projection onto convex sets and the split feasibility problem. Inverse Probl. 18, 441–453 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  12. Byrne, C.: A unified treatment of some iterative algorithms in signal processing and image reconstruction. Inverse Probl. 20, 103–120 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  13. Byrne, C., Censor, Y.: Proximity function minimization using multiple Bregman projections, with applications to split feasibility and Kullback–Leibler distance minimization. Ann. Oper. Res. 105, 77–98 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  14. Cai, J.-F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 1956–1982 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  15. Cai, J.-F., Osher, S., Shen, Z.: Convergence of the linearized Bregman iteration for \(\ell _1\)-norm minimization. Math. Comput. 78, 2127–2136 (2009)

    Article  MATH  Google Scholar 

  16. Candès, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  17. Censor, Y., Elfving, T.: A multiprojection algorithm using Bregman projections in a product space. Numer. Alg. 8, 221–239 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  18. Censor, Y., Elfving, T., Kopf, N., Bortfeld, T.: The multiple-sets split feasibility problem and its applications for inverse problems. Inverse Probl. 21, 2071–2084 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  20. Chen, X., Powell, A.M.: Almost sure convergence of the Kaczmarz algorithm with random measurements. J. Fourier Anal. Appl. 18(6), 1195–1214 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  21. Deutsch, F., Hundal, H.: The rate of convergence for the method of alternating projections, II. J. Math. Anal. Appl. 205(2), 381–405 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  22. Elad, M.: Sparse and redundant representations: from theory to applications in signal and image processing. Springer, New York (2010)

    Book  MATH  Google Scholar 

  23. Hansen, P.C., Saxild-Hansen, M.: AIR tools—a MATLAB package of algebraic iterative reconstruction methods. J. Comput. Appl. Math. 236(8), 2167–2178 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. Hoffman, A.J.: On approximate solutions of systems of linear inequalities. J. Res. Natl. Bur. Stand. 49(4), 263–265 (1952)

    Article  MathSciNet  Google Scholar 

  25. Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichungen. Bulletin International de l’Academie Polonaise des Sciences et des Lettres 35, 355–357 (1937)

    MATH  Google Scholar 

  26. Lai, M.J., Yin, W.: Augmented \(\ell _1\) and nuclear-norm models with a globally linearly convergent algorithm. SIAM J. Imaging Sci. 6(2), 1059–1091 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  27. Leventhal, D., Lewis, A.S.: Randomized methods for linear constraints: convergence rates and conditioning. Math. Oper. Res. 35(3), 641–654 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  28. Lorenz, D.A., Schöpfer, F., Wenger, S.: The linearized Bregman method via split feasibility problems: analysis and generalizations. SIAM J. Imaging Sci. 7(2), 1237–1262 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  29. Lorenz, D.A., Wenger, S., Schöpfer, F., Magnor, M.: A sparse Kaczmarz solver and a linearized Bregman method for online compressed sensing. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 1347–1351. IEEE (2014)

  30. Mansour, H., Yilmaz, O.: A fast randomized Kaczmarz algorithm for sparse solutions of consistent linear systems. arXiv preprint arXiv:1305.3803 (2013)

  31. Needell, D.: Randomized Kaczmarz solver for noisy linear systems. BIT Numer. Math. 50(2), 395–403 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  32. Needell, D., Srebro, N., Ward, R.: Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. Math. Program. 155(1), 549–573 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  33. Needell, D., Tropp, J.A.: Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl. 441, 199–221 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  34. Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  35. Petra, S.: Randomized sparse block Kaczmarz as randomized dual block-coordinate descent. Analele Stiintifice Ale Universitatii Ovidius Constanta-Seria Matematica 23(3), 129–149 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  36. Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  37. Richtárik, P., Takáč, M.: Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program. 144(1–2), 1–38 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  38. Robinson, S.M.: Some continuity properties of polyhedral multifunctions. Math. Program. Study 14, 206–214 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  39. Rockafellar, R.T., Wets, R.J.-B.: Variational analysis. Springer, Berlin (2009)

    MATH  Google Scholar 

  40. Schöpfer, F.: Exact regularization of polyhedral norms. SIAM J. Optim. 22(4), 1206–1223 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  41. Schöpfer, F.: Linear convergence of descent methods for the unconstrained minimization of restricted strongly convex functions. SIAM J. Optim. 26(3), 1883–1911 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  42. Schöpfer, F., Schuster, T., Louis, A.K.: An iterative regularization method for the solution of the split feasibility problem in Banach spaces. Inverse Probl. 24(5), 055008 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  43. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15(2), 262–278 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  44. Wang, M., Bertsekas, D.P.: Stochastic first-order methods with random constraint projection. SIAM J. Optim. 26(1), 681–717 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  45. Yin, W.: Analysis and generalizations of the linearized Bregman method. SIAM J. Imaging Sci. 3(4), 856–877 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  46. Zhang, H., Hui Cai, J.F., Cheng, L., Zhu, J.: Strongly convex programming for exact matrix completion and robust principal component analysis. Inverse Probl. Imaging 6(2), 357–372 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  47. Zhang, H., Cheng, L.: Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization. Optim. Lett. 9(5), 961–979 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  48. Zhao, J., Yang, Q.: Several solution methods for the split feasibility problem. Inverse Probl. 21, 1791–1799 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  49. Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34(2), 773–793 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institut für Mathematik, Carl von Ossietzky Universität Oldenburg, 26111, Oldenburg, Germany

    Frank Schöpfer

  2. Institute for Analysis and Algebra, TU Braunschweig, 38092, Braunschweig, Germany

    Dirk A. Lorenz

Authors
  1. Frank Schöpfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dirk A. Lorenz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dirk A. Lorenz.

Additional information

The work of D.L. was partially supported by the National Science Foundation under Grant DMS-1127914 to the Statistical and Applied Mathematical Sciences Institute. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (m 0 KB)

Supplementary material 2 (m 7 KB)

Supplementary material 3 (m 9 KB)

Supplementary material 4 (m 13 KB)

Supplementary material 5 (m 1 KB)

Supplementary material 6 (m 1 KB)

Supplementary material 7 (m 2 KB)

Supplementary material 8 (m 1 KB)

Supplementary material 9 (m 7 KB)

Supplementary material 10 (m 6 KB)

Supplementary material 11 (m 3 KB)

Supplementary material 12 (m 5 KB)

Supplementary material 13 (m 10 KB)

Supplementary material 14 (m 2 KB)

Supplementary material 15 (m 18 KB)

Supplementary material 16 (m 0 KB)

Supplementary material 17 (m 8 KB)

Supplementary material 18 (m 8 KB)

Supplementary material 19 (m 17 KB)

Supplementary material 20 (m 8 KB)

Supplementary material 21 (m 19 KB)

Supplementary material 22 (m 1 KB)

Supplementary material 23 (m 19 KB)

Supplementary material 24 (m 1 KB)

Supplementary material 25 (m 8 KB)

Supplementary material 26 (m 20 KB)

Supplementary material 27 (m 2 KB)

Supplementary material 28 (m 2 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schöpfer, F., Lorenz, D.A. Linear convergence of the randomized sparse Kaczmarz method. Math. Program. 173, 509–536 (2019). https://doi.org/10.1007/s10107-017-1229-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-017-1229-1

Keywords

Mathematics Subject Classification

Search

Navigation