Skip to main content
SpringerLink
Log in

On Motzkin’s method for inconsistent linear systems

  • Published:
BIT Numerical Mathematics Aims and scope Submit manuscript

Abstract

Iterative linear solvers have gained recent popularity due to their computational efficiency and low memory footprint for large-scale linear systems. The relaxation method, or Motzkin’s method, can be viewed as an iterative method that projects the current estimation onto the solution hyperplane corresponding to the most violated constraint. Although this leads to an optimal selection strategy for consistent systems, for inconsistent least square problems, the strategy presents a tradeoff between convergence rate and solution accuracy. We provide a theoretical analysis that shows Motzkin’s method offers an initially accelerated convergence rate and this acceleration depends on the dynamic range of the residual. We quantify this acceleration for Gaussian systems as a concrete example. Lastly, we include experimental evidence on real and synthetic systems that support the analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. We use the convention that an empty sum or product equates to one.

  2. This can be readily verified by observing that the distribution of \(\mathbf {a}_i\) is rotationally invariant and thus \(\left( \mathbf {a}_i^T\frac{\mathbf {x}}{\Vert \mathbf {x}\Vert }\right) ^2\) has the same distribution as \(\left( \mathbf {a}_i^T\mathbf {e}_1\right) ^2\), where \(\mathbf {e}_1\) is the first coordinate vector. Thus it has the same distribution as the ratio of chi-square random variables \(g_1^2/\sum _{i=1}^n g_i^2\), for i.i.d. standard normal \(g_i\). One then applies Slutsky’s theorem to obtain the asymptotic result.

References

  1. Agmon, S.: The relaxation method for linear inequalities. Can. J. Math. 6, 382–392 (1954)

    Article  MathSciNet  MATH  Google Scholar 

  2. Censor, Y.: Row-action methods for huge and sparse systems and their applications. SIAM Rev. 23(4), 444–466 (1981). https://doi.org/10.1137/1023097

    Article  MathSciNet  MATH  Google Scholar 

  3. Chen, X., Powell, A.: Almost sure convergence of the Kaczmarz algorithm with random measurements. J. Fourier Anal. Appl. (2012). https://doi.org/10.1007/s00041-012-9237-2

  4. De Loera, J.A., Haddock, J., Needell, D.: A sampling Kaczmarz–Motzkin algorithm for linear feasibility. SIAM J. Sci. Comput. 39(5), S66–S87 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  5. Eldar, Y.C., Needell, D.: Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss lemma. Numer. Algorithms 58(2), 163–177 (2011). https://doi.org/10.1007/s11075-011-9451-z. 65F20 (65F10); 2835851; Alexander N. Malyshev

    Article  MathSciNet  MATH  Google Scholar 

  6. Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36(4), 1660–1690 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  7. Kaczmarz, S.: Angenäherte auflösung von systemen linearer gleichungen. Bull. Int. Acad. Polon. Sci. Lett. Ser. A, 335–357 (1937)

  8. Leventhal, D., Lewis, A.S.: Randomized methods for linear constraints: convergence rates and conditioning. Math. Oper. Res. 35(3), 641–654 (2010). https://doi.org/10.1287/moor.1100.0456

    Article  MathSciNet  MATH  Google Scholar 

  9. Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36(4), 1590–1604 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  10. Motzkin, T.S., Schoenberg, I.J.: The relaxation method for linear inequalities. Can. J. Math. 6, 393–404 (1954)

    Article  MathSciNet  MATH  Google Scholar 

  11. Needell, D.: Randomized Kaczmarz solver for noisy linear systems. BIT Numer. Math. 50(2), 395–403 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  12. Netlib: The Netlib Linear Programming Library. www.netlib.org/lp. Accessed 7 Nov 2017

  13. Nutini, J., Sepehry, B., Virani, A., Laradji, I., Schmidt, M., Koepke, H.: Convergence rates for greedy Kaczmarz algorithms. Conference on Uncertainty in Artificial Intelligence (2016)

  14. Petra, S., Popa, C.: Single projection Kaczmarz extended algorithms. Numer. Algorithms. (2015). https://doi.org/10.1007/s11075-016-0118-7

  15. Richtárik, P., Takáč, M.: Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program. 144, 1–38 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  16. Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Cornell Aeronaut. Lab. Psychol. Rev. 65(6), 386–408 (1958)

    Google Scholar 

  17. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15(2), 262–278 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  18. Vershynin, R.: In: In: Compressed Sensing, (ed.) Introduction to the non-asymptotic analysis of random matrices, pp. 210–268. University Press, Cambridge (2012)

  19. Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34(2), 773–793 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of California, Davis, Davis, CA, 95616-5270, USA

    Jamie Haddock

  2. Department of Mathematics, University of California, Los Angeles, Los Angeles, CA, 90095-0001, USA

    Deanna Needell

Authors
  1. Jamie Haddock
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deanna Needell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jamie Haddock.

Additional information

Communicated by Lars Eldén.

This material is based upon work supported by the National Science Foundation under Grant No. DMS-1440140 while the authors were in residence at the Mathematical Sciences Research Institute in Berkeley, California, during the Fall 2017 semester. Jamie Haddock was also partially supported by NSF Grant DMS-1522158 and the University of California, Davis Dissertation Fellowship. Deanna Needell was also supported by NSF CAREER award \(\#1348721\) and NSF BIGDATA \(\#1740325\).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haddock, J., Needell, D. On Motzkin’s method for inconsistent linear systems. Bit Numer Math 59, 387–401 (2019). https://doi.org/10.1007/s10543-018-0737-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10543-018-0737-6

Keywords

Mathematics Subject Classification

Search

Navigation