Numerical Algorithms

, Volume 79, Issue 1, pp 1–17 | Cite as

Convergence rates for Kaczmarz-type algorithms

  • Constantin PopaEmail author
Original Paper


In this paper, we make a theoretical analysis of the convergence rates of Kaczmarz and extended Kaczmarz projection algorithms for some of the most practically used control sequences. We first prove an at least linear convergence rate for the Kaczmarz-Tanabe and its extended version methods (the one in which a complete set of projections using row/column indices is performed in each iteration). Then, we apply the main ideas of this analysis in establishing an at least sublinear, respectively, linear convergence rate for the Kaczmarz algorithm with almost cyclic and the remotest set control strategies, and their extended versions, respectively. These results complete the existing ones related to the random selection procedures.


Kaczmarz algorithm Extended Kaczmarz algorithm Control sequences Convergence rates 

Mathematics Subject Classification (2010)

65F10 65F20 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The author wants to thank to the anonymous referees for their valuable comments that have much improved the first versions of the paper.


  1. 1.
    Ansorge, R.: Connections between the Cimmino-method and the Kaczmarz-method for the solution of singular and regular systems of equations. Computing 33, 367–375 (1984)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Cegielski, A., Censor, Y.: Projection methods: an annotated bibliography of books and reviews. Optimization 64, 2343–2358 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Censor, Y.: Row-action methods for huge and sparse systems and their applications. SIAM Rev. 23, 444–466 (1981)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Censor, Y., Zenios, S.: Parallel Optimization: Theory, Algorithms and Applications. Oxford Univ. Press, Oxford (1997)zbMATHGoogle Scholar
  5. 5.
    Censor, Y., Chen, W., Pajoohesh, H.: Finite convergence of a subgradient projections method with expanding controls. Appl. Math. Optim. 64, 273–285 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Combettes, P.: Hilbertian convex feasibility problem: convergence of projection methods. Appl. Math. Optim. 35, 311–330 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Gautschi, W.: Numerical analysis, 2nd edn. Birkhäuser, Boston (2012)CrossRefzbMATHGoogle Scholar
  8. 8.
    Gordon, R., Bender, R., Herman, G.T.: Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theor. Biol. 29, 471–481 (1970)CrossRefGoogle Scholar
  9. 9.
    Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36(4), 1660–1690 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Kaczmarz, S.: Angenäherte Auflösung von Systemen linearer Gleichingen. Bull. Intern. Acad. Polonaise Sci. Lett., Cl. Sci. Math. Nat. A 35, 355–357 (1937)zbMATHGoogle Scholar
  11. 11.
    Kaczmarz, S.: Approximate solution of systems of linear equations. Int. J. Control. 57, 1269–1271 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Liang, D., Schön, T.B.: On the exponential convergence of the Kaczmarz algorithm. IEEE Signal Process. Lett. 22(10), 1571–1574 (2015)CrossRefGoogle Scholar
  13. 13.
    Loizou, N., Richtárik, P.: A new perspective on randomized gossip algorithms. In: Proceedings of the 4th IEEE Global conference on signal and information processing (GlobalSIP), pp. 440–444 (2016)Google Scholar
  14. 14.
    Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss–Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36(4), 1590–1604 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Mandel, J.: Convergence of the cyclical relaxation method for linear inequalities. Math. Program. 30(2), 218–228 (1984)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Needell, D., Tropp, J.A.: Paved with good intentions: analysis of a randomized block Kaczmarz method. Lin. Alg. Appl. 441, 199–221 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Nutini, J., et al.: Convergence rates for greedy Kaczmarz algorithms, and faster randomized Kaczmarz rules using the orthogonality graph. In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence. AUAI Press, Corvallis (2016)Google Scholar
  18. 18.
    Oswald, P., Zhou, W.: Convergence analysis for Kaczmarz-type methods in a Hilbert space framework. Linear Algebra Appl. 478, 131–161 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Petra, S., Popa, C.: Single projection Kaczmarz extended algorithms . Numer. Algor. 73(3), 791–806 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Popa, C.: A note on Kaczmarz algorithm with remotest set control sequence. arXiv:1702.02729v1
  21. 21.
    Popa, C.: On remotest set and random controls in Kaczmarz algorithm. arXiv:1704.04660
  22. 22.
    Popa, C.: Least-squares solution of overdetermined inconsistent linear systems using Kaczmarz’s relaxation. Int. J. Comp. Math. 55(1-2), 79–89 (1995)CrossRefzbMATHGoogle Scholar
  23. 23.
    Popa, C.: Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems. BIT Numer. Math. 38(1), 151–176 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Popa, C.: Projection Algorithms—Classical Results and Developments. Applications to Image Reconstruction. Lambert Academic Publishing - AV Akademikerverlag GmbH & Co. KG, Saarbrücken (2012)Google Scholar
  25. 25.
    Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15, 262–278 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Sznader, R.: Kaczmarz algorithm revisited.
  27. 27.
    Tanabe, K.: Projection method for solving a singular system of linear equations and its applications. Numer. Math. 17, 203–214 (1971)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Zouzias, A., Freris, N.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34(2), 773–793 (2013)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Faculty of Mathematics and Computer ScienceOvidius UniversityConstantaRomania
  2. 2.“Gheorghe Mihoc - Caius Iacob” Institute of Statistical Mathematics and Applied Mathematics of the Romanian AcademyBucharestRomania

Personalised recommendations