# Remarks on Kaczmarz Algorithm for Solving Consistent and Inconsistent System of Linear Equations

- 180 Downloads

## Abstract

In this paper we consider the classical Kaczmarz algorithm for solving system of linear equations. Based on the geometric relationship between the error vector and rows of the coefficient matrix, we derive the optimal strategy of selecting rows at each step of the algorithm for solving consistent system of linear equations. For solving perturbed system of linear equations, a new upper bound in the convergence rate of the randomized Kaczmarz algorithm is obtained.

## Keywords

Iterative methods Kaczmarz method Convergence rate Orthogonal projection Linear systems## 1 Introduction

*i*th row of

*A*and \(b_i\) is the

*i*th element of vector

*b*.

*i*th equation in (1), i.e., \(a_i^Tx_{k+1}=b_i\). The updating formula (3) implicitly produces a solution to the following constraint optimization problem [21, 37]

By comparing the projection processes displayed in Fig. 1, it is natural to have the intuition that convergence of the classical Kaczmarz algorithm highly depends on the geometric positions of the associated hyperplanes. If the normal vectors of every two successive hyperplanes keep reasonably large angles, the convergence of the classical Kaczmarz algorithm will be fast, whereas two nearly parallel consecutive hyperplanes will make the convergence slow down. The Kaczmarz algorithm can be regarded as a special application of famous von Neumann’s alternating projection [35] originally distributed in 1933. The fundamental idea can even trace the history back to Schwarz [38] in 1870s.

*A*. In some real applications, it is observed [25, 30] that instead of selecting rows of the matrix

*A*sequentially at each step of the Kaczmarz algorithm, randomly selection can often improve its convergence. Recently, in the remarkable paper [39], Strohmer and Vershynin proved the rate of convergence for the following randomized Kaczmarz algorithm

*r*(

*i*) is chosen from \(\{1, 2, \cdots , m\}\) with probabilities \(\frac{||a_{r(i)}||_2^2}{||A||_F^2}\). In particular, the following bound on the expected rate of convergence for the randomized Kaczmarz method is proved

*A*introduced by J. Demmel [14]. Due to this pioneering work that characterized the convergence rate for the randomized Kaczmarz algorithms, the idea stimulated considerable interest in this area and various investigations [1, 2, 6, 10, 15] have been performed recently. In particular, some acceleration strategies have been proposed [6, 16, 22] and convergence analysis was performed in [21, 23, 27, 29, 31, 32]. See also [21, 23] for some comments on equivalent interpretations of the randomized Kaczmarz algorithms.

## 2 Optimal Row Selecting Strategy of the Kaczmarz Algorithm for Solving Consistent System of Linear Equations

*x*is a solution. If the

*i*th row is selected at the \((k+1)\)th iteration of the Kaczmarz algorithm, i.e.,

*x*is the unknown solution, the above minimization problems seems unsolvable. However, noting that consistent linear system (1) implies

*optimal selecting strategy*, and call the Kaczmarz algorithm with the optimal selecting strategy as the

*optimal Kaczmarz algorithm*.

when \(\sin \theta _{p}^{\hat{i}}<1\), the algorithm converge exponentially,

- when \(\sin \theta _{p}^{\hat{i}}= 1\), we haveand thus,$$\max _{j}a_j^T(x_{p}-x)=0$$This implies that \(Ax_{p}=b\), i.e., \(x_{p}\) solves the system of linear equation (1).$$a_j^T(x_{p}-x)=0, j=1,2,\cdots , m.$$

## 3 Randomized Kaczmarz Algorithm for Solving Inconsistent System of Linear Equations

*r*as follows:

Firstly, we recall the Lemma 2.2 in [32].

### Lemma 1

**Remarks:**If the Lemma 1 is used to interpret the Kaczmarz algorithm for solving the perturbed and unperturbed equations, we need to introduce a vector \(a_i^{\perp }\) in the as the orthogonal complement of the vector \(a_i\), and write \(\tilde{x}_i\in \tilde{H}_i\) as

**Example 1.**Consider the \(2\times 2\) system of linear equations

In what follows, we will consider the convergence rate of the randomized Kaczmarz algorithm for solving (16) from a different perspective. We try to bound the difference between the solution for the unperturbed linear system (1) and approximate solutions generated by applying the randomized Kaczmarz algorithm to the perturbed linear system.

*k*th iteration.

*x*on both sides of (20) gives

In conclusion, we have derived the following theorem.

### Theorem 1

*A*be a matrix full column rank and assume the system \(Ax = b\) is consistent. Let \(\tilde{x}_{k}\) be the

*k*th iterate of the noisy randomized Kaczmarz method run with \(Ax \simeq b +r\), and let \(a_1,\cdots , a_m\) denote the rows of

*A*. Then we have

## 4 Conclusions

In this paper, we provide a new look at the Kaczmarz algorithm for solving system of linear equations. The optimal row selecting strategy of the Kaczmarz algorithm for solving consistent system of linear equations is derived. The convergence of the randomized Kaczmarz algorithm for solving perturbed system of linear equations is analyzed and a new bound of the convergence rate is obtained from a new perspective.

## References

- 1.Agaskar, A., Wang, C., Lu, Y.M.: Randomized Kaczmarz algorithms: exact MSE analysis and optimal sampling probabilities. In: Proceedings of the 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Atlanta, GA, 3–5 December, pp. 389–393 (2014)Google Scholar
- 2.Ivanov, A.A., Zhdanov, A.I.: Kaczmarz algorithm for Tikhonov regularization problem. Appl. Math. E-Notes
**13**, 270–276 (2013)MathSciNetzbMATHGoogle Scholar - 3.Ivanovy, A.A., Zhdanovz, A.I.: The block Kaczmarz algorithm based on solving linear systems with arrowhead matrices. Appl. Math. E-Notes
**17**, 142–156 (2017)MathSciNetGoogle Scholar - 4.Bai, Z.-Z., Liu, X.-G.: On the Meany inequality with applications to convergence ananlysis of several row-action methods. Numer. Math.
**124**, 215–236 (2013)MathSciNetCrossRefGoogle Scholar - 5.Bai, Z.-Z., Wu, W.-T.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput.
**40**, A592–A606 (2018)MathSciNetCrossRefGoogle Scholar - 6.Bai, Z.-Z., Wu, W.-T.: On relaxed greedy randomized Kaczmarz methods for solving large sparse linear systems. Appl. Math. Lett.
**83**, 21–26 (2018)MathSciNetCrossRefGoogle Scholar - 7.Benzi, M., Meyer, C.D.: A direct projection method for sparse linear systems. SIAM J. Sci. Comput.
**16**, 1159–1176 (1995)MathSciNetCrossRefGoogle Scholar - 8.Brenzinski, C.: Projection Methods for Systems of Equations. Elsevier Science B.V., Amsterdam (1997)Google Scholar
- 9.Benzi, M., Meyer, C.D.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys.
**7**, 200–217 (1967)MathSciNetGoogle Scholar - 10.Cai, J., Tang, Y.: A new randomized Kaczmarz based kernel canonical correlation analysis algorithm with applications to information retrieval. Neural Netw.
**98**, 178–191 (2018)CrossRefGoogle Scholar - 11.Censor, Y.: Row-action methods for huge and sparse systems and their applications. SIAM Rev.
**23**, 444–466 (1981)MathSciNetCrossRefGoogle Scholar - 12.Censor, Y., Eggermont, P.P.B., Gordon, D.: Strong underrelaxation in Kaczmarz’s method for inconsistent systems. Numer. Math.
**41**, 83–92 (1983)MathSciNetCrossRefGoogle Scholar - 13.Censor, Y., Herman, G.T., Jiang, M.: A note on the behaviour of the randomized Kaczmarz algorithm of Strohmer and Vershynin. J. Fourier Anal. Appl.
**15**, 431–436 (2009)MathSciNetCrossRefGoogle Scholar - 14.Demmel, J.: The probability that a numerical analysis problem is difficult. Math. Comput.
**50**, 449–480 (1988)MathSciNetCrossRefGoogle Scholar - 15.De Loera, J.A., Haddock, J., Needell, D.: A sampling Kaczmarz-Motzkin algorithm for linear feasibility. SIAM J. Sci. Comput.
**39**, S66–S87 (2017)MathSciNetCrossRefGoogle Scholar - 16.Eldar, Y.C., Needell, D.: Acceleration of randomized Kaczmarzmethod via the Johnson–Lindenstrauss lemma. Numer. Algorithms
**58**, 163–177 (2011)MathSciNetCrossRefGoogle Scholar - 17.Feichtinger, H.G., Cenker, C., Mayer, M., Steier, H., Strohmer, T.: New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling. In: VCIP. SPIE, pp. 299–310 (1992)Google Scholar
- 18.Galántai, A.: On the rate of convergence of the alternating projection method in finite dimensional spaces. J. Math. Anal. Appl.
**310**, 30–44 (2005)MathSciNetCrossRefGoogle Scholar - 19.Gordon, R., Bender, R., Herman, G.T.: Algebraic Reconstruction Techniques (ART) for threedimensional electron microscopy and x-ray photography. J. Theor. Biol.
**29**, 471–481 (1970)CrossRefGoogle Scholar - 20.Gordon, R., Herman, G.T., Johnson, S.A.: Image reconstruction from projections. Sci. Am.
**233**, 56–71 (1975)CrossRefGoogle Scholar - 21.Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal.
**36**, 1660–1690 (2015)MathSciNetCrossRefGoogle Scholar - 22.Hanke, M., Niethammer, W.: On the acceleration of Kaczmarz’s method for inconsistent linear systems. Linear Algebra Appl.
**130**, 83–98 (1990)MathSciNetCrossRefGoogle Scholar - 23.Hefny, A., Needell, D., Ramdas, A.: Rows vs. columns: randomized Kaczmarz or gauss-seidel for ridge regression. SIAM J. Sci. Comput.
**39**, S528–S542 (2016)CrossRefGoogle Scholar - 24.Herman, G.T.: Image Reconstruction from Projection, the Fundamentals of the Computerized Tomography. Academic Press, New York (1980)zbMATHGoogle Scholar
- 25.Herman, G.T., Lent, A., Lutz, P.H.: Relaxation methods for image reconstruction. Commun. Assoc. Comput. Mach.
**21**, 152–158 (1978)MathSciNetzbMATHGoogle Scholar - 26.Herman, G.T., Meyer, L.B.: Algebraic reconstruction techniques can be made computationally efficient. IEEE Trans. Med. Imaging
**12**, 600–609 (1993)CrossRefGoogle Scholar - 27.Jiao, Y.-L., Jin, B.-T., Lu, X.-L.: Preasymptotic convergence of randomized Kaczmarz method. Inverse Prob.
**33**, 125012 (2017)MathSciNetCrossRefGoogle Scholar - 28.Kaczmarz, S.: Angenäherte auflösung von systemen linearer gleichungen. Bull. Acad. Polon. Sci. Lett. A
**35**, 355–357 (1933)zbMATHGoogle Scholar - 29.Leventhal, L., Lewis, A.S.: Randomized methods for linear constraints: convergence rates and conditioning. Math. Oper. Res.
**35**, 641–654 (2010)MathSciNetCrossRefGoogle Scholar - 30.Natterer, F.: Themathematics of Computerized Tomography. SIAM (2001)Google Scholar
- 31.Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl.
**36**, 1590–1604 (2015)MathSciNetCrossRefGoogle Scholar - 32.Needell, D.: Randomized Kaczmarz solver for noisy linear systems. BIT
**50**, 395–403 (2010)MathSciNetCrossRefGoogle Scholar - 33.Needell, D., Zhao, R., Zouzias, A.: Randomized block Kaczmarz method with projection for solving least squares. Linear Algebra Appl.
**484**, 322–343 (2015)MathSciNetCrossRefGoogle Scholar - 34.Needell, D., Tropp, J.A.: Paved with good intentions: analysis of a randomized block Kaczmarz method. Linear Algebra Appl.
**441**, 199–221 (2014)MathSciNetCrossRefGoogle Scholar - 35.von Neumann, J.: The Geometry of Orthogonal Spaces, vol. 2. Princeton University Press, Princeton (1950). This is a mimeographed lecture notes, first distributed in 1933Google Scholar
- 36.Nutini, J., Sepehry, B., Laradji, I., Schmidt, M., Koepke, H., Virani, A.: Convergence rates for greedy Kaczmarz algorithms, and faster randomized Kaczmarz rules using the orthogonality graph, UAI (2016)Google Scholar
- 37.Schmidt, M.: Notes on randomized Kaczmarz, Lecture notes, 9 April 2015Google Scholar
- 38.Schwarz, H.A.: Ueber einen Grenzübergang durch alternirendes Verfahren. Vierteljahrsschrift der Naturforschenden Gessellschaft in Zurich
**15**, 272–286 (1870)zbMATHGoogle Scholar - 39.Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl.
**15**, 262–278 (2009)MathSciNetCrossRefGoogle Scholar - 40.Strohmer, T.: Comments on the randomized Kaczmarz method (2009, unpublished manuscript)Google Scholar
- 41.Tanabe, K.: Projection method for solving a singular system of linear equations and its applications. Numer. Math.
**17**, 203–214 (1971)MathSciNetCrossRefGoogle Scholar - 42.Trussell, H., Civanlar, M.: Signal deconvolution by projection onto convex sets. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 1984, vol. 9, pp. 496–499 (1984)Google Scholar
- 43.Youla, D.C., Webb, H.: Image restoration by the method of convex projections: part 1 theory. IEEE Trans. Med. Imaging
**1**, 81–94 (1982)CrossRefGoogle Scholar - 44.Youla, D.C., Webb, H.: Image restoration by the method of convex projections: part 2 applications and numerical results. IEEE Trans. Med. Imaging
**1**, 95–101 (1982)CrossRefGoogle Scholar - 45.Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal.
**34**, 773–793 (2013)MathSciNetCrossRefGoogle Scholar