Numerical Algorithms

, Volume 73, Issue 1, pp 33–76 | Cite as

Optimal Rates of Linear Convergence of Relaxed Alternating Projections and Generalized Douglas-Rachford Methods for Two Subspaces

  • Heinz H. Bauschke
  • J. Y. Bello Cruz
  • Tran T. A. Nghia
  • Hung M. Pha
  • Xianfu Wang
Original Paper


We systematically study the optimal linear convergence rates for several relaxed alternating projection methods and the generalized Douglas-Rachford splitting methods for finding the projection on the intersection of two subspaces. Our analysis is based on a study on the linear convergence rates of the powers of matrices. We show that the optimal linear convergence rate of powers of matrices is attained if and only if all subdominant eigenvalues of the matrix are semisimple. For the convenience of computation, a nonlinear approach to the partially relaxed alternating projection method with at least the same optimal convergence rate is also provided. Numerical experiments validate our convergence analysis


Convergent and semi-convergent matrix Friedrichs angle Generalized Douglas-Rachford method Linear convergence Principal angle Relaxed alternating projection method 

Mathematics Subject Classification (2010)

Primary 65F10, 65K05 Secondary 65F15, 65B05, 15A18, 90C25, 41A25 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aragón Artacho, F.J., Borwein, J.M., Tam, M.K.: Recent results on Douglas-Rachford methods for combinatorial optimization problems. J. Optim. Theory Appl. 163, 1–30 (2014)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Bauschke, H.H., Borwein, J.M.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Bauschke, H.H., Combettes, P.L.: Convex analysis and monotone operator theory in Hilbert spaces. Springer (2011)Google Scholar
  4. 4.
    Bauschke, H.H., Combettes, P.L., Kruk, S.G.: Extrapolation algorithm for affine-convex feasibility problems. Numer. Algorithms 4, 239–274 (2006)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Bauschke, H.H., Bello Cruz, J.Y., Nghia, T.T.A., Phan, H.M., Wang, X.: The rate of linear convergence of the Douglas-Rachford algorithm for subspaces is the cosine of the Friedrichs angle. J. Approx. Theory 185, 63–79 (2014)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Bauschke, H.H., Deutsch, F., Hundal, H., Park, S.-H.: Accelerating the convergence of the method of alternating projections. Trans. Amer. Math. Soc. 355, 3433–3461 (2003)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Bauschke, H.H., Kruk, S.G.: Reflection-projection method for convex feasibility problems with an obtuse cone. J. Optim. Theory Appl. 120, 503–531 (2004)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Bauschke, H.H., Dao, M.N., Noll, D., Phan, H.M.: Proximal point algorithm, Douglas-Rachford algorithm and alternating projections: a case study. J Convex Anal. (to appear)Google Scholar
  9. 9.
    Björck, A., Golub, G.H.: Numerical methods for computing angles between linear subspaces. Math. Comp. 27, 579–594 (1973)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Berman, A., Plemmons, R.J.: Nonnegative matrices in the mathematical sciences. SIAM, Philadelphia (1994)CrossRefMATHGoogle Scholar
  11. 11.
    Cegielski, A.: Iterative methods for fixed point problems in Hilbert spaces. Springer (2012)Google Scholar
  12. 12.
    Cegielski, A., Suchocka, A.: Relaxed alternating projection methods. SIAM J. Optim. 19, 1093–1106 (2008)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Censor, Y., Elfving, T., Herman, G.T., Nikazad, T.: On diagonally relaxed orthogonal projection methods. SIAM J. Sci. Comput. 30, 473–504 (2007/08)Google Scholar
  14. 14.
    Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing, Fixed-point algorithms for inverse problems in science and engineering, 185–212, Springer Optim. Appl., 49. Springer, New York (2011)Google Scholar
  15. 15.
    Combettes, P.L.: Solving monotone inclusions via compositions of nonexpansive averaged operators. Optimization 53, 475–504 (2004)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Demanet, L., Zhang, X.: Eventual linear convergence of the Douglas-Rachford iteration for basis pursuit. Math. Comput. 85, 209–238 (2016)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Deutsch, F.: The angle between subspaces of a Hilbert space. In: Singh, S.P. (ed.) Approximation theory, wavelets and applications, pp 107–130. Kluwer (1995)Google Scholar
  18. 18.
    Deutsch, F.: Best approximations in inner product spaces. Springer (2001)Google Scholar
  19. 19.
    Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Amer. Math. Soc. 82, 421–439 (1956)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Elfving, T., Nikazad, T., Hansen, P.C.: Semi-convergence and relaxation parameters for a class of SIRT algorithms. Electron. Trans. Numer. Anal. 37, 321–336 (2010)MathSciNetMATHGoogle Scholar
  22. 22.
    Escalante, R., Raydan, M.: Alternating projection methods, fundamentals of algorithms 8. SIAM, Philadelphia (2011)CrossRefMATHGoogle Scholar
  23. 23.
    Gearhart, W.B., Koshy, M.: Acceleration schemes for the method of alternating projections. J. Comp. Appl. Math. 26, 235–249 (1989)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
  25. 25.
    Gubin, L.G., Polyak, B.T., Raik, E.V.: The method of projections for finding the common point of convex sets. USSR Comp. Math. Math. Phys. 7, 1–24 (1967)CrossRefGoogle Scholar
  26. 26.
    Hensel, K.: Über Potenzreihen von Matrizen. J. Reine Angew. Math. 155, 107–110 (1926)MathSciNetMATHGoogle Scholar
  27. 27.
    Kayalar, S., Weinert, H.: Error bounds for the method of alternating projections. Math. Control Signals Syst. 1, 43–59 (1996)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Kirkland, S.: A cycle-based bound for subdominant eigenvalues of stochastic matrices. Linear Multilinear Algebra 57, 247–266 (2009)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Hesse, R., Luke, D.R., Neumann, P.: Alternating projections and Douglas-Rachford for sparse affine feasibility. IEEE Trans. Signal Process. 62, 4868–4881 (2014)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Hesse, R., Luke, D.R.: Nonconvex notions of regularity and convergence of fundamental algorithms for feasibility problems. SIAM J. Optim. 23, 2397–2419 (2013)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    The Julia language:
  32. 32.
    Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Luke, D.R.: Finding best approximation pairs relative to a convex and prox-regular set in a Hilbert space. SIAM J. Optim. 19, 714–739 (2008)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Marek, I.: On square roots of M-operators. Linear Algebra Appl. 223(/224), 501–520 (1995)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Marek, I., Szyld, D.B.: Comparison theorems for the convergence factor of iterative methods for singular matrices. Linear Algebra Appl. 316, 67–87 (2000)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Marek, I., Szyld, D.B.: Comparison of convergence of general stationary iterative methods for singular matrices. SIAM J. Matrix Anal. Appl. 24, 68–77 (2002)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Meyer, C.D.: Matrix analysis and applied linear algebra. SIAM, Philadelphia (2000)CrossRefGoogle Scholar
  38. 38.
    Meyer, C.D., Plemmons, R.J.: Convergent powers of a matrix with applications to iterative methods for singular linear systems. SIAM J. Numer. Anal. 14, 699–705 (1977)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Miao, J., Ben-Israel, A.: On principal angles between subspaces in ℝn. Linear Algebra Appl. 171, 81–98 (1992)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Nelson, S., Neumann, M.: Generalizations of the projection method with applications to SOR theory for Hermitian positive semidefinite linear systems. Numer. Math. 51, 123–141 (1987)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Oldenburger, R.: Infinite powers of matrices and characteristic roots. Duke Math. J. 6, 357–361 (1940)MathSciNetCrossRefMATHGoogle Scholar
  42. 42.
    Saad, Y.: Iterative methods for sparse linear systems. SIAM, Philadelphia (2003)CrossRefMATHGoogle Scholar
  43. 43.
    Stewart, G.W.: Matrix algorithms, II: eigensystems. SIAM, Philadelphia (2001)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Heinz H. Bauschke
    • 1
  • J. Y. Bello Cruz
    • 2
  • Tran T. A. Nghia
    • 3
  • Hung M. Pha
    • 4
  • Xianfu Wang
    • 1
  1. 1.MathematicsUniversity of British ColumbiaKelownaCanada
  2. 2.IMEFederal University of GoiasGoianiaBrazil
  3. 3.Mathematics & StatisticsOakland UniversityRochesterUSA
  4. 4.Department of Mathematical SciencesUniversity of Massachusetts LowellLowellUSA

Personalised recommendations