Advertisement

Convergence study on the proximal alternating direction method with larger step size

  • Feng MaEmail author
Original Paper
First Online:
  • 28 Downloads

Abstract

The alternating direction method of multipliers (ADMM) is a popular method for solving separable convex programs with linear constraints, and its proximal version is an important variant. In the literature, Fortin and Glowinski proved that the step size for updating the Lagrange multiplier of the ADMM can be chosen in the open interval of zero to the golden ratio, and subsequently this result has been proved to be also valid for the proximal ADMM. In this paper, we demonstrate that the dual step size can be larger than the golden ratio when the proximal regularization is positive definite. Thus, the feasible interval of the dual step size can be further enlarged for the proximal ADMM. Moreover, we establish the exact relationship between the dual step size and the proximal parameter. We also prove global convergence and establish a worst case convergence rate in the ergodic sense for this proximal scheme with the enlarged step size. Finally, we present numerical results to demonstrate the practical performance of the method.

Keywords

Alternating direction method of multipliers Convex programming Proximal regularization Convergence analysis 

Mathematics Subject Classification (2010)

65K10 90C25 90C30 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

The author is grateful to the anonymous referees and the editor for their valuable comments and suggestions which have helped us improve the presentation of this paper. He would like to thank Professor Bingsheng He for fruitful discussions and suggestions regarding this project and thank Professor Shiqian Ma for providing the SPCP codes.

References

  1. 1.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R (ed.) Optimization, pp 283–298. Academic Press, New York (1969)Google Scholar
  3. 3.
    Chan, T.F., Glowinski, R.: Finite element approximation and iterative solution of a class of mildly nonlinear elliptic equations. Stanford report STAN-CS-78-674, Computer Science Department, Stanford University, Palo Alto CA (1978)Google Scholar
  4. 4.
    Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. Revue franċaise d’automatique, informatique, recherche opérationnelle. Analyse numérique 9(2), 41–76 (1975)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefGoogle Scholar
  6. 6.
    Eckstein, J., Yao, W.: Understanding the convergence of the alternating direction method of multipliers: Theoretical and computational perspectives. Pac. J. Optim. 11(4), 619–644 (2015)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Glowinski, R.: On alternating directionmethods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmaki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology. Computational Methods in Applied Sciences, vol. 34, pp 59–82. Springer, Dordrecht (2014)Google Scholar
  8. 8.
    Glowinski, R., Pan, T.W., Tai, X.C.: Some facts about operator-splitting and alternating direction methods. In: Glowinski, R., Osher, S.J., Yin, W. (eds.) Splitting Methods in Communication, Imaging, Science, and Engineering, pp 19–94. Springer, New York (2016)CrossRefGoogle Scholar
  9. 9.
    Glowinski, R., Osher, S.J., Yin, W. (eds.): Splitting Methods in Communication, Imaging, Science, and Engineering. Springer, New York (2016)zbMATHGoogle Scholar
  10. 10.
    Cai, X.J., Gu, G.Y., He, B.S., Yuan, X.: A proximal point algorithm revisit on the alternating direction method of multipliers. Sci. China Math. 56(10), 2179–2186 (2013)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Ma, S.Q.: Alternating proximal gradient method for convex minimization. J. Sci. Comput. 68, 546–572 (2016)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55 (1-3), 293–318 (1992)MathSciNetCrossRefGoogle Scholar
  13. 13.
    He, B.S., Liu, H., Wang, Z.R., Yuan, X.: A strictly contractive Peaceman–Rachford splitting method for convex programming. SIAM J. Optim. 24 (3), 1011–1040 (2014)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Ma, F.: On relaxation of some customized proximal point algorithms for convex minimization: from variational inequality perspective. Comput. Optim. Appl. 73(3), 871–901 (2019)MathSciNetCrossRefGoogle Scholar
  15. 15.
    He, B., Ma, F., Yuan, X.: Convergence study on the symmetric version of ADMM with larger step sizes. SIAM J. Imaging Sci. 9(3), 1467–1501 (2016)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Fortin, M., Glowinski, R.: Méthodes de Lagrangien Augmenté: Application à la Résolution Numérique de Problèmes aux Limites, Dunod, Paris (1982)Google Scholar
  17. 17.
    Fortin, M., Glowinski, R.: Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, Amsterdam the Netherlands (1983)Google Scholar
  18. 18.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)CrossRefGoogle Scholar
  19. 19.
    Wen, Z.W., Goldfarb, D., Yin, W.T.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Chen, C.H., He, B.S., Yuan, X.: Matrix completion via alternating direction method. IMA J. Numer. Analy. 32, 227–245 (2012)MathSciNetCrossRefGoogle Scholar
  21. 21.
    He, B.S., Xu, M.H., Yuan, X.: Solving large-scale least squares covariance matrix problems by alternating direction methods. SIAM J. Matrix Anal. Appli. 32, 136–152 (2011)CrossRefGoogle Scholar
  22. 22.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite-element approximations. Comput. Math. Appli. 2, 17–40 (1976)CrossRefGoogle Scholar
  23. 23.
    Tao, M., Yuan, X.: On Glowinski’s open question on the alternating direction method of multipliers. J. Optim. Theory Appl. 179, 163–196 (2018)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Eckstein, J.: Some saddle-function splitting methods for convex programming. Optim. Methods Soft. 4, 75–83 (1994)MathSciNetCrossRefGoogle Scholar
  25. 25.
    He, B.S., Liao, L.Z., Han, D.R., Yang, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92, 103–118 (2002)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Yang, J., Yuan, X.: Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization. Math. Comput. 82(281), 301–329 (2013)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Deng, W., Yin, W.T.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66, 889–916 (2016)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Zhang, X.Q., Burger, M., Osher, S.: A unified primal-dual algorithm framework based on Bregman iteration. J. Sci. Comput. 96, 20–46 (2011)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Chan, R.H., Tao, M., Yuan, X.: Linearized alternating direction method of multipliers for constrained linear least-squares problem. East Asian J. Appl. Math. 2, 326–341 (2012)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Wang, X., Yuan, X.: The linearized alternating direction method of multipliers for dantzig selector. SIAM J. Sci. Comput. 34(5), A2792–A2811 (2012)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Xu, M.H.: Proximal alternating directions method for structured variational inequalities. J. Optim. Theory Appl. 134, 107–117 (2007)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Gonçalves, M. L. N., Melo, J.G., Monteiro, R.D.C.: Extending the ergodic convergence rate of the proximal ADMM. arXiv:1611.02903 (2016)
  33. 33.
    Wright, J., Ganesh, A., Min, K., Ma, Y.: Compressive principal component pursuit. Information and Inference: A Journal of the IMA 2(1), 32–68 (2013)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numerica 25, 161–319 (2016)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Sra, S., Nowozin, S., Wright, S.J.: Optimization for machine learning. Mit Press (2012)Google Scholar
  36. 36.
    Gonċalves, M. L. N., Melo, J.G., Monteiro, R.D.C.: Convergence rate bounds for a proximal ADMM with over-relaxation stepsize parameter for solving nonconvex linearly constrained problems. arXiv:1702.01850 (2017)
  37. 37.
    Gu, Y, Jiang, B, Han, D.: A semi-proximal-based strictly contractive Peaceman-Rachford splitting method. arXiv:1506.02221 (2015)
  38. 38.
    Gonçalves, M.L.N.: On the pointwise iteration-complexity of a dynamic regularized ADMM with over-relaxation stepsize. Appl. Math. Comput. 336, 315–325 (2018)MathSciNetGoogle Scholar
  39. 39.
    Gonçalves, M.L.N., Alves, M.M., Melo, J.G.: Pointwise and ergodic convergence rates of a variable metric proximal alternating direction method of multipliers. J. Optim. Theory Appl. 177(2), 448–478 (2018)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Facchinei, F, Pang, J S: Finite-Dimensional Variational Inequalities and Complementarity Problems, vol. 1, Springer Series in Operations Research. Springer, New York (2003)zbMATHGoogle Scholar
  41. 41.
    He, B., Yuan, X.: On the O(1/n) convergence rate of the Douglas-Rachford alternating direction method. SIAM J. Num. Anal. 50(2), 700–709 (2012)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Royal Stat. Soc., Series B. 58(1), 267–288 (1996)MathSciNetzbMATHGoogle Scholar
  43. 43.
    He, B., Ma, F., Yuan, X.: Optimal proximal augmented Lagrangian method and its application to full Jacobian splitting for multi-block separable convex minimization problems. IMA J. Num. Anal.  https://doi.org/10.1093/imanum/dry092 (2019)
  44. 44.
    He, B., Ma, F., Yuan, X.: Optimally linearizing the alternating direction method of multipliers for convex programming. Comput. Optim. Appl.  https://doi.org/10.1007/s10589-019-00152-3 (2017)
  45. 45.
    Gao, B., Ma, F.: Symmetric alternating direction method with indefinite proximal regularization for linearly constrained convex optimization. J. Optim. Theory Appl. 176(1), 178–204 (2018)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Xi’an Research Institute of High TechnologyXi’anChina

Personalised recommendations

Cite article

Buy options