Inexact alternating direction methods of multipliers for separable convex optimization

  • William W. HagerEmail author
  • Hongchao Zhang


Inexact alternating direction multiplier methods (ADMMs) are developed for solving general separable convex optimization problems with a linear constraint and with an objective that is the sum of smooth and nonsmooth terms. The approach involves linearized subproblems, a back substitution step, and either gradient or accelerated gradient techniques. Global convergence is established. The methods are particularly useful when the ADMM subproblems do not have closed form solution or when the solution of the subproblems is expensive. Numerical experiments based on image reconstruction problems show the effectiveness of the proposed methods.


Separable convex optimization Alternating direction method of multipliers Multiple blocks Inexact ADMM Global convergence 

Mathematics Subject Classification

90C06 90C25 65Y20 



  1. 1.
    Afonso, M., Bioucas-Dias, J., Figueiredo, M.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Mach. Learn. 3, 1–122 (2010)CrossRefzbMATHGoogle Scholar
  4. 4.
    Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66, 39–73 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155, 57–79 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Chen, C., Li, M., Liu, X., Ye, Y.: On the convergence of multi-block alternating direction method of multipliers and block coordinate descent method (2015). arXiv:1508.00193
  7. 7.
    Chen, C., Shen, Y., You, Y.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstr. Appl. Anal. (2013). MathSciNetzbMATHGoogle Scholar
  8. 8.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Chen, L., Sun, D., Toh, K.: An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming. Math. Program. 161, 237–270 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Chen, Y., Hager, W.W., Yashtini, M., Ye, X., Zhang, H.: Bregman operator splitting with variable stepsize for total variation image reconstruction. Comput. Optim. Appl. 54, 317–342 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set Valued Var. Anal. 25, 829–858 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Eckstein, J., Bertsekas, D.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Eckstein, J., Silva, P.J.S.: A practical relative error criterion for augmented Lagrangians. Math. Program. 141, 319–348 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Eckstein, J., Yao, W.: Approximate ADMM algorithms derived from Lagrangian splitting. Comput. Optim. Appl. 68, 363–405 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Eckstein, J., Yao, W.: Relative-error approximate versions of Douglas–Rachford splitting and special cases of the ADMM. Math. Program. 170, 417–444 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite-element approximations. Comput. Math. Appl. 2, 17–40 (1976)CrossRefzbMATHGoogle Scholar
  17. 17.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)CrossRefzbMATHGoogle Scholar
  18. 18.
    Goldfarb, D., Ma, S.: Fast multiple-splitting algorithms for convex optimization. SIAM J. Optim. 22, 533–556 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Golshtein, E., Tretyakov, N.: Modified Lagrangians in convex programming and their generalizations, in point-to-set maps and mathematical programming. In: Huard, P. (ed.) Mathematical Programming Studies, vol. 10, pp. 86–97. Springer, Berlin (1979)Google Scholar
  20. 20.
    Hager, W.W., Yashtini, M., Zhang, H.: An O(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm. SIAM J. Numer. Anal. 54, 1535–1556 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Hager, W.W., Zhang, H.: Projection onto a polyhedron that exploits sparsity. SIAM J. Optim. 29, 1773–1798 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155, 227–238 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    He, B., Liao, L., Han, D., Yan, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92, 103–118 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    He, B., Tao, M., Xu, M., Yuan, X.: An alternating direction-based contraction method for linearly constrained separable convex programming problems. Optimization 62, 573–596 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    He, B., Yuan, X.: On the \({\cal{O}}(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Hong, M., Luo, Z.-Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 165–199 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Li, J., Chen, G., Dong, Z., Wu, Z.: A fast dual proximal-gradient method for separable convex optimization with linear coupled constraints. Comput. Optim. Appl. 64, 671–697 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Li, M., Liao, L., Yuan, X.: Inexact alternating direction methods of multipliers with logarithmic–quadratic proximal regularization. J. Optim. Theory Appl. 159, 412–436 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Li, M., Sun, D., Toh, K.C.: A convergent 3-block semi-proximal ADMM for for convex minimization problems with one strongly convex block. Asia Pac. J. Oper. Res. 32, 1–19 (2015)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multiblock variables. SIAM J. Optim. 25, 1478–1497 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)CrossRefzbMATHGoogle Scholar
  33. 33.
    Solodov, M.V., Svaiter, B.F.: An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions. Math. Oper. Res. 25, 214–230 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of FloridaGainesvilleUSA
  2. 2.Department of MathematicsLouisiana State UniversityBaton RougeUSA

Personalised recommendations