Advertisement

Mathematical Programming

, Volume 173, Issue 1–2, pp 37–77 | Cite as

Extended ADMM and BCD for nonseparable convex minimization models with quadratic coupling terms: convergence analysis and insights

  • Caihua Chen
  • Min Li
  • Xin LiuEmail author
  • Yinyu Ye
Full Length Paper Series A

Abstract

In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) method for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proximal ADMM converges in point-wise manner to a primal-dual solution pair. Moreover, we apply randomly permuted ADMM (RPADMM) to nonseparable multi-block convex optimization, and prove its expected convergence for a class of nonseparable quadratic programming problems. When the linear constraint vanishes, the 2-block proximal ADMM and RPADMM reduce to the 2-block cyclic proximal BCD method and randomly permuted BCD (RPBCD). Our study provides the first iterate convergence result for 2-block cyclic proximal BCD without assuming the boundedness of the iterates. We also theoretically establish the expected iterate convergence result concerning multi-block RPBCD for convex quadratic optimization. In addition, we demonstrate that RPBCD may have a worse convergence rate than cyclic proximal BCD for 2-block convex quadratic minimization problems. Although the results on RPADMM and RPBCD are restricted to quadratic minimization models, they provide some interesting insights: (1) random permutation makes ADMM and BCD more robust for multi-block convex minimization problems; (2) cyclic BCD may outperform RPBCD for “nice” problems, and RPBCD should be applied with caution when solving general convex optimization problems especially with a few blocks.

Keywords

Nonseparable convex minimization Alternating direction method of multipliers Block coordinate descent method Iterate convergence Random permutation 

Mathematics Subject Classification

65K05 90C26 

Notes

Acknowledgements

Caihua Chen was supported by the National Natural Science Foundation of China [Grant No. 11401300, 71732003, 71673130]. Min Li was supported by the National Natural Science Foundation of China [Grant No.11771078, 71390335, 71661147004]. Xin Liu was supported by the National Natural Science Foundation of China [Grant No. 11622112, 11471325, 91530204, 11331012, 11461161005, 11688101], the National Center for Mathematics and Interdisciplinary Sciences, CAS, the Youth Innovation Promotion Association, CAS, and Key Research Program of Frontier Sciences, CAS. Yinyu Ye was supported by the AFOSR Grant [Grant No. FA9550-12-1-0396]. The authors would like to thank Dr. Ji Liu from University of Rochester and Dr. Ruoyu Sun from Stanford University for the helpful discussions on the block coordinate descent method. The authors would also like to thank the associate editor and two anonymous referees for their detailed and valuable comments and suggestions.

References

  1. 1.
    Agarwal, A., Negahban, S., Wainwright, M.J.: Noisy matrix decomposition via convex relaxation: optimal rates in high dimensions. Ann. Stat. 40(2), 1171–1197 (1997)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka-Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Beck, A.: On the convergence of alternating minimization for convex programming with applications to iteratively reweighted least squares and decomposition schemes. SIAM J. Optim. 25(1), 185–209 (2015)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Beck, A., Tetruashvili, L.: On the convergence of block coordinate descent type methods. SIAM J. Optim. 23(2), 2037–2060 (2013)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena-Scientific, Belmont (1999)zbMATHGoogle Scholar
  6. 6.
    Bertsekas, D.P., Tsitsiklis, J.N.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. In: Parallel and Distributed Computation: Numerical Methods, 2nd ed. Athena Scientific, Belmont, MA (1997)Google Scholar
  7. 7.
    Bolte, J., Sabach, S.Y., Teboulle, M.: Proximal alternating linearized minimization nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHGoogle Scholar
  9. 9.
    Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66(1), 39–73 (2017)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155(1), 57–79 (2016)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Chen, C., Shen, Y., You, Y.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstract and Applied Analysis, 2013, Article ID 183961, 7 pagesGoogle Scholar
  12. 12.
    Chen, L., Sun, D., Toh, K.-C.: A note on the convergence of ADMM for linearly constrained convex optimization problems. Comput. Optim. Appl. 66(2), 327–343 (2017)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Chen, L., Sun, D., Toh, K.-C.: An efficient inexact symmetric Gauss-Seidel based majorized ADMM for high-dimensional convex composite conic programming. Math. Program. 161(1), 237–270 (2017)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Cui, Y., Li, X., Sun, D., Toh, K.-C.: On the convergence properties of a majorized alternating direction method of multipliers for linearly constrained convex optimization problems with coupled objective functions. J. Optim. Theory Appl. 169(3), 1013–1041 (2016)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. UCLA CAM Report, 14–51 (2014)Google Scholar
  16. 16.
    Deng, W., Lai, M., Peng, Z., Yin, W.: Parallel multi-block ADMM with \(o(1/k)\) convergence. J. Sci. Comput. 71(2), 712–736 (2017)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55(1), 293–318 (1992)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Feng, C., Xu, H., Li, B.C.: An alternating direction method approach to cloud traffic management. IEEE Trans. Parallel Distrib. Syst. 28(8), 2145–2158 (2017)Google Scholar
  20. 20.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2, 17–40 (1976)zbMATHGoogle Scholar
  21. 21.
    Gao, X., Zhang, S.: First-order algorithms for convex optimization with nonseparable objective and coupled constraints. J. Oper. Res. Soc. China 5(2), 131–159 (2017)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)zbMATHGoogle Scholar
  23. 23.
    Glowinski, R., Marroco, A.: Sur l’approximation, par elements finis d’ordre un, et la resolution, par penalisation-dualite, d’une classe de problemes de dirichlet non lineares. Revue Franqaise d’Automatique, Informatique et Recherche Opirationelle 9, 41–76 (1975)zbMATHGoogle Scholar
  24. 24.
    Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155(1), 227–238 (2012)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Han, D., Yuan, X., Zhang, W., Cai, X.: An ADM-based splitting method for separable convex programming. Comput. Optim. Appl. 54, 343–369 (2013)MathSciNetzbMATHGoogle Scholar
  26. 26.
    He, B., Tao, M., Yuan, X.: A splitting method for separable convex programming. IMA J. Numer. Anal. 35, 394–426 (2015)MathSciNetzbMATHGoogle Scholar
  27. 27.
    He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)MathSciNetzbMATHGoogle Scholar
  28. 28.
    He, B., Yuan, X.: On the O\((1/n)\) convergence rate of the Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Hong, M., Chang, T., Wang, X., Razaviyayn, M., Ma, S., Luo, Z.: A block successive upper bound minimization method of multipliers for linearly constrained convex optimization. arXiv:1401.7079 (2014)
  30. 30.
    Hong, M., Luo, Z.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 165–199 (2017)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Hong, M., Luo, Z., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM J. Optim. 26(1), 337–364 (2016)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Hong, M., Wang, X., Razaviyayn, M., Luo, Z.: Iteration complexity analysis of block coordinate descent methods. arXiv:1310.6957v2 (2014)
  33. 33.
    Li, M., Sun, D., Toh, K.-C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia Pac. J Oper. Res. 32, 1550024 (2015)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Li, M., Sun, D., Toh, K.-C.: A majorized ADMM with indefinite proximal terms for linearly constrained convex composite optimization. SIAM J. Optim. 26(2), 922–950 (2016)MathSciNetzbMATHGoogle Scholar
  35. 35.
    Li, X., Sun, D., Toh, K.-C.: A Schur complement based semi-proximal ADMM for convex quadratic conic programming and extensions. Math. Program. 155(1), 333–373 (2016)MathSciNetzbMATHGoogle Scholar
  36. 36.
    Lin, T., Ma, S., Zhang, S.: Iteration complexity analysis of multi-block ADMM for a family of convex minimization without strong convexity. J. Sci. Comput. 69(1), 52–81 (2016)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multi-block variables. SIAM J. Optim. 25, 1478–1497 (2015)MathSciNetzbMATHGoogle Scholar
  38. 38.
    Lin, T., Ma, S., Zhang, S.: On the sublinear convergence rate of multi-block ADMM. J. Oper. Res. Soc. China 3(3), 251–274 (2015)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Lu, Z., Xiao, L.: On the complexity analysis of randomized block-coordinate descent methods. Math. Program. 152, 615–642 (2015)MathSciNetzbMATHGoogle Scholar
  40. 40.
    Monteiro, R., Svaiter, B.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Mota, J.F.C., Xavier, J.M.F., Aguiar, P.M.F., Puschel, M.: Distributed optimization with local domains: Applications in MPC and network flows. IEEE Trans. Autom. Control 60(7), 2004–2009 (2015)MathSciNetzbMATHGoogle Scholar
  42. 42.
    Peng, Y.G., Ganesh, A., Wright, J., Xu, W.L., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2233–2246 (2012)Google Scholar
  43. 43.
    Razaviyayn, M., Hong, M., Luo, Z.: A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM J. Optim. 23(2), 1126–1153 (2013)MathSciNetzbMATHGoogle Scholar
  44. 44.
    Richtárik, P., Takác̆, M.: Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program. 144(2), 1–38 (2014)MathSciNetzbMATHGoogle Scholar
  45. 45.
    Shalev-Shwartz, S., Zhang, T.: Stochastic dual coordinate ascent methods for regularized loss. J. Mach. Learn. Res. 14, 567–599 (2013)MathSciNetzbMATHGoogle Scholar
  46. 46.
    Shefi, R., Teboulle, M.: On the rate of convergence of the proximal alternating linearized minimization algorithm for convex problems. EURO J. Comput. Optim. 4(1), 27–46 (2016)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Sun, D., Toh, K.-C., Yang, L.: A convergent 3-block semi-proximal alternating direction method of multipliers for conic programming with 4-block constraints. SIAM J. Optim. 25(2), 882–915 (2015)MathSciNetzbMATHGoogle Scholar
  48. 48.
    Sun, R., Luo, Z., Ye, Y.: On the expected convergence of randomly permuted ADMM. arXiv:1503.06387v1 (2015)
  49. 49.
    Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109, 475–494 (2001)MathSciNetzbMATHGoogle Scholar
  50. 50.
    Tseng, P., Yun, S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117, 387–423 (2009)MathSciNetzbMATHGoogle Scholar
  51. 51.
    Wright, S.: Coordinate descent algorithms. Math. Program. 151(1), 3–34 (2015)MathSciNetzbMATHGoogle Scholar
  52. 52.
    Zhang Y (2010) Convergence of a class of stationary iterative methods for saddle point problems. Rice University Technique Report, TR10-24Google Scholar

Copyright information

© Springer-Verlag GmbH Germany and Mathematical Optimization Society 2017

Authors and Affiliations

  1. 1.International Center of Management Science and Engineering, School of Management and EngineeringNanjing UniversityNanjingChina
  2. 2.State Key Laboratory of Scientific and Engineering Computing, Academy of Mathematics and Systems ScienceChinese Academy of SciencesBeijingChina
  3. 3.School of Mathematical SciencesUniversity of Chinese Academy of SciencesBeijingChina
  4. 4.Department of Management Science and Engineering, School of EngineeringStanford UniversityStanfordUSA

Personalised recommendations