Accelerated primal–dual proximal block coordinate updating methods for constrained convex optimization

Article
  • 106 Downloads

Abstract

Block coordinate update (BCU) methods enjoy low per-update computational complexity because every time only one or a few block variables would need to be updated among possibly a large number of blocks. They are also easily parallelized and thus have been particularly popular for solving problems involving large-scale dataset and/or variables. In this paper, we propose a primal–dual BCU method for solving linearly constrained convex program with multi-block variables. The method is an accelerated version of a primal–dual algorithm proposed by the authors, which applies randomization in selecting block variables to update and establishes an O(1 / t) convergence rate under convexity assumption. We show that the rate can be accelerated to \(O(1/t^2)\) if the objective is strongly convex. In addition, if one block variable is independent of the others in the objective, we then show that the algorithm can be modified to achieve a linear rate of convergence. The numerical experiments show that the accelerated method performs stably with a single set of parameters while the original method needs to tune the parameters for different datasets in order to achieve a comparable level of performance.

Keywords

Primal–dual method Block coordinate update Alternating direction method of multipliers (ADMM) Accelerated first-order method 

Mathematics Subject Classification

90C25 95C06 68W20 

References

  1. 1.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear programs. SIAM J. Optim. 23(4), 2183–2207 (2013)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Bredies, K., Sun, H.: Accelerated douglas-rachford methods for the solution of convex-concave saddle-point problems. arXiv preprint arXiv:1604.06282 (2016)
  4. 4.
    Cai, X., Han, D., Yuan, X.: The direct extension of admm for three-block separable convex minimization models is convergent when one function is strongly convex. Optim. Online (2014)Google Scholar
  5. 5.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155(1–2), 57–79 (2016)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Rev. 43(1), 129–159 (2001)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Combettes, P.L., Pesquet, J.-C.: Stochastic quasi-fejér block-coordinate fixed point iterations with random sweeping. SIAM J. Optim. 25(2), 1221–1248 (2015)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)MATHGoogle Scholar
  11. 11.
    Dang, C., Lan, G.: Randomized methods for saddle point computation. arXiv preprint arXiv:1409.8625 (2014)
  12. 12.
    Deng, W., Lai, M.-J., Peng, Z., Yin, W.: Parallel multi-block admm with \(o (1/k)\) convergence. J. Sci. Comput. 71(2), 712–736 (2017)Google Scholar
  13. 13.
    Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2015)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefMATHGoogle Scholar
  15. 15.
    Gao, X., Xu, Y., Zhang, S.: Randomized primal-dual proximal block coordinate updates. arXiv preprint arXiv:1605.05969 (2016)
  16. 16.
    Gao, X., Zhang, S.-Z.: First-order algorithms for convex optimization with nonseparable objective and coupled onstraints. J. Oper. Res. Soc. China 5(2), 131–159 (2017)Google Scholar
  17. 17.
    Glowinski, R., Marrocco, A.: Sur l’approximation, par eléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires. ESAIM. Math. Modell. Numer. Anal. 9(R2), 41–76 (1975)MATHGoogle Scholar
  18. 18.
    Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci. 7(3), 1588–1623 (2014)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    He, B., Hou, L., Yuan, X.: On full Jacobian decomposition of the augmented Lagrangian method for separable convex programming. SIAM J. Optim. 25(4), 2274–2312 (2015)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    He, B., Tao, M., Yuan, X.: Alternating direction method with gaussian back substitution for separable convex programming. SIAM J. Optim. 22(2), 313–340 (2012)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    He, B., Yuan, X.: On the acceleration of augmented lagrangian method for linearly constrained optimization. Optim. Online (2010)Google Scholar
  22. 22.
    He, B., Yuan, X.: On the \(O(1/n)\) convergence rate of the Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50(2), 700–709 (2012)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    James, G. M., Paulson, C., Rusmevichientong, P.: Penalized and constrained regression. Technical report (2013)Google Scholar
  24. 24.
    Li, H., Lin, Z.: Optimal nonergodic \( o (1/k) \) convergence rate: when linearized adm meets nesterov’s extrapolation. arXiv preprint arXiv:1608.06366 (2016)
  25. 25.
    Li, M., Sun, D., Toh, K.-C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia-Pac. J. Oper. Res. 32(04), 1550024 (2015)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the admm with multiblock variables. SIAM J. Optim. 25(3), 1478–1497 (2015)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Lu, Z., Xiao, L.: On the complexity analysis of randomized block-coordinate descent methods. Math. Program. 152(1–2), 615–642 (2015)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Markowitz, H.: Portfolio selection. J. Finance 7(1), 77–91 (1952)Google Scholar
  29. 29.
    Monteiro, R.D., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \({O}(1/k^2)\). Soviet Math. Doklady 27(2), 372–376 (1983)MATHGoogle Scholar
  31. 31.
    Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Ouyang, Y., Chen, Y., Lan, G., Pasiliao Jr., E.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci. 8(1), 644–681 (2015)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Peng, Z., Wu, T., Xu, Y., Yan, M., Yin, W.: Coordinate friendly structures, algorithms and applications. Ann. Math. Sci. Appl. 1(1), 57–119 (2016)MATHGoogle Scholar
  35. 35.
    Peng, Z., Xu, Y., Yan, M., Yin, W.: Arock: an algorithmic framework for asynchronous parallel coordinate updates. SIAM J. Sci. Comput. 38(5), A2851–A2879 (2016)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Pesquet, J.-C., Repetti, A.: A class of randomized primal-dual algorithms for distributed optimization. arXiv preprint arXiv:1406.6404 (2014)
  37. 37.
    Richtárik, P., Takáč, M.: Parallel coordinate descent methods for big data optimization. Math. Program. 156(1–2), 433–484 (2016)Google Scholar
  38. 38.
    Richtárik, P., Takáč, M.: Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program. 144(1–2), 1–38 (2014)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Xu, Y.: Hybrid jacobian and gauss-seidel proximal block coordinate update methods for linearly constrained convex programming. arXiv preprint arXiv:1608.03928 (2016)
  40. 40.
    Xu, Y.: Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming. SIAM J. Optim. 27(3), 1459–1484 (2017)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Xu, Y.: Asynchronous parallel primal-dual block update methods. arXiv preprint arXiv:1705.06391 (2017)

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Department of Mathematical SciencesRensselaer Polytechnic InstituteTroyUSA
  2. 2.Department of Industrial and Systems EngineeringUniversity of MinnesotaMinneapolisUSA

Personalised recommendations