Advertisement

A simple convergence analysis of Bregman proximal gradient algorithm

  • Yi ZhouEmail author
  • Yingbin Liang
  • Lixin Shen
Article
  • 22 Downloads

Abstract

In this paper, we provide a simple convergence analysis of proximal gradient algorithm with Bregman distance, which provides a tighter bound than existing result. In particular, for the problem of minimizing a class of convex objective functions, we show that proximal gradient algorithm with Bregman distance can be viewed as proximal point algorithm that incorporates another Bregman distance. Consequently, the convergence result of the proximal gradient algorithm with Bregman distance follows directly from that of the proximal point algorithm with Bregman distance, and this leads to a simpler convergence analysis with a tighter convergence bound than existing ones. We further propose and analyze the backtracking line-search variant of the proximal gradient algorithm with Bregman distance.

Keywords

Proximal algorithms Bregman distance Convergence analysis Line-search 

Notes

References

  1. 1.
    Auslender, A., Teboulle, M.: Interior gradient and proximal methods for convex and conic optimization. SIAM J. Optim. 16(3), 697–725 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: Smoothing and first order methods: a unified framework. SIAM J. Optim. 22(2), 557–580 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bolte, J., Bauschke, H., Teboulle, M.: A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications. Math. Oper. Res. 42, 330–348 (2016)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  7. 7.
    Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Censor, Y., Zenios, S.A.: Proximal minimization algorithm with d-functions. J. Optim. Theory Appl. 73(3), 451–464 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Chen, G., Teboulle, M.: Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM J. Optim. 3(3), 538–543 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    De Pierro, A.R., Iusem, A.N.: A relaxed version of Bregman’s method for convex programming. J. Optim. Theory Appl. 51(3), 421–440 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Eckstein, J.: Nonlinear proximal point algorithms using Bregman functions, with applications to convex programming. Math. Oper. Res. 18(1), 202–226 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Fukushima, M., Mine, H.: A generalized proximal point algorithm for certain non-convex minimization problems. Int. J. Syst. Sci. 12(8), 989–1000 (1981)CrossRefzbMATHGoogle Scholar
  14. 14.
    Güler, O.: On the convergence of the proximal point algorithm for convex minimization. SIAM J. Control Optim. 29(2), 403–419 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Lions, P., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Martinet, B.: Bréve communication: régularisation d’inéquations variationnellespar approximations successives. ESAIM Math. Modell. Numer. Anal. 4(R3), 154–158 (1970)zbMATHGoogle Scholar
  17. 17.
    Micchelli, C.A., Shen, L., Xu, Y.: Proximity algorithms for image models: denoising. Inverse Probl. 27, 045009 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Rockafellar, R.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Teboulle, M.: Entropic proximal mappings with applications to nonlinear programming. Math. Oper. Res. 17(3), 670–690 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Teboulle, M.: Convergence of proximal-like algorithms. SIAM J. Optim. 7(4), 1069–1083 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Tseng, P.: Approximation accuracy, gradient methods, and error bound for structured convex optimization. Math. Program. 125(2), 263–295 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Zhou, Y., Liang, Y., Shen, L.: A new perspective of proximal gradient algorithms. arXiv: 1503.05601 (2015)

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringDuke UniversityDurhamUSA
  2. 2.Department of Electrical and Computer EngineeringThe Ohio State UniversityColumbusUSA
  3. 3.Department of MathematicsSyracuse UniversitySyracuseUSA

Personalised recommendations