Fast bundle-level methods for unconstrained and ball-constrained convex optimization

  • Yunmei Chen
  • Guanghui LanEmail author
  • Yuyuan Ouyang
  • Wei Zhang


In this paper, we study a special class of first-order methods, namely bundle-level (BL) type methods, which can utilize historical first-order information through cutting plane models to accelerate the solutions in practice. Recently, it has been shown in Lan (149(1–2):1–45, 2015) that an accelerated prox-level (APL) method and its variant, the uniform smoothing level (USL) method, have optimal iteration complexity for solving black-box and structured convex programming (CP) problems without requiring input of any smoothness information. However, these algorithms require the assumption on the boundedness of the feasible set and their efficiency relies on the solutions of two involved subproblems. Some other variants of BL methods which could handle unbounded feasible set have no iteration complexity provided. In this work we develop the fast APL (FAPL) method and fast USL (FUSL) method that can significantly improve the practical performance of the APL and USL methods in terms of both computational time and solution quality. Both FAPL and FUSL enjoy the same optimal iteration complexity as APL and USL, while the number of subproblems in each iteration is reduced from two to one, and an exact method is presented to solve the only subproblem in these algorithms. Furthermore, we introduce a generic algorithmic framework to solve unconstrained CP problems through solutions to a series of ball-constrained CP problems that also exhibits optimal iteration complexity. Our numerical results on solving some large-scale least squares problems and total variation based image reconstructions have shown advantages of these new BL type methods over APL, USL, and some other first-order methods.


Convex programming First-order Optimal method Bundle-level method Total variation Image reconstruction 

Mathematics Subject Classification

90C25 90C06 90C22 49M37 



  1. 1.
    Lan, G.: Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization. Math. Program. 149(1–2), 1–45 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60(2), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Osher, S., Burger, M., Goldfarb, D., Jinjun, X., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Modeling Simul. 4(2), 460–489 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Nemirovski, A.S., Yudin, D.: Problem complexity and method efficiency in optimization. Wiley-Interscience Series in Discrete Mathematics. John Wiley, XV (1983)Google Scholar
  5. 5.
    Nesterov, Y.E.: Smooth minimization of nonsmooth functions. Math. Program. 103, 127–152 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Nesterov, Y.E.: A method for unconstrained convex minimization problem with the rate of convergence $O(1/k^2)$. Dokl. AN USSR. 269, 543–547 (1983)Google Scholar
  7. 7.
    Nesterov, Y.E.: Introductory Lectures on Convex Optimization: a Basic Course. Kluwer Academic Publishers, Massachusetts (2004)CrossRefzbMATHGoogle Scholar
  8. 8.
    Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    He, Y., Monteiro, R.D.C.: An accelerated hpe-type algorithm for a class of composite convex-concave saddle-point problems. SIAM J. Optim. 26(1), 29–56 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Hong, M., Luo, Z.Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 1–35 (2012)MathSciNetGoogle Scholar
  11. 11.
    Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Program. 141(1–2), 349–382 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci. 7(3), 1588–1623 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Ouyang, Y., Chen, Y., Lan, G., Pasiliao Jr., E.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci. 8(1), 644–681 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Kelley, J.E.: The cutting plane method for solving convex programs. J. SIAM. 8, 703–712 (1960)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Kiwiel, K.C.: An aggregate subgradient method for nonsmooth convex minimization. Math. Program. 27, 320–341 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Lemaréchal, C.: An extension of davidon methods to non-differentiable problems. Math. Program. Study. 3, 95–109 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Kiwiel, K.C.: Proximal level bundle method for convex nondifferentable optimization, saddle point problems and variational inequalities. Math. Program. Ser. B. 69, 89–109 (1995)zbMATHGoogle Scholar
  20. 20.
    Lemaréchal, C., Nemirovski, A.S., Nesterov, Y.E.: New variants of bundle methods. Math. Program. 69, 111–148 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Kiwiel, K.C.: Proximity control in bundle methods for convex nondifferentiable minimization. Math. Program. 46, 105–122 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Ben-Tal, A., Nemirovski, A.S.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102, 407–456 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    van Ackooij, W., Sagastizábal, C.: Constrained bundle methods for upper inexact oracles with application to joint chance constrained energy problems. SIAM J. Optim. 24(2), 733–765 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    de Oliveira, W., Sagastizábal, C., Scheimberg, S.: Inexact bundle methods for two-stage stochastic programming. SIAM J. Optim. 21(2), 517–544 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    de Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. 148(1–2), 241–277 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Richtárik, P.: Approximate level method for nonsmooth convex minimization. J. Optim. Theory Appl. 152(2), 334–350 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Kiwiel, K.C.: A proximal bundle method with approximate subgradient linearizations. SIAM J. optim. 16(4), 1007–1023 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Kiwiel, Krzysztof C: Bundle methods for convex minimization with partially inexact oracles. Comput. Optim. Appl., available from the web site SemanticScholarGoogle Scholar
  29. 29.
    de Oliveira, W, Sagastizábal, C: Level bundle methods for oracles with on-demand accuracy. Optim. Methods Softw. (ahead-of-print): 29,1–30 (2014)Google Scholar
  30. 30.
    Kiwiel, K.C., Lemaréchal, C.: An inexact bundle variant suited to column generation. Math. program. 118(1), 177–206 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Brännlund, U., Kiwiel, K.C., Lindberg, P.O.: A descent proximal level bundle method for convex nondifferentiable optimization. Op. Res. Lett. 17(3), 121–126 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Cruz, J.B., de Oliveira, W.: Level bundle-like algorithms for convex optimization. J. Glob. Optim. 59, 1–23 (2013)MathSciNetGoogle Scholar
  33. 33.
    de Oliveira, W., Sagastizábal, C.: Bundle methods in the xxist century: a bird’s-eye view. Pesqui. Op. 34(3), 647–670 (2014)CrossRefGoogle Scholar
  34. 34.
    Becker, S., Bobin, J., Candès, E.J.: Nesta: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4(1), 1–39 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Astorino, A., Frangioni, A., Gaudioso, M., Gorgone, E.: Piecewise-quadratic approximations in convex numerical optimization. SIAM J. Optim. 21(4), 1418–1438 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Ouorou, A.: A proximal cutting plane method using chebychev center for nonsmooth convex optimization. Math. Program. 119(2), 239–271 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Mosek. The mosek optimization toolbox for matlab manual. version 6.0 (revision 93).
  38. 38.
    Chen, Y., Hager, W., Huang, F., Phan, D., Ye, X., Yin, W.: Fast algorithms for image reconstruction with application to partially parallel mr imaging. SIAM J. Imaging Sci. 5(1), 90–118 (2012)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of MathematicsUniversity of FloridaGainesvilleUSA
  2. 2.H. Milton Stewart School of Industrial and Systems EngineeringGeorgia Institute of TechnologyAtlantaUSA
  3. 3.Department of Mathematical SciencesClemson UniversityClemsonUSA

Personalised recommendations