# Fast bundle-level methods for unconstrained and ball-constrained convex optimization

- 160 Downloads

## Abstract

In this paper, we study a special class of first-order methods, namely bundle-level (BL) type methods, which can utilize historical first-order information through cutting plane models to accelerate the solutions in practice. Recently, it has been shown in Lan (149(1–2):1–45, 2015) that an accelerated prox-level (APL) method and its variant, the uniform smoothing level (USL) method, have optimal iteration complexity for solving black-box and structured convex programming (CP) problems without requiring input of any smoothness information. However, these algorithms require the assumption on the boundedness of the feasible set and their efficiency relies on the solutions of two involved subproblems. Some other variants of BL methods which could handle unbounded feasible set have no iteration complexity provided. In this work we develop the fast APL (FAPL) method and fast USL (FUSL) method that can significantly improve the practical performance of the APL and USL methods in terms of both computational time and solution quality. Both FAPL and FUSL enjoy the same optimal iteration complexity as APL and USL, while the number of subproblems in each iteration is reduced from two to one, and an exact method is presented to solve the only subproblem in these algorithms. Furthermore, we introduce a generic algorithmic framework to solve unconstrained CP problems through solutions to a series of ball-constrained CP problems that also exhibits optimal iteration complexity. Our numerical results on solving some large-scale least squares problems and total variation based image reconstructions have shown advantages of these new BL type methods over APL, USL, and some other first-order methods.

## Keywords

Convex programming First-order Optimal method Bundle-level method Total variation Image reconstruction## Mathematics Subject Classification

90C25 90C06 90C22 49M37## Notes

## References

- 1.Lan, G.: Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization. Math. Program.
**149**(1–2), 1–45 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 2.Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom.
**60**(2), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar - 3.Osher, S., Burger, M., Goldfarb, D., Jinjun, X., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Modeling Simul.
**4**(2), 460–489 (2005)MathSciNetCrossRefzbMATHGoogle Scholar - 4.Nemirovski, A.S., Yudin, D.:
*Problem complexity and method efficiency in optimization*. Wiley-Interscience Series in Discrete Mathematics. John Wiley, XV (1983)Google Scholar - 5.Nesterov, Y.E.: Smooth minimization of nonsmooth functions. Math. Program.
**103**, 127–152 (2005)MathSciNetCrossRefzbMATHGoogle Scholar - 6.Nesterov, Y.E.: A method for unconstrained convex minimization problem with the rate of convergence $O(1/k^2)$. Dokl. AN USSR.
**269**, 543–547 (1983)Google Scholar - 7.Nesterov, Y.E.: Introductory Lectures on Convex Optimization: a Basic Course. Kluwer Academic Publishers, Massachusetts (2004)CrossRefzbMATHGoogle Scholar
- 8.Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim.
**24**(4), 1779–1814 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 9.He, Y., Monteiro, R.D.C.: An accelerated hpe-type algorithm for a class of composite convex-concave saddle-point problems. SIAM J. Optim.
**26**(1), 29–56 (2016)MathSciNetCrossRefzbMATHGoogle Scholar - 10.Hong, M., Luo, Z.Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program.
**162**, 1–35 (2012)MathSciNetGoogle Scholar - 11.Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput.
**66**(3), 889–916 (2016)MathSciNetCrossRefzbMATHGoogle Scholar - 12.Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Program.
**141**(1–2), 349–382 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 13.Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim.
**23**(1), 475–507 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 14.Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci.
**7**(3), 1588–1623 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 15.Ouyang, Y., Chen, Y., Lan, G., Pasiliao Jr., E.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci.
**8**(1), 644–681 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 16.Kelley, J.E.: The cutting plane method for solving convex programs. J. SIAM.
**8**, 703–712 (1960)MathSciNetzbMATHGoogle Scholar - 17.Kiwiel, K.C.: An aggregate subgradient method for nonsmooth convex minimization. Math. Program.
**27**, 320–341 (1983)MathSciNetCrossRefzbMATHGoogle Scholar - 18.Lemaréchal, C.: An extension of davidon methods to non-differentiable problems. Math. Program. Study.
**3**, 95–109 (1975)MathSciNetCrossRefzbMATHGoogle Scholar - 19.Kiwiel, K.C.: Proximal level bundle method for convex nondifferentable optimization, saddle point problems and variational inequalities. Math. Program. Ser. B.
**69**, 89–109 (1995)zbMATHGoogle Scholar - 20.Lemaréchal, C., Nemirovski, A.S., Nesterov, Y.E.: New variants of bundle methods. Math. Program.
**69**, 111–148 (1995)MathSciNetCrossRefzbMATHGoogle Scholar - 21.Kiwiel, K.C.: Proximity control in bundle methods for convex nondifferentiable minimization. Math. Program.
**46**, 105–122 (1990)MathSciNetCrossRefzbMATHGoogle Scholar - 22.Ben-Tal, A., Nemirovski, A.S.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program.
**102**, 407–456 (2005)MathSciNetCrossRefzbMATHGoogle Scholar - 23.van Ackooij, W., Sagastizábal, C.: Constrained bundle methods for upper inexact oracles with application to joint chance constrained energy problems. SIAM J. Optim.
**24**(2), 733–765 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 24.de Oliveira, W., Sagastizábal, C., Scheimberg, S.: Inexact bundle methods for two-stage stochastic programming. SIAM J. Optim.
**21**(2), 517–544 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 25.de Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program.
**148**(1–2), 241–277 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 26.Richtárik, P.: Approximate level method for nonsmooth convex minimization. J. Optim. Theory Appl.
**152**(2), 334–350 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 27.Kiwiel, K.C.: A proximal bundle method with approximate subgradient linearizations. SIAM J. optim.
**16**(4), 1007–1023 (2006)MathSciNetCrossRefzbMATHGoogle Scholar - 28.Kiwiel, Krzysztof C: Bundle methods for convex minimization with partially inexact oracles. Comput. Optim. Appl., available from the web site SemanticScholarGoogle Scholar
- 29.de Oliveira, W, Sagastizábal, C: Level bundle methods for oracles with on-demand accuracy. Optim. Methods Softw. (ahead-of-print): 29,1–30 (2014)Google Scholar
- 30.Kiwiel, K.C., Lemaréchal, C.: An inexact bundle variant suited to column generation. Math. program.
**118**(1), 177–206 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 31.Brännlund, U., Kiwiel, K.C., Lindberg, P.O.: A descent proximal level bundle method for convex nondifferentiable optimization. Op. Res. Lett.
**17**(3), 121–126 (1995)MathSciNetCrossRefzbMATHGoogle Scholar - 32.Cruz, J.B., de Oliveira, W.: Level bundle-like algorithms for convex optimization. J. Glob. Optim.
**59**, 1–23 (2013)MathSciNetGoogle Scholar - 33.de Oliveira, W., Sagastizábal, C.: Bundle methods in the xxist century: a bird’s-eye view. Pesqui. Op.
**34**(3), 647–670 (2014)CrossRefGoogle Scholar - 34.Becker, S., Bobin, J., Candès, E.J.: Nesta: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci.
**4**(1), 1–39 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 35.Astorino, A., Frangioni, A., Gaudioso, M., Gorgone, E.: Piecewise-quadratic approximations in convex numerical optimization. SIAM J. Optim.
**21**(4), 1418–1438 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 36.Ouorou, A.: A proximal cutting plane method using chebychev center for nonsmooth convex optimization. Math. Program.
**119**(2), 239–271 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 37.Mosek. The mosek optimization toolbox for matlab manual. version 6.0 (revision 93). http://www.mosek.com
- 38.Chen, Y., Hager, W., Huang, F., Phan, D., Ye, X., Yin, W.: Fast algorithms for image reconstruction with application to partially parallel mr imaging. SIAM J. Imaging Sci.
**5**(1), 90–118 (2012)MathSciNetCrossRefzbMATHGoogle Scholar