Journal of Scientific Computing

, Volume 54, Issue 2–3, pp 684–695

# Error Forgetting of Bregman Iteration

Article

## Abstract

This short article analyzes an interesting property of the Bregman iterative procedure, which is equivalent to the augmented Lagrangian method, for minimizing a convex piece-wise linear function J(x) subject to linear constraints Ax=b. The procedure obtains its solution by solving a sequence of unconstrained subproblems of minimizing $$J(x)+\frac{1}{2}\|Ax-b^{k}\|_{2}^{2}$$, where b k is iteratively updated. In practice, the subproblem at each iteration is solved at a relatively low accuracy. Let w k denote the error introduced by early stopping a subproblem solver at iteration k. We show that if all w k are sufficiently small so that Bregman iteration enters the optimal face, then while on the optimal face, Bregman iteration enjoys an interesting error-forgetting property: the distance between the current point $$\bar{x}^{k}$$ and the optimal solution set X is bounded by ∥w k+1w k ∥, independent of the previous errors w k−1,w k−2,…,w 1. This property partially explains why the Bregman iterative procedure works well for sparse optimization and, in particular, for 1-minimization. The error-forgetting property is unique to J(x) that is a piece-wise linear function (also known as a polyhedral function), and the results of this article appear to be new to the literature of the augmented Lagrangian method.

## Keywords

Bregman iteration Error forgetting Sparse optimization 1 minimization Piece-wise linear function Polyhedral function Augmented Lagrangian method Method of multipliers

## Notes

### Acknowledgements

The authors thank Professor Yin Zhang and two anonymous reviewers for their valuable comments.

## References

1. 1.
Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4(2), 460–489 (2005)
2. 2.
Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for 1-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1(1), 143–168 (2008)
3. 3.
Figueiredo, M., Nowak, R., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 4(1), 586–597 (2007). Special Issue on Convex Optimization Methods for Signal Processing
4. 4.
Hale, E.T., Yin, W., Zhang, Y.: Fixed-point continuation for 1-minimization: methodology and convergence. SIAM J. Optim. 19(3), 1107–1130 (2008)
5. 5.
Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)
6. 6.
Xu, J., Osher, S.: Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Trans. Image Process. 16(2), 534–544 (2006)
7. 7.
Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Commun. Math. Sci. 4(1), 175–208 (2006)
8. 8.
Burger, M., Osher, S., Xu, J., Gilboa, G.: Nonlinear inverse scale space methods for image restoration. In: Variational, Geometric, and Level Set Methods in Computer Vision. Lecture Notes in Computer Science, vol. 3752, pp. 25–36 (2005)
9. 9.
He, L., Chang, T.-C., Osher, S., Fang, T., Speier, P.: MR image reconstruction by using the iterative refinement method and nonlinear inverse scale space methods. UCLA CAM Report 06-35 (2006) Google Scholar
10. 10.
Ma, S., Goldfarb, D., Chen, L.: Fixed point and Bregman iterative methods for matrix rank minimization. Math. Program. pp 1–33 (2009) Google Scholar
11. 11.
Guo, Z., Wittman, T., Osher, S.: L1 unmixing and its application to hyperspectral image enhancement. In: Proceedings of SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV, 7334:73341M (2009) Google Scholar
12. 12.
Bregman, L.: The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. U.S.S.R. Comput. Math. Math. Phys. 7, 200–217 (1967)
13. 13.
Cai, J.-F., Osher, S., Shen, Z.: Linearized Bregman iterations for compressed sensing. Math. Comput. 78(267), 1515–1536 (2008)
14. 14.
Cai, J.-F., Osher, S., Shen, Z.: Convergence of the linearized Bregman iteration for 1-norm minimization. Math. Comput. 78(268), 2127–2136 (2009)
15. 15.
Yin, W.: Analysis and generalizations of the linearized Bregman method. SIAM J. Imaging Sci. 3(4), 856–877 (2010)
16. 16.
Friedlander, M., Tseng, P.: Exact regularization of convex programs. SIAM J. Optim. 18, 4 (2007)
17. 17.
Lai, M.-J., Yin, W.: Augmented 1 and nuclear-norm models with a globally linearly convergent algorithm. Rice University CAAM Technical Report TR12-02 (2012) Google Scholar
18. 18.
Cai, J.-F., Candes, E., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20(4), 2008 (1956–1982)
19. 19.
Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)
20. 20.
Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1972) Google Scholar
21. 21.
Esser, E.: Applications of Lagrangian-based alternating direction methods and connections to split Bregman. UCLA CAM Report 09-31 (2009) Google Scholar
22. 22.
Bertsekas, D.P.: Combined primal-dual and penalty methods for constrained minimization. SIAM J. Control 13(3), 521–544 (1975)
23. 23.
Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Scientific, Athena (1996) Google Scholar
24. 24.
Mosek ApS Inc.: The Mosek optimization tools, ver. 4 (2006) Google Scholar
25. 25.
Bertsekas, D.P.: Necessary and sufficient conditions for a penalty method to be exact. Math. Program. 9(1), 87–99 (1975)
26. 26.
Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)
27. 27.
Eckstein, J., Silva, P.J.S.: A practical relative error criterion for augmented Lagrangians. Math. Program. pp. 1–30 (2010) Google Scholar
28. 28.
Burger, M., Moller, M., Benning, M., Osher, S.: An adaptive inverse scale space method for compressed sensing. UCLA CAM Report 11-08 (2011) Google Scholar