Advertisement

Journal of Scientific Computing

, Volume 70, Issue 3, pp 1204–1228 | Cite as

Two-Step Fixed-Point Proximity Algorithms for Multi-block Separable Convex Problems

  • Qia Li
  • Yuesheng Xu
  • Na Zhang
Article

Abstract

Multi-block separable convex problems recently received considerable attention. Optimization problems of this type minimize separable convex objective functions with linear constraints. Challenges encountered in algorithmic development applying the classic alternating direction method of multipliers (ADMM) come from the fact that ADMM for the optimization problems of this type is not necessarily convergent. However, it is observed that ADMM applying to problems of this type outperforms numerically many of its variants with guaranteed theoretical convergence. The goal of this paper is to develop convergent and computationally efficient algorithms for solving multi-block separable convex problems. We first characterize the solutions of the optimization problems by proximity operators of the convex functions involved in their objective functions. We then design a class of two-step fixed-point iterative schemes for solving these problems based on the characterization. We further prove convergence of the iterative schemes and show that they have \(O\left( \frac{1}{k}\right) \) of convergence rate in the ergodic sense and the sense of the partial primal-dual gap, where k denotes the iteration number. Moreover, we derive specific two-step fixed-point proximity algorithms (2SFPPA) from the proposed iterative schemes and establish their global convergence. Numerical experiments for solving the sparse MRI problem demonstrate the numerical efficiency of the proposed 2SFPPA.

Keywords

Multi-block separable convex problems Fixed-point proximity algorithms Two-step algorithms 

References

  1. 1.
    Attouch, H., Briceno-Arias, L.M., Combettes, P.L.: A parallel splitting method for coupled monotone inclusions. SIAM J. Control Optim. 48, 3246–3270 (2010)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, AMS Books in Mathematics. Springer, New York (2011)CrossRefMATHGoogle Scholar
  3. 3.
    Cai, J., Chan, R., Shen, Z.: A framelet-based image inpainting algorithm. Appl. Comput. Harmonic Anal. 24, 131–149 (2007)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. (2016). doi: 10.1007/s10589-016-9860-y Google Scholar
  5. 5.
    Cai, J., Osher, S., Shen, Z.: Linearized Bregman iteration for frame based image deblurring. SIAM J. Imaging Sci. 2, 226–252 (2009)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Chan, R., Chan, T., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24, 1408–1432 (2003)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Chan, R., Riemenschneider, S.D., Shen, L., Shen, Z.: Tight frame: the efficient way for high-resolution image reconstruction. Appl. Comput. Harmonic Anal. 17, 91–115 (2004)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of admm for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155, 57–79 (2016)Google Scholar
  10. 10.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)MATHGoogle Scholar
  11. 11.
    Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. UCLA CAM Report 15–13Google Scholar
  12. 12.
    Deng, W., Lai, M.-J., Peng, Z., Yin, W.: Parallel multi-block admm with o(1/k) convergence, UCLA CAM 13–64 (2014)Google Scholar
  13. 13.
    Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3, 1015–1046 (2010)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. comput. math. appl. 2(1), 17–40. Comput. Math.Appl. 2, 17–40 (1976)CrossRefMATHGoogle Scholar
  15. 15.
    Goldstein, T., Osher, S.: The split Bregman method for \(\ell ^1\) regularization problems. SIAM J. Imaging Sci. 2, 323–343 (2009)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    He, B., Tao, M., Yuan, X.: Alternating direction method with gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    He, B., Yuan, X.: Linearized alternating direction method of multipliers with gaussian back substitution for seperable convex programming. Numer. Algebra Control Optim. 22, 247–260 (2013)CrossRefMATHGoogle Scholar
  18. 18.
    Li, M., Sun, D., Toh, K.-C.: A convergent 3-block semi-proximal admm for convex minimization problems with one strongly convex block. Asia Pacific J. Oper. Res. 32, 1550024 (2015)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Li, Q., Micchelli, C.A., Shen, L., Xu, Y.: A proximity algorithm accelerated by Gauss-Seidel iterations for L1/TV denoising models. Inverse Probl. 28, 095003 (2012)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Li, Q., Shen, L., Yuesheng, X., Zhang, N.: Multi-step fixed-point proximity algorithms for solving a class of convex optimization problems arising from image processing. Adv. Comput. Math. 41, 387–422 (2015)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Li, Q., Shen, L., Yang, L.: Split-bregman iteration for framelet based image inpainting. Appl. Comput. Harmonic Anal. 32, 145–154 (2012)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Li, Q., Zhang, N.: Fast proximity-gradient algorithms for structured convex optimization problems. Appl. Comput. Harmonic Anal. 41, 491–517 (2016)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Lin, T., Ma, S., Zhang, S.: On the convergence rate of multi-block admm. J. Oper. Res. Soc. China 3, 251–274 (2015)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007)CrossRefGoogle Scholar
  25. 25.
    Micchelli, C.A., Shen, L., Xu, Y.: Proximity algorithms for image models: denoising. Inverse Probl. 27, 045009 (2011)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Moreau, J.J.: Fonctions convexes duales et points proximaux dans un espace hilbertien. C.R. Acad. Sci. Paris Sér. A Math. 255, 1897–2899 (1962)MathSciNetMATHGoogle Scholar
  27. 27.
    Nikolova, M.: Local strong homogeneity of a regularized estimator. SIAM J. Appl. Math. 61, 633–658 (2000)MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Rudin, L.I., Osher, S.: Total variation based image restoration with free local constraints, In: IEEE International Conference on Image Processing, pp. 31–35 (1994)Google Scholar
  29. 29.
    Ruszczynski, A.: Parallel decomposition of multistage stochastic programming problems. Math. Program. 58, 201–228 (1993)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Sawatzky, A., Qi, X., Schirra, C.O., Anastasio, M.A.: Proximal ADMM for multi-channel image reconstruction in spectral X-ray CT. IEEE Trans. Med. Imaging 33, 1657–1668 (2014)CrossRefGoogle Scholar
  31. 31.
    Shi, W., Ling, W., Wu, W., Yin, W.: Extra: an exact first-order algorithm for decentralized consensus optimization. SIAM J. Optim. 25, 944–966 (2015)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Sun, D., Toh, K.C., Yang, L.: A convergent 3-block semi-proximal alternating direction method of multipliers for conic programming with \(4\)-type of constraints. SIAM J. Optim. 25, 882–915 (2014)Google Scholar
  33. 33.
    Tyrrell Rockafellar, R.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. 67, 91–108 (2005)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. 58, 267–288 (1996)MathSciNetMATHGoogle Scholar
  36. 36.
    Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Applied Mathematics, College of Mathematics and InformaticsSouth China Agricultural UniversityGuangzhouPeople’s Republic of China
  2. 2.School of Data and Computer Sciences and Guangdong Province Key Laboratory of Computational ScienceSun Yat-sen UniversityGuangzhouPeople’s Republic of China
  3. 3.Syracuse UniversitySyracuseUSA

Personalised recommendations