Computational Optimization and Applications

, Volume 54, Issue 2, pp 343–369 | Cite as

An ADM-based splitting method for separable convex programming

  • Deren Han
  • Xiaoming Yuan
  • Wenxing Zhang
  • Xingju Cai


We consider the convex minimization problem with linear constraints and a block-separable objective function which is represented as the sum of three functions without coupled variables. To solve this model, it is empirically effective to extend straightforwardly the alternating direction method of multipliers (ADM for short). But, the convergence of this straightforward extension of ADM is still not proved theoretically. Based on ADM’s straightforward extension, this paper presents a new splitting method for the model under consideration, which is empirically competitive to the straightforward extension of ADM and meanwhile the global convergence can be proved under standard assumptions. At each iteration, the new method corrects the output of the straightforward extension of ADM by some slight correction computation to generate a new iterate. Thus, the implementation of the new method is almost as easy as that of ADM’s straightforward extension. We show the numerical efficiency of the new method by some applications in the areas of image processing and statistics.


Convex minimization Block-separable Alternating direction method of multipliers Operator splitting methods Global convergence 



The research of the first author is supported by the National Natural Science Foundation of China (NSFC) grants 11071122 and 10871098, and Grant 20103207110002 from MOE of China. The research of the second author is supported by the Hong Kong General Research Fund: 203311. The third author would like to thank Dr. Caihua Chen for his helpful comments.


  1. 1.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010) CrossRefMATHGoogle Scholar
  2. 2.
    Cai, J.F., Chan, R.H., Nikolova, M.: Two phase methods for deblurring images corrupted by impulse plus Gaussian noise. Inverse Probl. Imaging 2, 187–204 (2008) MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Candés, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis. J. ACM 58(1), 1–37 (2011) CrossRefGoogle Scholar
  4. 4.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011) MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Chandrasekaran, V., Sanghavi, S., Parrilo, P., Willskyc, A.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21(2), 572–596 (2011) MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994) MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Chen, C.H., He, B.S., Yuan, X.M.: Matrix completion via alternating direction method. IMA J. Numer. Anal. 32, 227–245 (2012) MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Esser, E.: Applications of Lagrangian-based alternating direction methods and connections to split Bregman. UCLA CAM Report. 9-31 (2009) Google Scholar
  9. 9.
    Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems, vols. I and II. Springer, Berlin (2003) Google Scholar
  10. 10.
    Fukushima, M.: Application of the alternating directions method of multipliers to separable convex programming problems. Comput. Optim. Appl. 2, 93–111 (1992) MathSciNetCrossRefGoogle Scholar
  11. 11.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 17–40 (1976) CrossRefMATHGoogle Scholar
  12. 12.
    Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM Studies in Applied Mathematics. SIAM, Philadelphia (1989) CrossRefMATHGoogle Scholar
  13. 13.
    Glowinski, R., Marrocco, A.: Approximation par éléments finis d’ordre un et résolution par pénalisation-dualité d’une classe de problèmes non linéaires. RAIRO. Anal. Numér. 9(2), 41–76 (1975) MathSciNetMATHGoogle Scholar
  14. 14.
    Hansen, P., Nagy, J., O’Leary, D.: Deblurring Images: Matrices, Spectra, and Filtering. SIAM, Philadelphia (2006) CrossRefMATHGoogle Scholar
  15. 15.
    He, B.S.: Parallel splitting augmented Lagrangian methods for monotone structured variational inequalities. Comput. Optim. Appl. 42, 195–212 (2009) MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    He, B.S., Liao, L., Han, D., Yang, H.: A new inexact alternating direction method for monotone variational inequalities. Math. Program. 92, 103–118 (2002) MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    He, B.S., Xu, M.H., Yuan, X.M.: Solving large-scale least squares covariance matrix problems by alternating direction methods. SIAM J. Matrix Anal. Appl. 32, 136–152 (2011) MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    He, B.S., Tao, M., Yuan, X.M.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012) CrossRefMATHGoogle Scholar
  19. 19.
    He, B.S., Tao, M., Xu, M.H., Yuan, X.M.: Alternating directions based contraction method for generally separable linearly constrained convex programming problems. Optimization (to appear) Google Scholar
  20. 20.
    He, B.S., Tao, M., Yuan, X.M.: A splitting method for separable convex programming. IMA J. Num. Anal. (in revision) Google Scholar
  21. 21.
    Hestenes, M.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969) MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Huang, Y.M., Ng, M.K., Wen, Y.W.: Fast image restoration methods for impulse and Gaussian noise removal. IEEE Signal Process. Lett. 16, 457–460 (2009) CrossRefGoogle Scholar
  23. 23.
    Hwang, H., Haddad, A.: Adaptive median filters: new algorithms and results. IEEE Trans. Image Process. 4, 499–502 (1995) CrossRefGoogle Scholar
  24. 24.
    Kontogiorgis, S., Meyer, R.: A variable-penalty alternating directions method for convex optimization. Math. Program. 83, 29–53 (1998) MathSciNetMATHGoogle Scholar
  25. 25.
    Martinet, B.: Regularisation, d’inéquations variationelles par approximations succesives. RAIRO. Rech. Opér. 4, 154–159 (1970) MathSciNetMATHGoogle Scholar
  26. 26.
    Ng, M.K., Weiss, P., Yuan, X.M.: Solving constrained total-variation problems via alternating direction methods. SIAM J. Sci. Comput. 32, 2710–2736 (2010) MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Ng, M.K., Wang, F., Yuan, X.M.: Inexact alternating direction methods for image recovery. SIAM J. Sci. Comput. 33(4), 1643–1668 (2011) MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intel. (to appear) Google Scholar
  29. 29.
    Powell, M.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969) Google Scholar
  30. 30.
    Rockafellar, R.: Convex Analysis. Princeton University Press, Princeton (1970) MATHGoogle Scholar
  31. 31.
    Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) CrossRefMATHGoogle Scholar
  32. 32.
    Sun, J., Zhang, S.: A modified alternating direction method for convex quadratically constrained quadratic semidefinite programs. Eur. J. Oper. Res. 207, 1210–1220 (2010) CrossRefMATHGoogle Scholar
  33. 33.
    Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011) MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused lasso. J. R. Stat. Soc. B 67, 91–108 (2005) MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Wen, Z.W., Goldfarb, D.: Line search multigrid method for large-scale convex optimization. SIAM J. Optim. 20, 1478–1503 (2009) MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Wen, Z.W., Goldfarb, D., Yin, W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010) MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Yang, J.F., Zhang, Y.: Alternating direction algorithms for l 1-problems in compressive sensing. SIAM J. Sci. Comput. 33, 250–278 (2011) MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Yuan, X.M.: Alternating direction methods for covariance selection models. J. Sci. Comput. 51, 261–273 (2012) MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  • Deren Han
    • 1
  • Xiaoming Yuan
    • 2
  • Wenxing Zhang
    • 3
  • Xingju Cai
    • 3
  1. 1.School of Mathematical Sciences and Jiangsu Key Laboratory for NSLSCSNanjing Normal UniversityNanjingP.R. China
  2. 2.Department of MathematicsHong Kong Baptist UniversityHong KongP.R. China
  3. 3.Department of MathematicsNanjing UniversityNanjingP.R. China

Personalised recommendations