Advertisement

Dykstra’s Splitting and an Approximate Proximal Point Algorithm for Minimizing the Sum of Convex Functions

  • Chin How Jeffrey PangEmail author
Article
  • 30 Downloads

Abstract

We show that Dykstra’s splitting for projecting onto the intersection of convex sets can be extended to minimize the sum of convex functions and a regularizing quadratic function. We give conditions for which convergence to the primal minimizer holds so that more than one convex function can be minimized at a time, the convex functions are not necessarily sampled in a cyclic manner, and the SHQP strategy for problems involving the intersection of more than one convex set can be applied. When the sum does not involve the regularizing quadratic function, we discuss an approximate proximal point method combined with Dykstra’s splitting to minimize this sum.

Keywords

Dykstra’s splitting Proximal point algorithm Block coordinate minimization 

Mathematics Subject Classification

90C25 65K05 68Q25 47J25 

Notes

Acknowledgements

We acknowledge Grant R-146-000-214-112 from the Faculty of Science, National University of Singapore. We gratefully acknowledge discussions with Ting-Kei Pong on Dykstra’s splitting which led to this paper. We also thank the two anonymous referees and the editorial staff.

References

  1. 1.
    Dykstra, R.: An algorithm for restricted least-squares regression. J. Am. Stat. Assoc. 78, 837–842 (1983)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Boyle, J., Dykstra, R.: A method for finding projections onto the intersection of convex sets in Hilbert spaces. In: Dykstra, R., Robertson, T., Wright, F.T. (eds.) Advances in Order Restricted Statistical Inference. Lecture notes in Statistics, pp. 28–47. Springer, New York (1985)Google Scholar
  3. 3.
    Han, S.: A successive projection method. Math. Program. 40, 1–14 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Gaffke, N., Mathar, R.: A cyclic projection algorithm via duality. Metrika 36, 29–54 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Iusem, A., Pierro, A.D.: On the convergence of Han’s method of convex programming with quadratic objective. Math. Program. 52, 265–284 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Pierra, G.: Decomposition through formalization in a product space. Math. Program. 28, 96–115 (1984)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Hundal, H., Deutsch, F.: Two generalizations of Dykstra’s cyclic projections algorithm. Math. Program. 77, 335–355 (1997)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Pang, C.: The supporting halfspace—quadratic programming strategy for the dual of the best approximation problem. SIAM J. Optim. 26(4), 2591–2619 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Deutsch, F.: Accelerating the convergence of the method of alternating projections via a line search: a brief survey. In: Butnariu, D., Censor, Y., Reich, S. (eds.) Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications, pp. 203–217. Elsevier, Amsterdam (2001)CrossRefGoogle Scholar
  10. 10.
    Deutsch, F.: Best Approximation in Inner Product Spaces. CMS Books in Mathematics. Springer, Berlin (2001)CrossRefGoogle Scholar
  11. 11.
    Bauschke, H., Combettes, P.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  12. 12.
    Escalante, R., Raydan, M.: Alternating Projection Methods. SIAM, Philadelphia, PA (2011)Google Scholar
  13. 13.
    Beck, A., Tetruashvili, L.: On the convergence of block coordinate descent type methods. SIAM J. Optim. 23(4), 2037–2060 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Beck, A.: On the convergence of alternating minimization for convex programming with applications to iteratively reweighted least squares and decomposition schemes. SIAM J. Optim. 25(1), 185–209 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Tseng, P., Yun, S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. Ser. B 117(117), 387–423 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Tseng, P., Yun, S.: Block-coordinate gradient descent method for linearly constrained nonsmooth separable optimization. J. Optim. Theory Appl. 140, 513–535 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Wright, S.: Coordinate descent algorithms. Math. Program. 151, 3–34 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Hong, M., Wang, X., Razaviyayn, M., Luo, Z.: Iteration complexity analysis of block coordinate descent methods. Math. Program. 163, 85–114 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Rech. Opér. 4, 154–158 (1970)zbMATHGoogle Scholar
  20. 20.
    Rockafellar, R.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Han, S.: A decomposition method and its application to convex programming. Math. Oper. Res. 14, 237–248 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Bauschke, H., Combettes, P.: A Dykstra-like algorithm for two monotone operators. Pac. J. Optim. 4, 383–391 (2008)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Tseng, P.: Dual coordinate ascent methods for non-strictly convex minimization. Math. Program. 59, 231–247 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Combettes, P., Dũng, D., Vũ, B.: Proximity for sums of composite functions. J. Math. Anal. Appl. 380(2), 680–688 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Combettes, P., Dũng, D., Vũ, B.: Dualization of signal recovery problems. Set-Valued Var. Anal. 18, 373–404 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Abboud, F., Chouzenoux, E., Pesquet, J.C., Chenot, J.H., Laborelli, L.: Dual block-coordinate forward–backward algorithm with application to deconvolution and deinterlacing of video sequences. J. Math. Imaging Vis. 59(3), 415–431 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Nedić, A.: Random algorithms for convex minimization problems. Math. Program. Ser. B 225, 225–253 (2011)MathSciNetzbMATHGoogle Scholar
  28. 28.
    Nesterov, Y.: Introductory Lectures on Convex Optimization. Kluwer, London (2004)CrossRefzbMATHGoogle Scholar
  29. 29.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization (2008)Google Scholar
  31. 31.
    Bauschke, H., Borwein, J.: On projection algorithms for solving convex feasibility problems. SIAM Rev. 38, 367–426 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Censor, Y., Chen, W., Combettes, P.L., Davidi, R., Herman, G.: On the effectiveness of projection methods for convex feasibility problems with linear inequality constraints. Comput. Optim. Appl. 51, 1065–1088 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Neto, E.H., Pierro, A.D.: Incremental subgradients for constrained convex optimization: a unified framework and new methods. SIAM J. Optim. 20, 1547–1572 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Ram, S., Nedić, A., Veeravalli, V.: Incremental stochastic subgradient algorithms for convex optimization. SIAM J. Optim. 20, 691–717 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Censor, Y., Davidi, R., Herman, G.: Perturbation resilience and superiorization of iterative algorithms. Inverse Probl. 26(6), 065008 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Censor, Y., Davidi, R., Herman, G., Schulte, R., Tetruashvili, L.: Projected subgradient minimization versus superiorization. J. Optim. Theory Appl. 160, 730–747 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Bauschke, H., Borwein, J., Li, W.: Strong conical hull intersection property, bounded linear regularity, Jameson’s property (G), and error bounds in convex optimization. Math. Program., Ser. A 86(1), 135–160 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Burke, J., Deng, S.: Weak sharp minima revisited. II. Application to linear regularity and error bounds. Math. Program., Ser. B 104(2–3), 235–261 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Ng, K., Yang, W.: Regularities and their relations to error bounds. Math. Program., Ser. A 99, 521–538 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Kruger, A.: About regularity of collections of sets. Set-Valued Anal. 14, 187–206 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2010)CrossRefzbMATHGoogle Scholar
  42. 42.
    Combettes, P., Pesquet, J.C.: Proximal splitting methods in signal processing. In: Bauschke, H., Burachik, R., Combettes, P., Elser, V., Luke, D., Wolkowicz, H. (eds.) Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer, New York, NY (2011)Google Scholar
  43. 43.
    Pang, C.H.J.: Distributed deterministic asynchronous algorithms in time-varying graphs through Dykstra splitting. SIAM J. Optim. 29(1), 484–510 (2018)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of MathematicsNational University of SingaporeSingaporeSingapore

Personalised recommendations