Advertisement

An Envelope for Davis–Yin Splitting and Strict Saddle-Point Avoidance

  • Yanli LiuEmail author
  • Wotao Yin
Article

Abstract

It is known that operator splitting methods based on forward–backward splitting, Douglas–Rachford splitting, and Davis–Yin splitting decompose difficult optimization problems into simpler subproblems under proper convexity and smoothness assumptions. In this paper, we identify an envelope (an objective function), whose gradient descent iteration under a variable metric coincides with Davis–Yin splitting iteration. This result generalizes the Moreau envelope for proximal-point iteration and the envelopes for forward–backward splitting and Douglas–Rachford splitting iterations identified by Patrinos, Stella, and Themelis. Based on the new envelope and the stable–center manifold theorem, we further show that, when forward–backward splitting or Douglas–Rachford splitting iterations start from random points, they avoid all strict saddle points with probability one. This result extends the similar results by Lee et al. from gradient descent to splitting methods.

Keywords

Splitting methods Strict saddle points Envelope Stable–center manifold theorem 

Mathematics Subject Classification

37L10 49J52 65K05 65K10 90C26 

Notes

Acknowledgements

This work is supported in part by NSF Grant DMS-1720237 and ONR Grant N000141712162.

References

  1. 1.
    Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)CrossRefGoogle Scholar
  2. 2.
    Candes, E.J., Plan, Y.: Matrix completion with noise. Proc. IEEE 98(6), 925–936 (2010)CrossRefGoogle Scholar
  3. 3.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  4. 4.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Passty, G.B.: Ergodic convergence to a zero of the sum of monotone operators in hilbert space. J. Math. Anal. Appl. 72(2), 383–390 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set-valued Var. Anal. 25(4), 829–858 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Stella, L., Themelis, A., Patrinos, P.: Forward–backward quasi-Newton methods for nonsmooth optimization problems. Comput. Optim. Appl. 67(3), 443–487 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Themelis, A., Stella, L., Patrinos, P.: Forward–backward envelope for the sum of two nonconvex functions: further properties and nonmonotone line-search algorithms. arXiv:1606.06256 (2016)
  9. 9.
    Li, G., Pong, T.K.: Douglas–Rachford splitting for nonconvex optimization with application to nonconvex feasibility problems. Math. Program. 159(1–2), 371–401 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Artacho, F.J.A., Borwein, J.M., Tam, M.K.: Douglas–Rachford feasibility methods for matrix completion problems. The ANZIAM J. 55(4), 299–326 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Chen, P., Fannjiang, A.: Fourier phase retrieval with a single mask by Douglas–Rachford algorithms. Appl. Comput. Harmon. Anal. 44(3), 665–699 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Themelis, A., Stella, L., Patrinos, P.: Douglas–Rachford splitting and ADMM for nonconvex optimization: new convergence results and accelerated versions. arXiv:1709.05747 (2017)
  14. 14.
    Guo, K., Han, D., Yuan, X.: Convergence analysis of Douglas–Rachford splitting method for “strongly \(+\) weakly” convex programming. SIAM J. Numer. Anal. 55(4), 1549–1577 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Li, G., Liu, T., Pong, T.K.: Peaceman–Rachford splitting for a class of nonconvex optimization problems. Comput. Optim. Appl. 68(2), 407–436 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Lee, J.D., Simchowitz, M., Jordan, M.I., Recht, B.: Gradient descent only converges to minimizers. In: Conference on Learning Theory, pp. 1246–1257 (2016)Google Scholar
  17. 17.
    Lee, J.D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M.I., Recht, B.: First-order methods almost always avoid saddle points. arXiv:1710.07406 (2017)
  18. 18.
    Shub, M.: Global Stability of Dynamical Systems. Springer, Berlin (2013)Google Scholar
  19. 19.
    Sun, J., Qu, Q., Wright, J.: Complete dictionary recovery over the sphere i: overview and the geometric picture. IEEE Trans. Inf. Theory 63(2), 853–884 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Brutzkus, A., Globerson, A.: Globally optimal gradient descent for a convnet with Gaussian inputs. In: International Conference on Machine Learning, pp. 605–614 (2017)Google Scholar
  21. 21.
    Sun, J., Qu, Q., Wright, J.: A geometric analysis of phase retrieval. In: IEEE International Symposium on Information Theory (ISIT), 2016, pp. 2379–2383. IEEE (2016)Google Scholar
  22. 22.
    Ge, R., Huang, F., Jin, C., Yuan, Y.: Escaping from saddle points—online stochastic gradient for tensor decomposition. In: Conference on Learning Theory, pp. 797–842 (2015)Google Scholar
  23. 23.
    Bhojanapalli, S., Neyshabur, B., Srebro, N.: Global optimality of local search for low rank matrix recovery. In: Advances in Neural Information Processing Systems, pp. 3873–3881 (2016)Google Scholar
  24. 24.
    Giselsson, P., Fält, M.: Envelope functions: unifications and further properties. J. Optim. Theory Appl. 178(3), 673–698 (2018)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, vol. 317. Springer, Berlin (2009)zbMATHGoogle Scholar
  26. 26.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, vol. 408. Springer, Berlin (2011)CrossRefzbMATHGoogle Scholar
  27. 27.
    Patrinos, P., Stella, L., Bemporad, A.: Douglas–Rachford splitting: complexity estimates and accelerated variants. In: 2014 IEEE 53rd Annual Conference on Decision and Control (CDC), pp. 4234–4239. IEEE (2014)Google Scholar
  28. 28.
    Stella, L.: Proximal envelopes: smooth optimization algorithms for nonsmooth problems. Ph.D. thesis, IMT School for Advanced Studies Lucca, Lucca, Italy (2017)Google Scholar
  29. 29.
    Zhang, F., Zhang, Q.: Eigenvalue inequalities for matrix product. IEEE Trans. Autom. Control 51(9), 1506–1509 (2006)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.University of California, Los AngelesLos AngelesUSA

Personalised recommendations