Dual Convergence Estimates for a Family of Greedy Algorithms in Banach Spaces

  • S. P. SidorovEmail author
  • S. V. Mironov
  • M. G. Pleshakov
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10710)


The paper examines four weak relaxed greedy algorithms for finding approximate sparse solutions of convex optimization problems in a Banach space. First, we present a review of primal results on the convergence rate of the algorithms based on the geometric properties of the objective function. Then, using the ideas of [16], we define the duality gap and prove that the duality gap is a certificate for the current approximation to the optimal solution. Finally, we find estimates of the dependence of the duality gap values on the number of iterations for weak greedy algorithms.


Greedy algorithms Nonlinear optimization Sparsity 


  1. 1.
    Barron, A.R., Cohen, A., Dahmen, W., DeVore, R.A.: Approximation and learning by greedy algorithms. Ann. Stat. 36(1), 64–94 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Blumensath, T., Davies, M.E.: Gradient pursuits. IEEE Trans. Signal Process. 56, 2370–2382 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Blumensath, T., Davies, M.: Stagewise weak gradient pursuits. IEEE Trans. Signal Process. 57, 4333–4346 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Bubeck, S.: Convex optimization: algorithms and complexity. Found. Trends Mach. Learn. 8(3–4), 231–358 (2015)zbMATHCrossRefGoogle Scholar
  5. 5.
    Clarkson, K.L.: Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm. ACM Trans. Algorithms 6(4), 1–30 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  6. 6.
    Davis, G., Mallat, S., Avellaneda, M.: Adaptive greedy approximation. Constr. Approx. 13, 57–98 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Demyanov, V., Rubinov, A.: Approximate Methods in Optimization Problems. American Elsevier Publishing Co., New York (1970)Google Scholar
  8. 8.
    Dereventsov, A.V.: On the approximate weak Chebyshev greedy algorithm in uniformly smooth banach spaces. J. Math. Anal. Appl. 436(1), 288–304 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    DeVore, R.A., Temlyakov, V.N.: Some remarks on greedy algorithms. Adv. Comput. Math. 5, 173–187 (1996)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    DeVore, R.A., Temlyakov, V.N.: Convex optimization on Banach spaces. Found. Comput. Math. 16(2), 369–394 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Frank, M., Wolfe, P.: An algorithm for quadratic programming. Naval Res. Logis. Quart. 3, 95–110 (1956)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Freund, R.M., Grigas, P.: New analysis and results for the Frank-Wolfe method. Math. Program. 155(1), 199–230 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Friedman, J.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Georgiev, P.G., Sánchez-González, L., Pardalos, P.M.: Construction of pairs of reproducing kernel Banach spaces. In: Demyanov, V., Pardalos, P., Batsyn, M. (eds.) Constructive Nonsmooth Analysis and Related Topics. SOIA, vol. 87, pp. 39–57. Springer, New York (2014). Scholar
  15. 15.
    Huber, P.J.: Projection pursuit. Ann. Statist. 13, 435–525 (1985)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Jaggi, M.: Revisiting Frank-Wolfe: projection-free sparse convex optimization. In: Proceedings of the 30th International Conference on Machine Learning (ICML-13), pp. 427–435 (2013)Google Scholar
  17. 17.
    Jones, L.: On a conjecture of Huber concerning the convergence of projection pursuit regression. Ann. Statist. 15, 880–882 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Konyagin, S.V., Temlyakov, V.N.: A remark on greedy approximation in Banach spaces. East J. Approx. 5(3), 365–379 (1999)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Levitin, E.S., Polyak, B.T.: Constrained minimization methods. USSR Comp. Math. M. Phys. 6(5), 1–50 (1966)CrossRefGoogle Scholar
  20. 20.
    Nemirovski, A.: Optimization II: Numerical methods for nonlinear continuous optimization. Lecture Notes, Israel Institute of Technology (1999)Google Scholar
  21. 21.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Boston (2004)zbMATHCrossRefGoogle Scholar
  22. 22.
    Nguyen, H., Petrova, G.: Greedy strategies for convex optimization. Calcolo 41(2), 1–18 (2016)zbMATHGoogle Scholar
  23. 23.
    Polyak, B.T.: Introduction to Optimization. Optimization Software Inc., New York (1987)zbMATHGoogle Scholar
  24. 24.
    Temlyakov, V.N.: Greedy approximation in convex optimization. Constr. Approx. 41(2), 269–296 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Temlyakov, V.N.: Dictionary descent in optimization. Anal. Mathematica 42(1), 69–89 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  26. 26.
    Zhang, H., Zhang, J.: Learning with reproducing Kernel Banach spaces. In: Dang, P., Ku, M., Qian, T., Rodino, L.G. (eds.) New Trends in Analysis and Interdisciplinary Applications. TM, pp. 417–423. Springer, Cham (2017). Scholar
  27. 27.
    Zhang, Z., Shwartz, S., Wagner, L., Miller, W.: A greedy algorithm for aligning DNA sequences. J. Comput. Biol. 7(1–2), 203–214 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • S. P. Sidorov
    • 1
    Email author
  • S. V. Mironov
    • 1
  • M. G. Pleshakov
    • 1
  1. 1.Saratov State UniversitySaratovRussian Federation

Personalised recommendations