Skip to main content

A limited memory steepest descent method

Mathematical Programming volume 135, pages 413–436 (2012)Cite this article

Abstract

The possibilities inherent in steepest descent methods have been considerably amplified by the introduction of the Barzilai–Borwein choice of step-size, and other related ideas. These methods have proved to be competitive with conjugate gradient methods for the minimization of large dimension unconstrained minimization problems. This paper suggests a method which is able to take advantage of the availability of a few additional ‘long’ vectors of storage to achieve a significant improvement in performance, both for quadratic and non-quadratic objective functions. It makes use of certain Ritz values related to the Lanczos process (Lanczos in J Res Nat Bur Stand 45:255–282, 1950). Some underlying theory is provided, and numerical evidence is set out showing that the new method provides a competitive and more simple alternative to the state of the art l-BFGS limited memory method.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

  1. Barzilai J., Borwein J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    MathSciNet  MATH  Article  Google Scholar 

  2. Byrd R.H., Nocedal J., Schnabel R.B.: Representations of Quasi-Newton matrices and their use in limited memory methods. Math. Progr. 63, 129–156 (1994)

    MathSciNet  MATH  Article  Google Scholar 

  3. Cauchy A.: Méthode générale pour la résolution des systèms d’équations simultanées. Comp. Rend. Sci. Paris 25, 536–538 (1847)

    Google Scholar 

  4. Dai Y.H., Fletcher R.: On the asymptotic behaviour of some new gradient methods. Math. Progr. 103, 541–559 (2005)

    MathSciNet  MATH  Article  Google Scholar 

  5. Dai Y.H., Fletcher R.: Projected Barzilai–Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21–47 (2005)

    MathSciNet  MATH  Article  Google Scholar 

  6. Dai Y.H., Yuan Y.: Analysis of monotone gradient methods. J. Ind. Manag. Optim. 1, 181–192 (2005)

    MathSciNet  MATH  Article  Google Scholar 

  7. Fletcher R.: Practical Methods of Optimization. 2nd edn. Wiley, Chichester (1987)

    MATH  Google Scholar 

  8. Fletcher, R.: Low storage methods for unconstrained optimization. In: Allgower, E.L., Georg, K. (eds.) Computational Solution of Nonlinear Systems of Equations. Lectures in Applied Mathematics (AMS), vol. 26, pp. 165–179 (1990)

  9. Fletcher, R.: On the Barzilai–Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications, Series in Applied Optimization, vol. 96. Kluwer, pp. 235–256 (2005)

  10. Fletcher R., Powell M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6, 163–168 (1963)

    MathSciNet  MATH  Google Scholar 

  11. Friedlander A., Martínez J.M., Molina B., Raydan M.: Gradient method with retards and generalizations. SIAM J. Numer. Anal. 36, 275–289 (1999)

    MathSciNet  Article  Google Scholar 

  12. Golub G.H., Van Loan C.F.: Matrix Computations. 3rd edn. The Johns Hopkins Press, Baltimore (1996)

    MATH  Google Scholar 

  13. Gould N.I.M., Orban D., Toint Ph.L.: CUTEr (and SifDec), a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29, 373–394 (2003)

    MathSciNet  MATH  Article  Google Scholar 

  14. Grippo L., Lampariello F., Lucidi S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986)

    MathSciNet  MATH  Article  Google Scholar 

  15. Lanczos C.: An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Res. Nat. Bur. Stand. 45, 255–282 (1950)

    MathSciNet  Article  Google Scholar 

  16. Molina B., Raydan M.: Preconditioned Barzilai–Borwein method for the numerical solution of partial differential equations. Numer. Algoritm. 13, 45–60 (1996)

    MathSciNet  MATH  Article  Google Scholar 

  17. Nocedal J.: Updating quasi-Newton matrices with limited storage. Math. Comput. 35, 773–782 (1980)

    MathSciNet  MATH  Article  Google Scholar 

  18. Paige C.C., Parlett B.N., van der Vorst H.: Approximate solutions and eigenvalue bounds from Krylov subspaces. Numer. Linear Algebra Appl. 2, 115–133 (1995)

    MathSciNet  MATH  Article  Google Scholar 

  19. Raydan M.: On the Barzilai and Borwein choice of steplength for the gradient method. IMA J. Numer. Anal. 13, 321–326 (1993)

    MathSciNet  MATH  Article  Google Scholar 

  20. Raydan M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7, 26–33 (1997)

    MathSciNet  MATH  Article  Google Scholar 

  21. Raydan M., Svaiter B.F.: Relaxed steepest descent and Cauchy-Barzilai–Borwein method. Comput. Optim. Appl. 21, 155–167 (2002)

    MathSciNet  MATH  Article  Google Scholar 

  22. Serafini T., Zanghirati G., Zanni L.: Gradient projection methods for quadratic programs and applications in support vector machines. Optim. Methods Softw. 20, 353–378 (2005)

    MathSciNet  MATH  Article  Google Scholar 

  23. Toint Ph.L.: Some numerical results using a sparse matrix updating formula in unconstrained optimization. Math. Comput. 32, 839–852 (1978)

    MathSciNet  MATH  Article  Google Scholar 

  24. Varadhan R., Gilbert P.D.: BB: an R package for solving a large system of nonlinear equations and for optimizing a high-dimensional nonlinear objective function. J. Stat. Softw. 32, 1–26 (2009)

    Google Scholar 

  25. Vargas W.E., Azofeifa D.E., Clark N.: Retrieved optical properties of thin films on absorbing substrates from transmittance measurements by application of a spectral projected gradient method. Thin Solid Films 425, 1–8 (2003)

    Article  Google Scholar 

  26. Yuan Y.: A new stepsize for the steepest descent method. J. Comput. Math. 24, 149–156 (2006)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Dundee, Dundee, DD1 4HN, Scotland, UK

    Roger Fletcher

Authors
  1. Roger Fletcher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roger Fletcher.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fletcher, R. A limited memory steepest descent method. Math. Program. 135, 413–436 (2012). https://doi.org/10.1007/s10107-011-0479-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-011-0479-6

Mathematics Subject Classification (2000)

  • 90C06
  • 90C26
  • 65K05