Abstract
The possibilities inherent in steepest descent methods have been considerably amplified by the introduction of the Barzilai–Borwein choice of step-size, and other related ideas. These methods have proved to be competitive with conjugate gradient methods for the minimization of large dimension unconstrained minimization problems. This paper suggests a method which is able to take advantage of the availability of a few additional ‘long’ vectors of storage to achieve a significant improvement in performance, both for quadratic and non-quadratic objective functions. It makes use of certain Ritz values related to the Lanczos process (Lanczos in J Res Nat Bur Stand 45:255–282, 1950). Some underlying theory is provided, and numerical evidence is set out showing that the new method provides a competitive and more simple alternative to the state of the art l-BFGS limited memory method.
This is a preview of subscription content, log in via an institution to check access.
References
Barzilai J., Borwein J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)
Byrd R.H., Nocedal J., Schnabel R.B.: Representations of Quasi-Newton matrices and their use in limited memory methods. Math. Progr. 63, 129–156 (1994)
Cauchy A.: Méthode générale pour la résolution des systèms d’équations simultanées. Comp. Rend. Sci. Paris 25, 536–538 (1847)
Dai Y.H., Fletcher R.: On the asymptotic behaviour of some new gradient methods. Math. Progr. 103, 541–559 (2005)
Dai Y.H., Fletcher R.: Projected Barzilai–Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21–47 (2005)
Dai Y.H., Yuan Y.: Analysis of monotone gradient methods. J. Ind. Manag. Optim. 1, 181–192 (2005)
Fletcher R.: Practical Methods of Optimization. 2nd edn. Wiley, Chichester (1987)
Fletcher, R.: Low storage methods for unconstrained optimization. In: Allgower, E.L., Georg, K. (eds.) Computational Solution of Nonlinear Systems of Equations. Lectures in Applied Mathematics (AMS), vol. 26, pp. 165–179 (1990)
Fletcher, R.: On the Barzilai–Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications, Series in Applied Optimization, vol. 96. Kluwer, pp. 235–256 (2005)
Fletcher R., Powell M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6, 163–168 (1963)
Friedlander A., Martínez J.M., Molina B., Raydan M.: Gradient method with retards and generalizations. SIAM J. Numer. Anal. 36, 275–289 (1999)
Golub G.H., Van Loan C.F.: Matrix Computations. 3rd edn. The Johns Hopkins Press, Baltimore (1996)
Gould N.I.M., Orban D., Toint Ph.L.: CUTEr (and SifDec), a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29, 373–394 (2003)
Grippo L., Lampariello F., Lucidi S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986)
Lanczos C.: An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Res. Nat. Bur. Stand. 45, 255–282 (1950)
Molina B., Raydan M.: Preconditioned Barzilai–Borwein method for the numerical solution of partial differential equations. Numer. Algoritm. 13, 45–60 (1996)
Nocedal J.: Updating quasi-Newton matrices with limited storage. Math. Comput. 35, 773–782 (1980)
Paige C.C., Parlett B.N., van der Vorst H.: Approximate solutions and eigenvalue bounds from Krylov subspaces. Numer. Linear Algebra Appl. 2, 115–133 (1995)
Raydan M.: On the Barzilai and Borwein choice of steplength for the gradient method. IMA J. Numer. Anal. 13, 321–326 (1993)
Raydan M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7, 26–33 (1997)
Raydan M., Svaiter B.F.: Relaxed steepest descent and Cauchy-Barzilai–Borwein method. Comput. Optim. Appl. 21, 155–167 (2002)
Serafini T., Zanghirati G., Zanni L.: Gradient projection methods for quadratic programs and applications in support vector machines. Optim. Methods Softw. 20, 353–378 (2005)
Toint Ph.L.: Some numerical results using a sparse matrix updating formula in unconstrained optimization. Math. Comput. 32, 839–852 (1978)
Varadhan R., Gilbert P.D.: BB: an R package for solving a large system of nonlinear equations and for optimizing a high-dimensional nonlinear objective function. J. Stat. Softw. 32, 1–26 (2009)
Vargas W.E., Azofeifa D.E., Clark N.: Retrieved optical properties of thin films on absorbing substrates from transmittance measurements by application of a spectral projected gradient method. Thin Solid Films 425, 1–8 (2003)
Yuan Y.: A new stepsize for the steepest descent method. J. Comput. Math. 24, 149–156 (2006)