Conjugate Gradient Algorithms

Abstract

A first form of the conjugate gradient routine was given in Section 6, Chapter II. It was redeveloped in Section 3, Chapter III, as a CGS-method. In the present chapter we shall study the CG-algorithm in depth, giving several alternative versions of the CG-routine. It is shown that the number of steps required to obtain the minimum point of a quadratic function F is bounded by the number of distinct eigenvalues of the Hessian A of F. Good estimates of the minimum point are obtained early when the eigenvalues of A are clustered. If the Hessian A of F is nonnegative but not positive definite, a Cg-algorithm will yield the minimum point of F when minimum points of F exist. If A is indefinite and nonsingular, the function F has a unique saddle point. It can be found by a CG-routine except in special circumstances. However, a modified Cg-routine, called a planar Cg-algorithm, yields the critical point of F in all cases. In this modified routine-we obtain critical points of F successively on mutually conjugate lines and 2-planes, whereas, in the standard Cg-algorithm, we restrict ourselves to successively locating critical points of F on mutually conjugate lines. A generalized Cg-algorithm is given in Section 9 involving a generalized gradient HF′ of F. Such algorithms are useful when the Hessian A of F is sparse. They also enable us to minimize F on a prescribed N-plane. It is shown, in Section 11, that, in general, a conjugate direction algorithm is equivalent to a generalized Cg-algorithm in the sense that they generate the same estimates of the minimum point of F.