Some Convergence Properties of the Steepest Descent Algorithm Revealed by Renormalisation

Abstract

The behaviour of the standard steepest-descent algorithm for a quadratic function in IR ^d is investigated. By rescaling the iterates to remain always on the unit sphere one can reveal special features of this behaviour: the renormalized algorithm converges to a two-point cycle on the unit circle; the cycle depends on the starting point in a complicated manner (the set of points converging to the same cycle is fractal), but all cycles belong to a particular plane, given by certain eigenvectors of the Hessian matrix of the objective function. The stability of the attractor is analysed. The rate of convergence of the algorithm is investigated. It is shown that the worst value of this rate is obtained only for some particular starting points. The introduction of a relaxation coefficient in the steepest-descent algorithm completely changes its behaviour, which may become chaotic. Different attractors are presented. We show that relaxation allows a significantly improved rate of convergence.