Error Analysis of Least Squares Algorithms

Abstract

A finite algebraic algorithm starts with a set of data d ₁,...,d _r , from which it computes via fundamental arithmetic operations a solution f _l,...,f _t . In forward error analysis one attempts to bound $\left| {{{\bar f}_j} - {f_j}} \right|$ , where ${\bar f_j}$ denotes the computed element In backward error analysis, pioneered by J.H. Wilkinson in the late fifties, one attempts to determine a modified set of data ${\bar d_i}$ such that the computed solution ${\bar f_j}$ is the exact solution. When it applies it tends to be very markedly superior to forward analysis. To yield error bounds for the solution, the backward error analysis has to be complemented with a perturbation analysis, which naturally leads to the concept of condition number of a problem.

There are several possible definitions of the stability of an algorithm related to different types of error analysis. The concepts of forward and backward stability and of weak and strong stability are discussed.

Many of the common problems in signal processing can be formulated as solutions to (a sequence of) linear least squares problems of the form min _x ∥X w − y∥₂. We review the perturbation theory of such problems and discuss methods for the estimation of the corresponding condition numbers. We survey stability results for the method of normal equations and methods based on orthogonal reductions.

Very often it is required to recursively recalculate the solution x when equations are successively added to and/or deleted from the least squares problem. Many different algorithms have been proposed to effectuate this. Most of these involve updating or downdating the Cholesky factor R of A ^T A. This can be achieved using orthogonal and hyperbolic transformations. The numerical stability of such recursive algorithms is not yet completely analyzed. A new method, using iterative refinement, is suggested as a means of increasing the reliability of downdating algorithms