Using Exact Additive and Multiplicative Parts in Quasi-Newton Methods

Abstract

We consider approaches in algorithms for optimization, how certain problem inherent structure, that gives rise to a special form of the Hessian, can be exploited to construct suitable approximations to the Hessian. Taking the unconstrained nonlinear least squares problem:

$$\min imize f(x) = \frac{1}{2}R{(x)^T}R(x) = \frac{1}{2}\sum\limits_{i = 1}^m {{r_i}} {(x)^2},$$

where R: ℝⁿ → ℝ^m denotes the residual function, as an example for our considerations, it is apparent that the well-known algorithms for this problem like the Gauss-Newton method, the Levenberg-Marquardt method and Quasi-Newton methods [4, 7] work with different kinds of approximations to the Hessian:

$${\nabla ^2}f(x) = J{(x)^T}J(x) + \sum\limits_{i = 1}^m {{r_i}(x)} {\nabla ^2}{r_i}(x)$$

where J(x) denotes the Jacobian of R at x and r_i(x) the i-th residual function.