Unconstrained optimization

INTRODUCTION

Unconstrained optimization is concerned with finding the minimizing or maximizing points of a nonlinear function, where the variables are free to take on any value. Unconstrained optimization problems occur in a wide range of applications from the fields of engineering and science. A rich source of unconstrained optimization problems are data fitting problems, in which some model function with unknown parameters is fitted to data, using some criterion of “best fit.” This criterion may be the minimum sum of squared errors, or the maximum of a likelihood or entropy function. Unconstrained problems also arise from constrained optimization problems, since these are often solved by solving a sequence of unconstrained problems.

In mathematical terms, an unconstrained minimization problem can be written in the form

where x is a vector of n unrestricted variables. Ideally, one would like to find a global minimizer of the function, that is, a point x ^∗that yields the lowest value...