Unconstrained Optimization

Abstract

Most practical optimization problems arise with constraints on the solutions. Nevertheless, unconstrained optimization techniques serve as a major tool in finding solutions for both unconstrained and constrained optimization problems. In this chapter we present techniques for solving the unconstrained optimization problem. In Chap. 5 we will see how the unconstrained solution methods aid in finding the solutions to constrained optimization problems.

In order to find the solution of a given unconstrained optimization problem, a general procedure includes an initial solution estimate (guess) followed by iterative updates of the solution directed toward minimizing the objective function. If we consider the solution of an N-dimensional objective function as an N-vector, iterative updates can be performed by adding a vector to the current solution vector. We call the vector augmented to the current solution vector the step vector or simply the step.

Various unconstrained optimization methods can be classified by the method of determining step vectors. We place direct search methods, which are the simplest, in group one. In this group, the step vectors are randomly determined. The second group is known as derivative-based methods. In this case, the step vectors are determined based on the derivative of the objective function. Gradient methods and Newton’s methods fall into this category.

In practice, more sophisticated methods, such as conjugate gradient and quasi-Newton methods, have shown comparable performance with greatly reduced computational cost and memory space. A brief discussion of the conjugate gradient algorithm will also be included in this chapter.