Quadratic Forms and Optimization

Abstract

We now begin several applications of the theory developed in the last two chapters. In the present chapter we consider first the geometry of the solution sets of quadratic equations in n variables. These solution sets generalize ellipses, parabolas, and hyperbolas in two variables, and a classification of their geometry in n dimensions is given by the inertia of the coefficient matrix of the quadratic form. In the next section we treat the unconstrained quadratic optimization problem and show that a necessary and sufficient condition for a unique solution is that the coefficient matrix be definite. We then consider the special constrained optimization problem of a quadratic function on the unit sphere and show that the maximum and minimum are just the largest and smallest eigenvalues of the coefficient matrix. This leads to the famous min-max representation of the eigenvalues of a Hermitian matrix. In Section 4.3 we specialize the minimization problem to the very important least squares problem and give the basic result on the existence and uniqueness of a solution. Particular examples are the linear regression and polynomial approximation problems. Then we treat the least squares problem in a more general way and obtain a minimum norm solution in the case that the original problem has infinitely many solutions. We show that this minimum norm solution can be represented in terms of a generalized inverse based on the singular value decomposition.