Convex Optimization Theory

Abstract

Many machine learning tasks can be formulated as an optimization problem given in the form of

$$\displaystyle \min _{x \in X} f(x), $$

where f, x, and X denote the objective function, decision variables, and feasible set, respectively. Unfortunately, solving an optimization problem is challenging. In general, we cannot guarantee whether one can find an optimal solution, and if so, how much computational effort one needs. However, it turns out that we can provide such guarantees for a special but broad class optimization problems, namely convex optimization, where X is a convex set and f is a convex function. In fact, many machine learning models we formulated so far, such as least square linear regression, logistic regression, and support vector machine, are convex optimization problems.