Level-set methods for convex optimization

Abstract

Convex optimization problems arising in applications often have favorable objective functions and complicated constraints, thereby precluding first-order methods from being immediately applicable. We describe an approach that exchanges the roles of the objective with one of the constraint functions, and instead approximately solves a sequence of parametric level-set problems. Two Newton-like zero-finding procedures for nonsmooth convex functions, based on inexact evaluations and sensitivity information, are introduced. It is shown that they lead to efficient solution schemes for the original problem. We describe the theoretical and practical properties of this approach for a broad range of problems, including low-rank semidefinite optimization, sparse optimization, and gauge optimization.

A Proofs

Proof

(Proof of Theorem 2.1) Since f is convex, the subdifferential $\partial f(\tau )$ is nonempty for all $\tau \in (a,b)$. The claim concerning finite termination is easy to deduce from convexity; we leave the details to the reader. Suppose neither sequence terminates finitely at $\tau _*$. Let us first consider the Newton iteration. Convexity of f immediately implies that the sequence $\tau _i$ is well-defined and satisfies $\tau _0<\tau _1<\tau _2<\dots < \tau _*$. Monotonicity of the subdifferential then implies $g_0\le g_1\le g_2\le \dots \le g_*<0$. Due to the inequalities $f(\tau _*)+\bar{g}(\tau _k-\tau _*)\le f(\tau _k)$ and $g_k<0$, we have

$$\begin{aligned} \frac{f(\tau _k)-f(\tau _*)}{g_k}\le -\frac{g_*}{g_k}(\tau _*-\tau _k), \end{aligned}$$

and so

$$\begin{aligned} 0<\tau _*-\tau _{k+1}= \tau _*-\tau _k+\frac{f(\tau _k)-f(\tau _*)}{g_k} \le \left( 1-\frac{g_*}{g_k}\right) (\tau _*-\tau _k). \end{aligned}$$

Upper semi-continuity of $\partial f$ on its domain implies $g_k\uparrow g_*$. Hence $\tau _k$ converge q-superlinearly to $\tau _*$.

Now consider the secant iteration. As in the Newton iteration, it is immediate from convexity that the sequence $\tau _i$ is well-defined and satisfies $\tau _0<\tau _1<\tau _2<\dots <\tau _*$. Monotonicity of the subdifferential then implies $g_0\le g_1\le g_2\le \dots \le g_*<0$. We have

$$\begin{aligned}0< g_*(\tau _k- \tau _*)\le f(\tau _k)-f(\tau _*),\end{aligned}$$

and $f(\tau _{k-1})+g_{k-1}(\tau _k-\tau _{k-1})\le f(\tau _k)$, and hence

$$\begin{aligned} \frac{\tau _k-\tau _{k-1}}{f(\tau _k)-f(\tau _{k-1})}(f(\tau _k)-f(\tau _*)) \le \frac{f(\tau _k)-f(\tau _*)}{g_{k-1}}<0. \end{aligned}$$

Combining the two inequalities yields

$$\begin{aligned} \frac{f(\tau _k)-f(\tau _*)}{f(\tau _k)-f(\tau _{k-1})}(\tau _k-\tau _{k-1}) \le \frac{f(\tau _k)-f(\tau _*)}{g_{k-1}}\le \frac{g_*}{g_{k-1}}(\tau _k-\tau _*)<0. \end{aligned}$$

Consequently, we deduce

$$\begin{aligned} 0<\tau _* -\tau _{k+1}=\tau _*-\tau _k+ \frac{f(\tau _k)-f(\tau _*)}{f(\tau _k)-f(\tau _{k-1})}(\tau _k-\tau _{k-1}) \le \left( 1-\frac{g_*}{g_{k-1}}\right) (\tau _*-\tau _k). \end{aligned}$$

The result follows. $\square$

Proof

(Proof of Theorem 2.2) It is easy to see by convexity that the iterates $\tau _k$ are strictly increasing and satisfy $f(\tau _k) >0$. For each index $j \ge 2$ for which the algorithm has not terminated, define the following quantities:

$$\begin{aligned} h_{j}:=\tau _j-\tau _{j-1},\quad \quad \theta _j:=\frac{s_{j}}{s_{j-1}},\quad \text { and } \quad \gamma _j:=\frac{\ell _j}{\ell _{j-1}}. \end{aligned}$$

Note that using the equation $\tau _{j-1}-\tau _j = \frac{\ell _{j-1}}{s_{j-1}}$, we can write $\theta _j=\frac{u_{j-1} - \ell _j}{\ell _{j-1}}$. Clearly then the bound, $0\le \theta _j\le \alpha -\gamma _j$, is valid. Define now constants $\beta _j\in [0,1]$ by the equation $\gamma _j = \beta _j \alpha$. Suppose $k \ge 2$ is an index at which the algorithm has not terminated, i.e., $u_k > \epsilon$. Taking into account the inequality $\ell _k \ge \frac{u_k}{\alpha }> \frac{\epsilon }{\alpha }$, we deduce

$$\begin{aligned} \frac{\epsilon }{\alpha } \le \ell _k = \ell _1 \prod _{j=2}^k \gamma _j \le C \alpha ^{k-1} \prod _{j=2}^k \beta _j .\ \end{aligned}$$

(A.1)

The defining equation for $\tau _{k+1}$ and the definition of $\theta _j$ yield the equality

$$\begin{aligned} h_{k+1} = \frac{\ell _k}{|s_k|} = \frac{\ell _k}{|s_1|} \cdot \prod _{j=2}^k \theta _j^{-1}. \end{aligned}$$

The bounds $\tau _* - \tau _1 \ge h_{k+1}$, $\ell _k \ge \frac{\epsilon }{\alpha }$, and $\theta _j \le \alpha - \gamma _j$ imply

$$\begin{aligned} \tau _*-\tau _1 \ge \frac{\ell _k}{|s_1|} \cdot \prod _{j=2}^k \theta _j^{-1} \ge \frac{\epsilon }{\alpha |s_1|} (\alpha ^{-1})^{k-1} \prod _{j=2}^k (1-\beta _j)^{-1}, \end{aligned}$$

and rearranging gives

$$\begin{aligned} \epsilon \le (\tau _*-\tau _1) | s_1 | \alpha ^k \prod _{j=2}^k (1 - \beta _j) \le C \alpha ^k \prod _{j=2}^k (1 - \beta _j). \end{aligned}$$

(A.2)

Combining (A.1) and (A.2), we get

$$\begin{aligned} \epsilon \le C \alpha ^k \min \left\{ \prod _{j=2}^k \beta _j, \ \prod _{j=2}^k (1 - \beta _j) \right\} . \end{aligned}$$

(A.3)

One the other hand, observe

$$\begin{aligned} \left( \prod _{j=2}^k \beta _j\right) \left( \prod _{j=2}^k ( 1- \beta _j)\right) = \prod _{j=2}^k \beta _j( 1- \beta _j) \le 0.5^{2(k-1)}, \end{aligned}$$

and hence

$$\begin{aligned} \min \left\{ \prod _{j=2}^k \beta _j, \ \prod _{j=2}^k (1 - \beta _j) \right\} \le 0.5^{k-1}. \end{aligned}$$

(A.4)

Combining Eqs. (A.4) and (A.3), the claimed estimate $k-1 \le \log _{2/\alpha } \left( \frac{ \alpha C }{ \epsilon } \right)$ follows.

Proof

(Proof of Theorem 2.3) The proof is identical to the proof of Theorem 2.2, except for some minor modifications. The only nontrivial change is how we arrive at the bound $\theta _j \le \alpha - \gamma _j$. For this, observe $\tau _{j-1} - \tau _j = \ell _{j-1}/s_{j-1}$, and because the function $\tau \mapsto \ell _j + s_j ( \tau - \tau _j)$ minorizes f, we see

$$\begin{aligned} u_{j-1}&\ge \ell _j + s_j( \tau _{j-1} - \tau _j ) = \ell _j + s_j \left( \frac{\ell _{j-1}}{s_{j-1}} \right) = \ell _j + \theta _j \ell _{j-1}. \end{aligned}$$

After rearranging, we get the desired upper bound on $\theta _j$:

$$\begin{aligned} \theta _j&\le \frac{u_{j-1} - \ell _j}{\ell _{j-1}} \le \alpha - \gamma _j. \end{aligned}$$

Finally, we remark that with the approximate Newton method, we can start indexing at $j = 0$ instead of $j = 1$. This explains the different constants in the convergence result.

Lemma A.1

(Concavity of the parametric support function) For any convex function $f:\mathbb {R}^n\rightarrow \overline{\mathbb {R}}$ and vector $z\in \mathbb {R}^n$, the univariate function $t\mapsto \delta ^*_{[f\le t]}(z)$ is concave.

Proof

It follows from convexity of f that

$$\begin{aligned} \lambda \cdot [f\le a]+(1-\lambda ) \cdot [f\le b]\subseteq [f\le \lambda a+(1-\lambda ) b] \qquad \forall a,b\in \mathbb {R}\text { and } \lambda \in [0,1], \end{aligned}$$

where $[f\le \alpha ]$ defines the $\alpha$-level set of f, and the summation of the level sets indicates their Minkowski (i.e., direct) sum. Moreover, for any convex sets $\mathcal C$ and $\mathcal D$ such that $\mathcal C\subseteq \mathcal D$, $\delta ^*_{\mathcal C}\le \delta ^*_{\mathcal D}$. Thus,

$$\begin{aligned} \lambda \cdot \delta ^*_{[f\le a]}(z)+(1-\lambda ) \cdot \delta ^*_{[f\le b]}(z)=\delta ^*_{\lambda \cdot [f\le a]+(1-\lambda ) \cdot [f\le b]}(z)\le \delta ^*_{[f\le \lambda a+(1-\lambda ) b]}(z), \end{aligned}$$

which implies concavity of the function at hand..

Proof

(Proof of Proposition 3.1) For this proof only, let $\Vert \cdot \Vert$ denote the 2-norm. Note the inclusion $s/\Vert y\Vert \in \partial _{\tau }\varPhi _1\left( y/\Vert y\Vert ,\,\tau \right)$. Use the same computation from (2.2) to deduce that the affine function

$$\begin{aligned} \tau '\mapsto (\hat{\ell }-\sigma )-\frac{s}{\Vert y\Vert }(\tau '-\tau ) \end{aligned}$$

minorizes $f_1$.

From the definition of $\hat{\ell }$, $\varPhi _1$, and $\varPhi _2$, it follows that

$$\begin{aligned} \begin{aligned} \frac{u-\sigma }{\hat{\ell }-\sigma } = \frac{(u-\sigma )\Vert y\Vert }{\varPhi _2(y,\tau ) + \frac{1}{2}\Vert y\Vert ^2 - \sigma \Vert y\Vert } = \frac{2(u-\sigma )\Vert y\Vert }{\ell ^2 + \Vert y\Vert ^2 - 2\sigma \Vert y\Vert }. \end{aligned} \end{aligned}$$

(A.5)

Taking into account the equivalence

$$\begin{aligned} \frac{u-\sigma }{\ell -\sigma } \le \alpha \quad \iff \quad \frac{u + (\alpha -1)\sigma }{\alpha } \le \ell , \end{aligned}$$

we deduce

$$\begin{aligned} \ell ^2+\Vert y\Vert ^2-2\sigma \Vert y\Vert\ge & {} \alpha ^{-2} \Big ( (u+(\alpha -1)\sigma )^2+\Vert \alpha y\Vert ^2-2\sigma \alpha \Vert \alpha y\Vert \Big )\\\ge & {} 2\alpha ^{-1}(u-\sigma )\Vert y\Vert , \end{aligned}$$

where the rightmost inequality follows from the computation

$$\begin{aligned} \begin{aligned} (u + [\alpha - 1]\sigma )^2&+ \Vert \alpha y\Vert ^2 - 2\alpha \sigma \Vert \alpha y\Vert - 2(u-\sigma )\Vert \alpha y\Vert \\ {}&= \left( u + [\alpha -1]\sigma \right) ^2 + \Vert \alpha y\Vert ^2 - 2\Vert \alpha y\Vert (u + [\alpha -1]\sigma ) \\ {}&= \left( u + [\alpha -1]\sigma - \Vert \alpha y\Vert \right) ^2 \ge 0. \end{aligned} \end{aligned}$$

Because the right-hand side of (A.5) is non-negative, we can deduce that $\hat{\ell }\ge \sigma$. Finally, the required inequality $(u-\sigma )/(\hat{\ell }-\sigma )\le \alpha$ also follows from (A.5).

Lemma A.2

$(-\lambda _{\min })^{\star }(y) = \delta _\mathcal {S}(-y)$, where $\mathcal {S}= {\mathcal {K}}^{*} \cap \left\{ x \mid {{\langle }e},{x{\rangle }} = 1 \right\}$.

Proof

The following formula is established in [49]:

$$\begin{aligned} \partial (-\lambda _{\min }) ( x ) = \left\{ -y \mid {{\langle }y},{e{\rangle }} = 1, \ {{\langle } y},{ z - (x - \lambda _{\min }(x) e ) {\rangle }} \ge 0 \text { for all } z \in \mathcal {K} \right\} \end{aligned}$$

or equivalently

$$\begin{aligned} \partial (-\lambda _{\min }) ( x )&= \left\{ -y \,\left| \ {{\langle }y},{e{\rangle }} = 1, -y\in N_{\mathcal {K}}\left( x - \lambda _{\min }(x) e\right) \right. \right\} \\&= \left\{ -y\,\left| \ {{\langle }y},{e{\rangle }} = 1, \ y \in \mathcal {K}^*, \ 0 = \lambda _{\min }(x) - {{\langle }y},{x{\rangle }} \right. \right\} . \end{aligned}$$

Here the symbol $N_{\mathcal {K}}$ denotes the normal cone to $\mathcal {K}$. Now for any $y\in \partial (-\lambda _{\min }) ( x )$, we have $\langle x,y\rangle =-\lambda _{\min }(x)$. Observe $\mathrm {range}\,\partial (-\lambda _{\min })=-\mathcal {S}$. Hence by the equality in the Fenchel-Young inequality, for any $y\in -\mathcal {S}$, we have $(-\lambda _{\min })^{\star }(y)=0$. On the other hand, for any y with $\langle y,e\rangle \ne -1$, we have $(-\lambda _{\min })^{\star }(y)\ge \langle te,y\rangle -(-\lambda _{\min })(te)=t(\langle y,e\rangle +1)$ for any $t \ge 0$. Letting $t\rightarrow \infty$, we deduce $(-\lambda _{\min })^{\star }(y)=+\infty$. Similarly, consider $y\notin -\mathcal {K}^*$. Then we may find some $x\in \mathcal {K}$ satisfying $\langle x,y\rangle >0$. We deduce $(-\lambda _{\min })^{\star }(y)\ge \langle tx,y\rangle -(-\lambda _{\min })(tx)=t(\langle y,x\rangle -(-\lambda _{\min })(x))$ for any $t \ge 0$. Letting $t\rightarrow \infty$, we deduce $(-\lambda _{\min })^{\star }(y)=+\infty$. We deduce that $(-\lambda _{\min })^{\star }$ is the indicator function of $-\mathcal {S}$, as claimed. $\square$

Remark A.1

(Projection onto a conic slice sets) This remark is standard. Fix a proper convex cone ${\mathcal {K}}$ and consider the projection problem

$$\begin{aligned} \min _{x}\ \left\{ \tfrac{1}{2}\Vert x-z\Vert ^2\,\left| \ \langle c,x\rangle =1,\, x\in \mathcal {K}\right. \right\} . \end{aligned}$$

Equivalently, we can consider the univariate concave maximization problem

$$\begin{aligned} \max _{\beta }\min _{x\in {\mathcal {K}}}\, L(x,\beta )&= \max _{\beta }\min _{x\in {\mathcal {K}}}\, \tfrac{1}{2}\Vert x-z\Vert ^2+\beta (\langle c,x\rangle -1)\\&=\max _{\beta }\min _{x\in {\mathcal {K}}}\, \tfrac{1}{2}\Vert x-(z-\beta c)\Vert ^2+\beta (\langle c,z\rangle -1)-\tfrac{1}{2}\beta ^2\Vert c\Vert ^2\\&=\max _{\beta }~ \tfrac{1}{2}\hbox {dist}^2_{\mathcal {K}}(z-\beta c)+\beta (\langle c,z\rangle -1)-\tfrac{1}{2}\beta ^2\Vert c\Vert ^2. \end{aligned}$$

We can solve this problem for example by bisection, provided projections onto ${\mathcal {K}}$ are available.