Gradient Sampling Methods for Nonsmooth Optimization

Abstract

This article reviews the gradient sampling methodology for solving nonsmooth, nonconvex optimization problems. We state an intuitively straightforward gradient sampling algorithm and summarize its convergence properties. Throughout this discussion, we emphasize the simplicity of gradient sampling as an extension of the steepest descent method for minimizing smooth objectives. We provide an overview of various enhancements that have been proposed to improve practical performance, as well as an overview of several extensions that have been proposed in the literature, such as to solve constrained problems. We also clarify certain technical aspects of the analysis of gradient sampling algorithms, most notably related to the assumptions one needs to make about the set of points at which the objective is continuously differentiable. Finally, we discuss possible future research directions.

Dedicated to Krzysztof Kiwiel, in recognition of his fundamental work on algorithms for nonsmooth optimization

Appendices

Appendix 1

This appendix is devoted to justifying the requirement that D, the set of points on which the locally Lipschitz function f is continuously differentiable, must be an open full-measure subset of $\mathbb {R}^n$, instead of the original assumption in [8] that D should be an open and dense set in $\mathbb {R}^n$.

There are two ways in which the analyses in [8, 28] actually depend on D having full measure:

• 1. The most obvious is that both papers require that the points sampled in each iteration should lie in D, and a statement is made in both papers that this occurs with probability one, but this is not the case if D is assumed only to be an open dense subset of $\mathbb {R}^n$. However, as already noted earlier and justified in Appendix 2, this requirement can be relaxed, as in Algorithm GS given in Sect. 6.2, to require only that f be differentiable at the sampled points.

• 2. The set D must have full measure for Property 6.1, stated below, to hold. The proofs in [8, 28] depend critically on this property, which follows from [6, Eq. (1.2)] (where it was stated without proof). For completeness we give a proof here, followed by an example that demonstrates the necessity of the full measure assumption.

Property 6.1

Assume that D has full measure and let

$$\displaystyle \begin{aligned} G_\epsilon({\boldsymbol x}):= \operatorname{\mathrm{cl}}\operatorname{\mathrm{conv}}\nabla f\left( \bar{B}({\boldsymbol x};\epsilon)\cap D \right). \end{aligned}$$

For all 𝜖 > 0 and all ${\boldsymbol x}\in \mathbb {R}^n$, one has ∂f(x) ⊆ G _𝜖(x), where ∂f is the Clarke subdifferential set presented in Definition 1.8.

Proof of Property 6.1

Let ${\boldsymbol x}\in \mathbb {R}^n$ and v ∈ ∂f(x). We have from [10, Theorem 2.5.1] that Theorem 1.2 can be stated in a more general manner. Indeed, for any set S with zero measure, and considering Ω _f to be the set of points at which f fails to be differentiable, the following holds:

$$\displaystyle \begin{aligned} {\partial} f({\boldsymbol x}) = \operatorname{\mathrm{conv}}\left\{ \lim_j \nabla f({\boldsymbol y}^j) : {\boldsymbol y}^j\to {\boldsymbol x}\ \text{where}\ {\boldsymbol y}^j\notin S \cup\varOmega_f\ \text{for all}\ j \in \mathbb{N}\right\}. \end{aligned}$$

In particular, since D has full measure and f is differentiable on D, it follows that

$$\displaystyle \begin{aligned} {\partial} f({\boldsymbol x}) = \operatorname{\mathrm{conv}}\left\{ \lim_j \nabla f({\boldsymbol y}^j) : {\boldsymbol y}^j\to {\boldsymbol x}\ \text{with}\ {\boldsymbol y}^j\in D\ \text{for all}\ j \in \mathbb{N}\right\}\text. \end{aligned}$$

Considering this last relation and Carathéodory’s theorem, it follows that , where, for all , one has $\boldsymbol {\xi }^i = \lim \limits _j \nabla f({\boldsymbol y}^{j,i})$ for some sequence $\{{\boldsymbol y}^{j,i}\}_{j\in \mathbb {N}} \subset D$ converging to x. Hence, there must exist a sufficiently large $j_i \in \mathbb {N}$ such that, for all j ≥ j _i, one obtains

$$\displaystyle \begin{aligned} {\boldsymbol y}^{j,i}\in \bar{B}({\boldsymbol x};\epsilon)\cap D \implies \nabla f({\boldsymbol y}^{j,i})\in \nabla f\left(\bar{B}({\boldsymbol x}; \epsilon)\cap D\right) \subseteq \operatorname{\mathrm{conv}}\nabla f\left(\bar{B}({\boldsymbol x}; \epsilon)\cap D\right)\text. \end{aligned}$$

Recalling that G _𝜖(x) is the closure of $\operatorname {\mathrm {conv}}\nabla f\left (\bar {B}({\boldsymbol x}; \epsilon )\cap D\right )$, it follows that ξ ⁱ ∈ G _𝜖(x) for all . Moreover, since G _𝜖(x) is convex, we have v ∈ G _𝜖(x). The result follows since ${\boldsymbol x} \in \mathbb {R}^n$ and v ∈ ∂f(x) were arbitrarily chosen. $\square$

With the assumption that D has full measure, Property 6.1 holds and hence the proofs of the results in [8, 28] are all valid. In particular, the proof of (ii) in [28, Lemma 3.2], which borrows from [8, Lemma 3.2], depends on Property 6.1. See also [8, the top of p. 762].

The following example shows that Property 6.1 might not hold if D is assumed only to be an open dense set, not necessarily of full measure.

Example 6.2

Let δ ∈ (0, 1) and $\{q_k\}_{k\in \mathbb {N}}$ be the enumeration of the rational numbers in (0, 1). Define

$$\displaystyle \begin{aligned} D:= \bigcup_{k=1}^\infty \mathcal{Q}_k\text{, where }\mathcal{Q}_k:=\left(q_k - \frac{\delta}{2^{k+1}} , q_k + \frac{\delta}{2^{k+1}}\right)\text. \end{aligned}$$

Clearly, its Lebesgue measure satisfies 0 < λ(D) ≤ δ. Moreover, the set D is an open dense subset of [0, 1]. Now, let $i_{D}:[0,1]\to \mathbb {R}$ be the indicator function of the set D,

$$\displaystyle \begin{aligned} i_{D}(x) = \left\{ \begin{array}{cl} 1\text, & \text{if}\ x\in D, \\ 0\text, & \text{if}\ x\notin D\text. \end{array} \right. \end{aligned}$$

Then, considering the Lebesgue integral, we define the function $f:[0,1]\to \mathbb {R}$,

$$\displaystyle \begin{aligned} f(x) = \int_{[0,x]} i_{D}d\lambda\text. \end{aligned}$$

Let us prove that f is a Lipschitz continuous function on (0, 1). To see this, note that given any a, b ∈ (0, 1) with b > a, it follows that

$$\displaystyle \begin{aligned} |f(b) - f(a)| = \left| \int_{[0,b]} i_{D}d\lambda - \int_{[0,a]} i_{D}d\lambda \right| = \left| \int_{(a,b]} i_{D}d\lambda \right| \leq \int_{(a,b]} 1d\lambda = b-a\text, \end{aligned}$$

which ensures that f is a Lipschitz continuous function on (0, 1). Consequently, the Clarke subdifferential set of f at any point in (0, 1) is well defined. Moreover, we claim that, for all $k\in \mathbb {N}$, f is continuously differentiable at any point $q\in \mathcal {Q}_k$ and the following holds

$$\displaystyle \begin{aligned} f'(q) = i_{D}(q) = 1\text. \end{aligned} $$

(6.7)

Indeed, given any $q\in \mathcal {Q}_k$, we have

$$\displaystyle \begin{aligned} f(q+t) - f(q) = \int_{[0,q+t]}i_{D}d\lambda - \int_{[0,q]}i_{D}d\lambda = \int_{(q,q+t]}i_{D}d\lambda\text{, for }t>0\text. \end{aligned}$$

Since $\mathcal {Q}_k$ is an open set, we can find $\overline t>0$ such that $[q,q+t]\subset \mathcal {Q}_k\subset D$, for all $t\leq \overline t$. Hence, given any $t\in (0,\overline {t}]$, it follows that

$$\displaystyle \begin{aligned} f(q+t) - f(q) = \int_{(q,q+t]} 1d\lambda = t\ \Longrightarrow \lim_{t\downarrow 0}\frac{f(q+t) - f(q)}{t} = 1 = i_{D}(q)\text. \end{aligned}$$

The same reasoning can be used to see that the left derivative of f at q exists and it is equal to i _D(q). Consequently, we have f′(q) = i _D(q) = 1 for all $q\in \mathcal {Q}_k$, which yields that f is continuously differentiable on D.

By the Lebesgue differentiation theorem, we know that f′(x) = i _D(x) almost everywhere. Since the set [0, 1] ∖ D does not have measure zero, this means that there must exist z ∈ [0, 1] ∖ D such that f′(z) = i _D(z) = 0. Defining $\epsilon := \min \{z,1-z\}/2$, we see, by (6.7), that the set

$$\displaystyle \begin{aligned} G_\epsilon(z):= \operatorname{\mathrm{cl}}\operatorname{\mathrm{conv}}\nabla f([z-\epsilon,z+\epsilon]\cap D) \end{aligned}$$

is a singleton G _𝜖(z) = {1}. However, since f′(z) = 0, it follows that 0 ∈ ∂f(z), which implies ∂f(z)⊄G _𝜖(z).

Note that it is stated on [8, p. 754] and [28, p. 381] that the following holds: for all 0 ≤ 𝜖 ₁ < 𝜖 ₂ and all ${\boldsymbol x}\in \mathbb {R}^n$, one has $\bar \partial _{\epsilon _1} f({\boldsymbol x}) \subseteq G_{\epsilon _2}({\boldsymbol x})$. Property 6.1 is a special case of this statement with 𝜖 ₁ = 0, and hence this statement too holds only under the full measure assumption.

Finally, it is worth mentioning that in practice, the full measure assumption on D usually holds. In particular, whenever a real-valued function is semi-algebraic (or, more generally, “tame”)—in other words, for all practical purposes virtually always—it is continuously differentiable on an open set of full measure. Hence, the original proofs hold in such contexts.

Appendix 2

In this appendix, we summarize why it is not necessary that the iterates and sampled points of the algorithm lie in the set D in which f is continuously differentiable, and that rather it is sufficient to ensure that f is differentiable at these points, as in Algorithm GS. We do this by outlining how to modify the proofs in [28] to extend to this case.

• 1. That the gradients at the sampled points {x ^{k, j}} exist follows with probability one from Rademacher’s theorem, while the existence of the gradients at the iterates {x ^k} is ensured by the statement of Algorithm GS. Notice that the proof of part (ii) of [28, Theorem 3.3] still holds in our setting with the statement that the components of the sampled points are “sampled independently and uniformly from $\bar {B}({\boldsymbol x}^k;\epsilon )\cap D$” replaced with “sampled independently and uniformly from $\bar {B}({\boldsymbol x}^k;\epsilon )$”.

• 2. One needs to verify that f being differentiable at x ^k is enough to ensure that the line search procedure presented in (6.3) terminates finitely. This is straightforward. Since ∇f(x ^k) exists, it follows that the directional derivative along any vector ${\boldsymbol d}\in \mathbb {R}^n\setminus \{0\}$ is given by f′(x ^k;d) = ∇f(x ^k)^Td. Furthermore, since −∇f(x ^k)^Tg ^k ≤−∥g ^k∥² (see [8, p. 756]), it follows, for any β ∈ (0, 1), that there exists $\overline t>0$ such that

  $$\displaystyle \begin{aligned} f({\boldsymbol x}^k-t{\boldsymbol g}^k) < f({\boldsymbol x}^k) - t\beta\|{\boldsymbol g}^k\|{}^2\ \ \text{for any}\ \ t \in (0,\overline t). \end{aligned}$$

  This shows that the line search is well defined.

• 3. The only place where we actually need to modify the proof in [28] concerns item (ii) in Lemma 3.2, where it is stated that $\nabla f({\boldsymbol x}^k) \in G_\epsilon (\bar {\boldsymbol x})$ (for a particular point $\bar {\boldsymbol x}$) because ${\boldsymbol x}^k \in \bar {B}(\bar {\boldsymbol x};\epsilon /3) \cap D$; the latter is not true if x ^k∉D. However, using Property 6.1, we have

  $$\displaystyle \begin{aligned} \nabla f({\boldsymbol x}^k)\in {\partial} f({\boldsymbol x}^k)\subset G_{\epsilon/3}({\boldsymbol x}^k) \subset G_{\epsilon}(\bar {\boldsymbol x}) \text{ when } {\boldsymbol x}^k\in \bar{B}(\bar {\boldsymbol x};\epsilon/3), \end{aligned}$$

  and therefore, $\nabla f({\boldsymbol x}^k) \in G_\epsilon (\bar {\boldsymbol x})$ even when x ^k∉D.

Finally, although it was convenient in Appendix 1 to state Property 1 in terms of D, it actually holds if D is replaced by any full measure set on which f is differentiable. Nonetheless, it is important to note that the proofs of the results in [8, 28] do require that f be continuously differentiable on D. This assumption is used in the proof of (i) in [28, Lemma 3.2].

Acknowledgements

The authors would like to acknowledge the following financial support. J.V. Burke was supported in part by the U.S. National Science Foundation grant DMS-1514559. F.E. Curtis was supported in part by the U.S. Department of Energy grant DE-SC0010615. A.S. Lewis was supported in part by the U.S. National Science Foundation grant DMS-1613996. M.L. Overton was supported in part by the U.S. National Science Foundation grant DMS-1620083. L.E.A. Simões was supported in part by the São Paulo Research Foundation (FAPESP), Brazil, under grants 2016/22989-2 and 2017/07265-0.