First-order methods almost always avoid strict saddle points

Abstract

We establish that first-order methods avoid strict saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including (manifold) gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid strict saddle points.

Appendices

Proof of Claim 4

Proof

We assume that $\alpha |e_j^{T}Hz_j| < \delta \left\| z_j \right\| _2$ for all $j \in \{1,\ldots ,n\}$, for some $\delta$ to be chosen later. For the base case $j=2$, it holds that $\left\| y_t-z_2 \right\| _2 = \left\| z_1-z_2 \right\| _2 = \alpha |e_1^{T}Hz_1|< \delta \left\| z_1 \right\| _2 < 2\delta \left\| y_t \right\| _2$ and $\left\| z_2 \right\| _2 < (1+2\delta )\left\| y_t \right\| _2$. Suppose for $j \ge 2$ that $\left\| y_t-z_{j} \right\| _2 < 2(j-1)\delta \left\| y_t \right\| _2$ and thus $\left\| z_j \right\| _2 < [1+2(j-1)\delta ] \left\| y_t \right\| _2.$ Using induction and triangle inequality we get

$$\begin{aligned} \left\| y_t - z_{j+1} \right\| _2&\le \left\| y_t - z_{j} \right\| _2+ \left\| z_{j} -z_{j+1} \right\| _2\\&= 2(j-1)\delta \left\| y_t \right\| _2 + \alpha |e_j^{T}Hz_j| \\&< 2(j-1)\delta \left\| y_t \right\| _2 + \delta \left\| z_j \right\| _2 \\&< 2(j-1)\delta \left\| y_t \right\| _2+ \delta [1+2(j-1)\delta ] \left\| y_t \right\| _2\\&\le 2j\delta \left\| y_t \right\| _2, \end{aligned}$$

where we assume $\delta < \frac{1}{2n}$ so that $2(j-1)\delta <1$ for all $j \in [n]$. Using the above calculation,

$$\begin{aligned} \alpha |e_i^{T}Hy_t|&< \alpha |e_i^{T}Hz_i|+\alpha |e_i^{T}H(y_t - z_i)| \\&< \delta \left\| z_i \right\| _2+ \alpha \left\| He_i \right\| _2\left\| y_t - z_i \right\| _2\\&< \delta \big (1 +2(i-1)\delta \big ) \left\| y_t \right\| _2 + \alpha \left\| He_i \right\| _2 \big (2(i-1)\delta \big ) \left\| y_t \right\| _2\\&\le \delta \big ( 1+2n \delta + 2n \alpha L\big )\left\| y_t \right\| . \end{aligned}$$

Thus $\alpha \left\| Hy_t \right\| _2 < \sqrt{n}\delta \big ( 1+2n \delta + 2n \alpha L\big ) \left\| y_t \right\| _2$, and

$$\begin{aligned} \sigma _{\min ^+} (H) \left\| y_t \right\| _2 \le \left\| Hy_t \right\| _2 < \frac{\sqrt{n}}{\alpha }\delta \big ( 1+2n \delta + 2n \alpha L\big ) \left\| y_t \right\| _2, \end{aligned}$$

where $\sigma _{\min ^+}$ is the smallest non-zero singular value of H. Thus by choosing $\delta$ small enough such that

$$\begin{aligned} \sigma _{min^+} (H) \ge \frac{\sqrt{n}}{\alpha }\delta \big ( 1+2n\delta + 2n\alpha L\big ), \end{aligned}$$

we have obtained a contradiction. $\square$

Proof of Proposition 7

Proof

Let $H= \nabla ^2 f(x^*)$, $J =\mathrm {D}g(x^*) = \prod _{i=1}^b (I- \alpha P_{S_{b-i+1}} H)$, and $y_0$ be an eigenvector of the Hessian at $x^*$.

The proof technique is very similar to that of the proof of Proposition 5. We shall prove that $\left\| J^t y_0 \right\| _2 \ge c(1+\eta )^t$. Hence by Gelfand’s theorem J must have at least one eigenvalue with magnitude greater than one.

We fix some arbitrary iteration t and let $y_t = J^t y_0$. We will first show that there exists an $\epsilon >0$,

$$\begin{aligned} y_{t+1}^{T} Hy_{t+1} \le (1+\epsilon ) y_{t}^{T}Hy_{t}, \end{aligned}$$

(13)

for all $t\in {\mathbb {N}}$. Let $z_1 = y_t$ and $z_{i+1} = (I- \alpha P_{S_i} H)z_i = z_i - \alpha \sum _{j \in S_i}(e_j^{T}Hz_i)e_j$, so that $y_{t+1} = Jy_t = z_{b+1}$. We get that

$$\begin{aligned}&z_{i+1}^{T}Hz_{i+1} = \left( z_i^{T} - 2 \alpha \sum _{j \in S_i}(e_j^{T}Hz_i)e_j^{T}\right) H \left( z_i - \alpha \sum _{j \in S_i}(e_j^{T}Hz_i)e_j\right) \\&= z_i^{T}Hz_i -2\alpha \sum _{j \in S_i}(e_j^{T} H z_i)^2 + \alpha ^2 \left( \sum _{j \in S_i}(e_j^{T}Hz_i)e_j\right) ^{T} H \left( \sum _{j \in S_i}(e_j^{T}Hz_i)e_j \right) \\&< z_i^{T}Hz_i - 2\alpha \sum _{j \in S_i}(e_j ^{T} H z_i)^2 + \alpha ^2 L_{b} \left\| \sum _{j \in S_i}(e_j^{T}Hz_i)e_j \right\| _2^2 \quad { \text { (using } \left\| H_{S_i} \right\| _2 \le L_b)}\\&= z_i^{T}Hz_i - \alpha (2 -\alpha L_b )\left\| \sum _{j \in S_i}(e_j^{T}Hz_i)e_j \right\| _2^2 \\&\le z_i^{T}Hz_i - \alpha \left\| \sum _{j \in S_i}(e_j^{T}Hz_i)e_j \right\| _2^2 \quad { \text {(using } \alpha L_b <1)}. \end{aligned}$$

Thus $z_{i}^T H z_i$ is a decreasing (non-increasing) sequence.

We shall prove that there exists an $i \in [b]$ so that $z_{i+1}^{T}Hz_{i+1} \le (1+\delta ) z_{i}^{T}Hz_{i}$ for some global constant $\delta$ to be chosen later.

Claim

Let $y_t$ be in the range of H. There exists an $i \in [b]$ so that $\alpha \sum _{j \in S_i} \left| e_j^{T}Hz_i\right| \ge \delta \left\| z_i \right\| _2$ for some $\delta >0$.

To finish the proof of the lemma, suppose that Claim B applies. Then by Cauchy–Schwarz, there exists an index i such that

$$\begin{aligned} z_{i+1}^{T}Hz_{i+1}< & {} z_i^{T}Hz_i - \alpha \sum _{j \in S_i}(z_i^{T}He_j)^2 \\< & {} z_i^{T}Hz_i - \frac{\alpha }{n} \left( \sum _{j \in S_i} \left| z_i^{T}He_j\right| \right) ^2 < z_i ^{T}Hz_i - \frac{\delta ^2}{n\alpha } \left\| z_i \right\| ^2_2. \end{aligned}$$

However, $w^{T}Hw \ge \lambda _{\min }(H)\left\| w \right\| ^2_2 \ge - L \left\| w \right\| ^2_2$, hence we get that

$$\begin{aligned} z_{i+1}^{T}Hz_{i+1} < \left( 1+ \frac{\delta ^2}{\alpha L n}\right) z_i ^{T}Hz_i . \end{aligned}$$

(14)

By choosing $\epsilon =\frac{\delta ^2}{\alpha L n}$ we showed that $y_{t+1}^{T}Hy_{t+1} \le (1+\epsilon )y_t^{T}Hy_t$ as long as $y_t$ is in the range of H.

Assume that $y_t = y_{{\mathcal {N}}} + y_{{\mathcal {R}}}$. It is easy to see $y_t^{T}Hy_t = y_{{\mathcal {R}}}^{T}Hy_{{\mathcal {R}}}$ and also $y_{t+1} = Jy_t = y_{{\mathcal {N}}} + Jy_{{\mathcal {R}}}$, hence $y_{t+1}^{T}Hy_{t+1} =(Jy_{{\mathcal {R}}})^{T} H (Jy_{{\mathcal {R}}})$. Therefore from Inequality (14) proved above, if the starting vector is $y_{{\mathcal {R}}}$, which Claim B applies too, then $(Jy_{{\mathcal {R}}})^{T}HJy_{{\mathcal {R}}} \le (1+\epsilon )y_{{\mathcal {R}}}^{T}H y_{{\mathcal {R}}} = (1+\epsilon )y_t^{T}Hy_t$.

To sum up, we showed that $y_t^{T}Hy_t \le (1+\epsilon )^t y_0^{T} H y_0$ and since $y_0$ is an eigenvector of H (of norm one) with corresponding negative eigenvalue $\lambda$, it follows that $y_t^{T}Hy_t \le \lambda (1+\epsilon )^t.$ Finally using $y_t^{T}Hy_t \ge \lambda _{\min }(H)\left\| y_t \right\| ^2_2$, we get $\left\| y_t \right\| _2 \ge (1+\epsilon )^{t/2} \frac{\lambda }{\lambda _{\min }(H)}$. Observe that $\frac{\lambda }{\lambda _{\min }(H)}>0$ is a positive constant, $(1+\epsilon )^{t/2} \ge (1+\epsilon /4)^t$ (since $\epsilon \le 1/2$) and the proof follows (the parameters as claimed in the beginning will be $c = \frac{\lambda }{\lambda _{\min }(H)}$ and $\eta = \epsilon /4$). $\square$

Proof of Claim B

We assume that $\alpha \sum _{j \in S_i}\left| e_j^{T}Hz_i \right| < \delta \left\| z_i \right\| _2$ for all $i \in [b]$. For base case $i=2$, it holds that $\left\| y_t-z_2 \right\| _2 = \left\| z_1-z_2 \right\| _2 = \alpha |\sum _{j \in S_1}e_j^{T}Hz_1|< \alpha \sum _{j \in S_1} |e_j^{T}Hz_1|< \delta \left\| z_1 \right\| _2 < 2\delta \left\| y_t \right\| _2$ and $\left\| z_2 \right\| _2 < (1+2\delta )\left\| y_t \right\| _2$. Suppose for $i \ge 2$ that $\left\| y_t-z_{i} \right\| _2 < 2(i-1)\delta \left\| y_t \right\| _2$ and thus $\left\| z_i \right\| _2 < [1+2(i-1)\delta ] \left\| y_t \right\| _2.$ Using induction and triangle inequality we obtain

$$\begin{aligned} \left\| y_t - z_{i+1} \right\| _2&\le \left\| y_t - z_{i} \right\| _2+ \left\| z_{i} -z_{i+1} \right\| _2\\&= 2(i-1)\delta \left\| y_t \right\| _2 + \alpha \left| \sum _{j \in S_i}e_j^{T}Hz_i\right| \\&\le 2(i-1)\delta \left\| y_t \right\| _2 + \alpha \sum _{j \in S_i}\left| e_j^{T}Hz_i\right| \\&< 2(i-1)\delta \left\| y_t \right\| _2 + \delta \left\| z_i \right\| _2 \\&< 2(i-1)\delta \left\| y_t \right\| _2+ \delta [1+2(i-1)\delta ] \left\| y_t \right\| _2 \\&\le 2i\delta \left\| y_t \right\| _2, \end{aligned}$$

where we assume $\delta < \frac{1}{2b}$ so that $2(i-1)\delta <1$ for all $i \in [b]$. Using the above,

$$\begin{aligned} \alpha \sum _{j \in S_i} \left| e_j^{T}Hy_t\right|&< \alpha \sum _{j \in S_i}\left| e_j^{T}Hz_i \right| +\alpha \sum _{j \in S_i}\left| e_j^{T}H(y_t - z_i)\right| \\&< \delta \left\| z_i \right\| _2+ \alpha \left( \sum _{j \in S_i}\left\| He_j \right\| _2 \right) \left\| y_t - z_i \right\| _2 \\&< \delta [1+2(i-1)\delta ]\left\| y_t \right\| _2 + \alpha [ 2(i-1)\delta ] \left\| y_t \right\| _2 \left( \sum _{j \in S_i}\left\| He_j \right\| _2 \right) . \end{aligned}$$

Since $\left\| He_i \right\| _2 < \sigma _{\max }(H) \le L$, we get that $\alpha \sum _{j \in S_i} \left\| He_j \right\| _2 < |S_i| \le n$ and we conclude

$$\begin{aligned} \alpha \sum _{j \in S_i}\left| e_j^{T}Hy_t\right| < 2n^2 \delta \left\| y_t \right\| _2. \end{aligned}$$

(15)

Finally, using Inequality (15) it follows that $\alpha \left\| Hy_t \right\| _2 < 2n^2 \delta \sqrt{n} \left\| y_t \right\| _2$. Let $w \in \text {Im}(H)$ be a vector that is orthogonal to $\text {null}(H)$ (since H is symmetric). Then it holds that $\left\| Hw \right\| _2 \ge \sigma _{\min ^+}(H) \left\| w \right\| _2$ where $\sigma _{\min ^+}(H)$ denotes the smallest positive singular value of H (greater than zero). Assume that $y_t \in \text {Im}(H)$ and we get $\left\| Hy_t \right\| _2 < \frac{2n^2 \delta \sqrt{n}}{\alpha } \left\| y_t \right\| _2$. However, $\left\| Hy_t \right\| _2 \ge \sigma _{\min ^+}(H) \left\| y_t \right\| _2$ thus by choosing $\frac{2n^2 \sqrt{n}\delta }{\alpha } < \sigma _{\min ^+}(H)$ we reach a contradiction. $\square$