A framework for parallel second order incremental optimization algorithms for solving partially separable problems

Abstract

We propose Hessian Approximated Multiple Subsets Iteration (HAMSI), which is a generic second order incremental algorithm for solving large-scale partially separable convex and nonconvex optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to speed-up the convergence. HAMSI is inherently parallel and it scales nicely with the number of processors. We prove the convergence properties of our algorithm when the subset selection step is deterministic. Combined with techniques for effectively utilizing modern parallel computer architectures, we illustrate that a particular implementation of the proposed method based on L-BFGS updates converges more rapidly than a parallel gradient descent when both methods are used to solve large-scale matrix factorization problems. This performance gain comes only at the expense of using memory that scales linearly with the total size of the optimization variables. We conclude that HAMSI may be considered as a viable alternative in many large scale problems, where first order methods based on variants of gradient descent are applicable.

Appendix A: Omitted proofs

1.1 Proof of Lemma 1

By using Assumption A.2, we have

$$\begin{aligned} \begin{array}{rl} \left\| \nabla f_{S_{[l]}}(x^{(t,l-1)}) - \nabla f_{S_{[l]}}(x^{(t)}) \right\| &{} \le L\Vert x^{(t,l-1)} - x^{(t)} \Vert \\ &{} = L\Vert x^{(t,l-1)} - x^{(t, l-2)} + x^{(t, l-2)} - x^{(t, l-3)} \\ &{}\quad + \dots + x^{(t,1)} - x^{(t)}\Vert \\ &{} \le L \sum \limits _{j=1}^{\ell -1}\Vert x^{(t,j)} - x^{(t, j-1)}\Vert . \end{array} \end{aligned}$$

Note for $j=1, \dots , \ell -1$ that

$$\begin{aligned} \left\| x^{(t,j)} - x^{(t, j-1)}\right\| = \left\| x^{(t,j-1)} - (H^{(t)} + \beta _tI)^{-1}\nabla f_{S_{[j]}}(x^{(t,j-1)}) - x^{(t, j-1)}\right\| \le M_t C, \end{aligned}$$

where the last inequality holds by Assumption A.4. Therefore, we have

$$\begin{aligned} \left\| \nabla f_{S_{[l]}}(x^{(t,l-1)}) - \nabla f_{S_{[l]}}(x^{(t)}) \right\| \le LM_tC(\ell - 1). \end{aligned}$$

1.2 Proof of Lemma 2

At iteration $t + 1$, we have

$$\begin{aligned} \begin{array}{rl} x^{(t+1)} &{} = x^{(t)} - \sum \limits _{\ell =1}^{K+1} \left( H^{(t)} + \beta _tI\right) ^{-1}\nabla f_{S_{[\ell ]}}(x^{(t, \ell -1)}) \\ &{} = x^{(t)} - (H^{(t)} + \beta _tI)^{-1}\nabla f(x^{(t)}) \\ &{} \quad +\, (H^{(t)} + \beta _tI)^{-1}\sum \limits _{\ell =1}^{K+1} \left( \nabla f_{S_{[\ell ]}}(x^{(t)}) - \nabla f_{S_{[\ell ]}}(x^{(t,\ell -1)})\right) . \end{array} \end{aligned}$$

This shows that

$$\begin{aligned} x^{(t+1)} - x^{(t)} = \Delta _t - (H^{(t)} + \beta _tI)^{-1}\nabla f(x^{(t)}), \end{aligned}$$

(11)

where

$$\begin{aligned} \Delta _t\equiv (H^{(t)} + \beta _tI)^{-1}\sum _{\ell =1}^{K+1} \left( \nabla f_{S_{[\ell ]}}(x^{(t)}) - \nabla f_{S_{[\ell ]}}(x^{(t,\ell -1)})\right) . \end{aligned}$$

Using now (9) implies

$$\begin{aligned} \begin{array}{rl} \Vert \Delta _t\Vert &{} \le M_t\sum \limits _{\ell =1}^{K+1} \left\| \nabla f_{S_{[\ell ]}}(x^{(t)}) - \nabla f_{S_{[\ell ]}}(x^{(t,\ell -1)})\right\| \\ &{} \le M_t\sum \limits _{\ell =1}^{K+1}LM_tC(\ell -1) = \frac{1}{2}LM^2_tC K(K+1) = BM_t^2. \\ \end{array} \end{aligned}$$

(12)

Then, we obtain

$$\begin{aligned} \begin{array}{rl} \Vert x^{(t+1)} - x^{(t)}\Vert &{} = \Vert \Delta _t - (H^{(t)} + \beta _tI)^{-1}\nabla f(x^{(t)})\Vert \\ &{} \le \Vert \Delta _t\Vert + \Vert (H^{(t)} + \beta _tI)^{-1}\Vert \nabla f(x^{(t)})\Vert \\ &{} \le BM_t^2 + CM_t \le \frac{B + C(M+1)}{M+1}M_t. \end{array} \end{aligned}$$

1.3 Proof of Theorem 3

As f is a twice differentiable function, we have

$$\begin{aligned} f(x^{(t+1)}) - f(x^{(t)}) \le {\nabla f(x^{(t)})}^{T} (x^{(t+1)} - x^{(t)}) + \frac{LK}{2} \Vert x^{(t+1)} - x^{(t)}\Vert ^2. \end{aligned}$$

Using now Lemma 2 along with (11) and (12), we obtain

$$\begin{aligned} \begin{array}{rl} f(x^{(t+1)}) - f(x^{(t)}) &{} \le \nabla f(x^{(t)})^T \Delta _t - \nabla f(x^{(t)})^T (H^{(t)} \\ &{}\quad + \beta _tI)^{-1}\nabla f(x^{(t)}) + \frac{LK}{2}\Vert x^{(t+1)} - x^{(t)}\Vert ^2 \\ &{} \le \Vert \nabla f(x^{(t)})\Vert \Vert |\Delta _t\Vert - U_t\Vert \nabla f(x^{(t)}) \Vert ^2 \\ &{}\quad + \frac{LK}{2}\left( \frac{B + C(M+1)}{M+1}\right) ^2M^2_t \\ &{} \le - U_t\Vert \nabla f(x^{(t)}) \Vert ^2 + \bar{B}M^2_t, \end{array} \end{aligned}$$

(13)

where $\bar{B} \equiv CB + \frac{LK}{2}\left( \frac{B + C(M+1)}{M+1}\right) ^2$. Due to Assumption A.1, we can write $\inf _{x \in \mathbb {R}^n} f(x) = f^* > -\infty$. Thus, we obtain

$$\begin{aligned} 0 \le f(x^{(t+1)}) - f^* \le f(x^{(t)}) - f^* + \bar{B}M^2_t. \end{aligned}$$

Relation (10) and Lemma 2.2 in Mangasarian and Solodov [21] together show that the sequence $\{f(x^{(t)})\}$ converges. By using (13), we further have

$$\begin{aligned} \begin{array}{rl} f(x^{(1)}) - f^* &{} \ge f(x^{(1)}) - f(x^{(t)}) = \sum \limits _{j=1}^{t-1}\left( f(x^{(t)}) - f(x^{(t+1)})\right) \\ &{} \ge \sum \limits _{j=1}^{t-1}U_j \Vert \nabla f(x^{(j)}) \Vert ^2 - \bar{B}\sum \limits _{j=1}^{t-1}M^2_j \\ &{} \ge \underset{1 \le j \le t-1}{\inf } \Vert \nabla f(x^{(j)}) \Vert ^2 \sum \limits _{j=1}^{t-1}U_j - \bar{B}\sum \limits _{j=1}^{t-1}M^2_j. \end{array} \end{aligned}$$

Let now $t \rightarrow \infty$, then

$$\begin{aligned} f(x^{(1)}) - f^* \ge \inf _{j \ge 1}\Vert \nabla f(x^{(j)}) \Vert ^2 \sum _{j=1}^{\infty }U_j - \bar{B}\sum _{j=1}^{\infty }M^2_j. \end{aligned}$$

(14)

Using again Assumption A.1 and conditon (10), we obtain

$$\begin{aligned} \inf _{t \ge 1}\Vert \nabla f(x^{(t)}) \Vert = 0. \end{aligned}$$

(15)

Now, suppose for contradiction that the sequence $\{\nabla f(x^{(t)})\}$ does not converge to zero. Then, there exists an increasing sequence of integers such that for some $\varepsilon > 0$, we have $\Vert \nabla f(x^{(t_\tau )})\Vert \ge \varepsilon$ for all $\tau$. On the other hand, the relation (15) implies that there exist some $j > t_\tau$ such that $\Vert \nabla f(x^{(j)})\Vert \le \frac{\varepsilon }{2}$. Let $j_\tau$ be the least integer for each $\tau$ satisfying these inequalities. Then, we have

$$\begin{aligned} \begin{array}{rl} \frac{\varepsilon }{2} &{} \le \Vert \nabla f(x^{(t_\tau )})\Vert - \Vert \nabla f(x^{(j_\tau )})\Vert \\ &{} \le \Vert \nabla f(x^{(t_\tau )}) - \nabla f(x^{(j_\tau )})\Vert \\ &{} \le LK\Vert x^{(t_\tau )} - x^{(j_\tau )}\Vert \le LK\frac{B+C(M+1)}{M+1}\sum \limits _{k=t_\tau }^{j_\tau -1} M_k, \end{array} \end{aligned}$$

where the last inequality follows from Lemma 2. Since $0 < M \le U$, there exists $\zeta \le \frac{M}{U} \le 1$ such that

$$\begin{aligned} M + \beta _k \ge \zeta U + \beta _k \ge \zeta (U + \beta _k) \implies M_k \le \frac{1}{\zeta } U_k. \end{aligned}$$

Thus, we obtain

$$\begin{aligned} 0 < \hat{B} \equiv \frac{\varepsilon (M+1)\zeta }{2LK(B+C(M+1))} \le \sum _{k=t_\tau }^{j_\tau - 1} U_k. \end{aligned}$$

Then, using together with inequality (13), we obtain

$$\begin{aligned} \begin{array}{rl} f(x^{(t_\tau )}) - f(x^{(j_\tau )}) &{} \ge \sum \limits _{k=t_\tau }^{j_\tau -1} U_k\Vert \nabla f(x^{(k)})\Vert ^2 - \bar{B}\sum \limits _{k=t_\tau }^{j_\tau -1}M_k^2 \\ &{} \ge \hat{B} \underset{t_\tau \le k \le j_\tau -1}{\inf } \Vert \nabla f(x^{(k)})\Vert ^2 - \bar{B}\sum \limits _{k=t_\tau }^{\infty }M_k^2. \end{array} \end{aligned}$$

Since the left-hand-side of the inequality converges and the condition (10) holds, we have

$$\begin{aligned} \lim _{\tau \uparrow \infty }~\underset{t_\tau \le k \le j_\tau -1}{\inf } \Vert \nabla f(x^{(k)})\Vert ^2 = 0. \end{aligned}$$

(16)

But our choice of $t_\tau$ and $j_\tau$ guarantees $\Vert \nabla f(x^{(k)})\Vert > \frac{\varepsilon }{2}$ for all $t_\tau \le k \le j_\tau$, and hence, we arrive at a contradiction with (16). Therefore, ${\nabla f(x^{(t)})}$ converges, and with the continuity of the gradient, we conclude for each accumulation point $x^*$ of the sequence $\{x^{(t)}\}$ that $\nabla f(x^*) = 0$ holds.