Projective Approximation Based Quasi-Newton Methods

Abstract

We consider a problem of optimizing convex function of vector parameter. Many quasi-Newton optimization methods require to construct and store an approximation of Hessian matrix or its inverse to take function curvature into account, thus imposing high computational and memory requirements. We propose four quasi-Newton methods based on consecutive projective approximation. The idea of these methods is to approximate the product of the function Hessian inverse and function gradient in a low-dimensional space using appropriate projection and then reconstruct it back to original space as a new direction for the next estimate search. By exploiting Hessian rank deficiency in a special way it does not require to store Hessian matrix neither its inverse thus reducing memory requirements. We give a theoretical motivation for the proposed algorithms and prove several properties of corresponding estimates. Finally, we provide a comparison of the proposed methods with several existing ones on modelled data. Despite the fact that the proposed algorithms turned out to be inferior to the limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) one, they have important advantage of being easy to extent and improve. Moreover, two of them do not require the function gradient knowledge.

A Proofs

Proof

(of Proposition 1)

$$\begin{aligned}&\partial _{\mathbf {x}} \mathop {\arg \!\min }\limits _{ \{\mathbf {x}\in \mathbb {R}^d\,:\,\mathbf {P}\mathbf {x} = \widehat{\mathbf {z}}\} } \sum \limits _{t=1}^{K} \left\| \mathbf {x}_t - \mathbf {x} \right\| _2^2 \\&=\partial _{\mathbf {x}} \sum \limits _{t=1}^{K} \left\| \mathbf {P}^\top \mathbf {P}\mathbf {x}_t + \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \mathbf {x}_t - \left( \mathbf {P}^\top \widehat{\mathbf {z}} + \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \mathbf {x}\right) \right\| _2^2 \\&=\partial _{\mathbf {x}} \sum \limits _{t=1}^{K} \left\| \mathbf {P}^\top \mathbf {P}\mathbf {x}_t - \mathbf {P}^\top \widehat{\mathbf {z}} \right\| _2^2 + \partial _{\mathbf {x}} \sum \limits _{t=1}^{K} \left\| \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \left( \mathbf {x}_t - \mathbf {x} \right) \right\| _2^2 \\&= \sum \limits _{t=1}^{K} 2 (\mathbf {I} - \mathbf {P}^\top \mathbf {P})\left( (\mathbf {I} - \mathbf {P}^\top \mathbf {P}) \mathbf {x}_t - \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \mathbf {x} \right) \\&=2 (\mathbf {I} - \mathbf {P}^\top \mathbf {P}) \sum \limits _{t=1}^{K} \mathbf {x}_t - 2 K \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \mathbf {x} = 0. \end{aligned}$$

Since $(\mathbf {I}-\mathbf {P}^\top \mathbf {P})$ is not invertible the above equation has infinite number of solutions. Hence, we are free to choose any one of them, e.g. $\mathbf {x}=\frac{1}{K}\sum _1^K \mathbf {x}_t$.    $\square$

Proof

(of Proposition 2). From Proposition 1 and the fact that $\mathop {\arg \!\min }\limits _{\mathbf {z}\in \mathbb {R}^q}\, f(\mathbf {P}^\top \mathbf {z} + \mathbf {v}) = -\mathbf {P} \mathbf {H}^{-1} \mathbf {b}$ it follows that $\widehat{\mathbf {x}} = \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \overline{\mathbf {x}} - \mathbf {P}^\top \mathbf {P} \mathbf {H}^{-1}\mathbf {b}$. Hence

$$\begin{aligned}&\left\| \arg \!\min \, f - \widehat{\mathbf {x}} \right\| _2^2 = \left\| -\mathbf {H}^{-1}\mathbf {b} - \widehat{\mathbf {x}} \right\| _2^2 \\&\qquad = \left\| - \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \overline{\mathbf {x}} + \mathbf {P}^\top \mathbf {P} \mathbf {H}^{-1}\mathbf {b} - \mathbf {H}^{-1} \mathbf {b} \right\| _2^2 \\&\qquad = \left\| (\mathbf {I} - \mathbf {P}^\top \mathbf {P})\left( \mathbf {H}^{-1} \mathbf {b} - \overline{\mathbf {x}} \right) \right\| _2^2. \end{aligned}$$

   $\square$

Proof

(of Proposition 3). First,

$$\begin{aligned} \Vert \mathbf {H} - \widehat{\mathbf {H}}\Vert _F^2 = \Vert (\mathbf {I} - \mathbf {P}^\top \mathbf {P})\mathbf {H} (\mathbf {I} - \mathbf {P}^\top \mathbf {P})\Vert _F ^2 + \Vert \mathbf {P}^\top (\mathbf {P} \mathbf {H}\mathbf {P}^\top - \widehat{\mathbf {Q}}) \mathbf {P}\Vert _F^2 \end{aligned}$$

One can note that $\widehat{\mathbf {Q}}_{i,j}$ are normally distributed variables s.t. $\mathrm {E}\left[ \widehat{\mathbf {Q}}\right] = \mathbf {P}\mathbf {H}\mathbf {P}^\top$ (for example, see [1]). Moreover, consider a vectorization of the matrix $\widehat{\mathbf {Q}}$ upper triangle $\widehat{\mathbf {\theta }}$:

$$\begin{aligned} \widehat{\mathbf {\theta }} = \left( \widehat{\mathbf {Q}}_{1,1}, \widehat{\mathbf {Q}}_{1,2}, \ldots , \widehat{\mathbf {Q}}_{q-1,q}, \widehat{\mathbf {Q}}_{q,q}\right) , \end{aligned}$$

— its covariance matrix is equal to $\mathbf {\Sigma }_\theta = \frac{\sigma _\varepsilon }{m}\ddot{\mathbf {Z}}\ddot{\mathbf {Z}}^\top$, where $\ddot{\mathbf {Z}}_{i,\cdot }$ consist of quadratic elements of $\mathbf {z}_i = \mathbf {P}\mathbf {x}_i$:

$$\begin{aligned} \ddot{\mathbf {Z}}_{i,\cdot } = \left( \mathbf {z}_{i}^{(1)} \mathbf {z}_{i}^{(1)}, \mathbf {z}_{i}^{(1)} \mathbf {z}_{i}^{(2)}, \ldots , \mathbf {z}_{i}^{(q - 1)} \mathbf {z}_{i}^{(q)}, \mathbf {z}_{i}^{(q)} \mathbf {z}_{i}^{(q)}\right) ^\top . \end{aligned}$$

Next, denote $\mathbf {\theta }$ as a vectorization of the $\mathbf {P} \mathbf {H}\mathbf {P}^\top$ matrix upper triangle and consider eigendecomposition of $\mathbf {\Sigma }_\theta = \mathbf {U} \mathbf {\Lambda } \mathbf {U}^\top$. Then, vector $\widehat{\mathbf {\beta }} = \mathbf {U}\widehat{\mathbf {\theta }}$ would have gaussian distribution with covariance matrix $\mathbf {\Lambda }$, and

$$\begin{aligned} \Vert \mathbf {P}^\top (\mathbf {P} \mathbf {H}\mathbf {P}^\top - \widehat{\mathbf {Q}}) \mathbf {P}\Vert _F^2 \le \Vert \mathbf {P}^\top \Vert _F^2 \Vert \mathbf {P} \mathbf {H}\mathbf {P}^\top - \widehat{\mathbf {Q}} \Vert _F^2 \Vert \mathbf {P}\Vert _F^2 = q^2 \Vert \mathbf {P} \mathbf {H}\mathbf {P}^\top - \widehat{\mathbf {Q}} \Vert _F^2 . \\ \Vert \mathbf {P} \mathbf {H}\mathbf {P}^\top - \widehat{\mathbf {Q}} \Vert _F^2 = \Vert \widehat{\mathbf {\theta }} - \mathbf {\theta } \Vert _2^2 = \left( \mathbf {U}(\widehat{\mathbf {\theta }} - \mathbf {\theta })\right) ^\top \left( \mathbf {U}(\widehat{\mathbf {\theta }} - \mathbf {\theta })\right) = \sum _{i=1}^{q^2} \xi _2 \sim \sum _{i=1}^{q^2} \lambda _i^2 \chi ^2\ (1). \end{aligned}$$

Thus, $C(\mathbf {X}\mathbf {P}^\top ) = \max \limits _i \lambda _i^2$.

   $\square$