Accelerating Gradient Descent with Projective Response Surface Methodology

Abstract

We present a new modification of gradient descent algorithm based on surrogate optimization with projection into low-dimensional space. It consequently builds an approximation of the target function in low-dimensional space and takes the approximation optimum point mapped back to original parameter space as the next parameter estimate. An additional projection step is used to fight the curse of dimensionality. Major advantage of the proposed modification is that it does not change gradient descent iterations, thus it may be used with almost any zero- or first-order iterative method. We give a theoretical motivation for the proposed algorithm and experimentally illustrate its properties on modelled data.

A Proofs

Proof

(of Proposition 1)

Since $(\mathbf {I}-\mathbf {P}^\top \mathbf {P})$ is not invertible, the above equation has an infinite number of solutions. Hence, we are free to choose any one of them, e.g. $\mathbf {x}=\frac{1}{K}\sum _1^K \mathbf {x}_t$.

Proof

(of Proposition 2)

$$\begin{aligned} f(\mathbf {x})&= \left( \mathbf {P}^\top \mathbf {z} + \mathbf {v}\right) ^\top \mathbf {A} \left( \mathbf {P}^\top \mathbf {z} + \mathbf {v}\right) + \mathbf {b}^\top \left( \mathbf {P}^\top \mathbf {z} + \mathbf {v}\right) + c \\&= \mathbf {z}^\top \mathbf {P} \mathbf {A} \mathbf {P}^\top \mathbf {z} + \left( \mathbf {b}^\top + \mathbf {v}^\top \mathbf {A} \right) \mathbf {P}^\top \mathbf {z} + \left( \mathbf {v}^\top \mathbf {A} \mathbf {v} - \mathbf {b}^\top \mathbf {v} + c \right) . \end{aligned}$$

Substituting $\mathbf {v}$ back and taking derivative with respect to $\mathbf {z}$, we obtain:

$$\begin{aligned} 0&=\; 2 \mathbf {P} \mathbf {A} \mathbf {P}^\top \widehat{\mathbf {z}} + \left( \mathbf {b}^\top + \mathbf {x}^\top \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) ^\top \mathbf {A}\right) \mathbf {P}^\top \\ \leadsto \widehat{\mathbf {z}}&= -\frac{1}{2} \left( \mathbf {P} \mathbf {A} \mathbf {P}^\top \right) ^{-1} \left( \left( \mathbf {b}^\top + \mathbf {x}^\top \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) ^\top \mathbf {A}\right) \mathbf {P}^\top \right) ^\top \\&= -\frac{1}{2} \mathbf {P} \left( \mathbf {A}^{-1} \mathbf {b} + \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P} \right) \mathbf {x}\right) = -\frac{1}{2} \mathbf {P} \mathbf {A}^{-1} \mathbf {b}. \end{aligned}$$

Proof

(of Proposition 3) From Propositions 1 and 2: $\widehat{\mathbf {x}} = \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \overline{\mathbf {x}} - \frac{1}{2} \mathbf {P}^\top \mathbf {P} \mathbf {A}^{-1}\mathbf {b}$. Hence

$$\begin{aligned}&\Vert \arg \!\min f - \widehat{\mathbf {x}} \Vert _2^2 = \bigg \Vert -\frac{1}{2}\mathbf {A}^{-1}\mathbf {b} - \widehat{\mathbf {x}} \bigg \Vert _2^2 \\&\qquad = \bigg \Vert - \left( \mathbf {I} - \mathbf {P}^\top \mathbf {P}\right) \overline{\mathbf {x}} + \frac{1}{2} \mathbf {P}^\top \mathbf {P} \mathbf {A}^{-1}\mathbf {b} - \frac{1}{2}\mathbf {A}^{-1} \mathbf {b} \bigg \Vert _2^2 \\&\qquad = \bigg \Vert (\mathbf {I} - \mathbf {P}^\top \mathbf {P})\left( \frac{1}{2} \mathbf {A}^{-1} \mathbf {b} - \overline{\mathbf {x}} \right) \bigg \Vert _2^2. \end{aligned}$$