Frank-Wolfe Style Algorithms for Large Scale Optimization

Abstract

We introduce a few variants on Frank-Wolfe style algorithms suitable for large scale optimization. We show how to modify the standard Frank-Wolfe algorithm using stochastic gradients, approximate subproblem solutions, and sketched decision variables in order to scale to enormous problems while preserving (up to constants) the optimal convergence rate $\mathcal {O}\left (\frac {1}{k}\right )$.

Appendix

We prove the following simple proposition about L-smooth functions used in Sect. 9.2.

Proposition 3

If f is a real valued differentiable convex function with domain R ⁿ and satisfies ∥∇f(x) −∇f(y)∥≤ L∥x − y∥, then for all x, y ∈R ⁿ ,

$$\displaystyle \begin{aligned}f(x)\leq f(y) + \nabla f(y)^T (x-y)+\frac{L}{2}\|x-y\|{}^2.\end{aligned}$$

Proof

The inequality follows from the following computation:

$$\displaystyle \begin{aligned} \begin{array}{rcl} f(x) - f(y) -\nabla f(y)^T(x-y)&\displaystyle =&\displaystyle \int_0^1 \left(\nabla f(y+t(x-y))\right.\\ &\displaystyle &\displaystyle -\left.\nabla f(y)\right)^T(x-y)dt \\ &\displaystyle \leq&\displaystyle \int_0^1 \| \left(\nabla f(y+t(x-y))\right.\\ &\displaystyle &\displaystyle -\left.\nabla f(y)\right)^T(x-y)\|dt \\ &\displaystyle \leq&\displaystyle \int_0^1 \| \left(\nabla f(y+t(x-y))\right.\\ &\displaystyle &\displaystyle \left.-\nabla f(y)\right)\|\|(x-y)\|dt \\ &\displaystyle \leq&\displaystyle \int_0^1 Lt\|x-y\|{}^2dt\\ &\displaystyle =&\displaystyle \frac{L}{2}\|x-y\|{}^2. \end{array} \end{aligned} $$

(9.26)

□