Improved Sublinear Primal-Dual Algorithm for Support Vector Machines

Abstract

Sublinear primal-dual algorithm (SUPDA) is a well established sublinear time algorithm. However, SUPDA performs the primal step in every iteration, which is unnecessary since the overall regret of SUPDA is dominated by the dual step. To improve the efficiency of SUPDA, we propose an improved SUPDA (ISUPDA), and apply ISUPDA to linear support vector machines, which yields an improved sublinear primal-dual algorithm for linear support vector machines (ISUPDA-SVM). Specifically, different from SUPDA that conducts the primal step in every iteration, ISUPDA executes the primal step with a probability at each iteration, which can reduce the time complexity of SUPDA. We prove that the expected regret of ISUPDA is still dominated by the dual step and hence ISUPDA guarantees the convergence. We further convert linear support vector machines into saddle-point forms in order to apply ISUPDA to linear support vector machines, and provide the theoretical guarantee of the quality of solution and efficiency for ISUPDA-SVM. Comparison experiments on multiple datasets demonstrate that ISUPDA outperforms SUPDA and that ISUPDA-SVM is an efficient algorithm for linear support vector machines.

Appendix

1.1 Proof of Lemma 3

Proof

We first give a lemma from [8] used in the proof.

Lemma 5

Consider functions $f_1,\ldots ,f_T$, the norm of their gradients is bounded by $G\ge \max _t\max _{\varvec{\omega }\in \mathbb {R}^d}\Vert \nabla f_t(\varvec{\omega })\Vert$. Let $\varvec{\omega }_{t+1}\leftarrow \arg \max _{\varvec{\omega }\in \mathbb {R}^d}\sum _{\tau =1}^{t}f_{\tau }(\varvec{\omega }).$ Then

$$ \max _{\varvec{\omega }\in \mathbb {R}^d}\sum _{t=1}^{T}f_t(\varvec{\omega })-\sum _{t=1}^{T}f_t(\varvec{\omega }_t)\le 2G\log T. $$

Consider the sequence of functions $\widehat{f}_t$, let $\widehat{f}_t=\frac{f_t}{\alpha }$ with probability $\alpha$, and with probability $1-\alpha$, $\widehat{f}_t=\varvec{0}$, where $\varvec{0}$ denotes the all-zero function. Then, their gradients are bounded by $\frac{G}{a}$. Applying Lemma 5 on $\widehat{f}_t$ we have

$$\mathbb {E}\left[ \sum _{t=1}^{T}f_t(\varvec{\omega }^*)-\sum _{t=1}^{T}f_t(y_t)\right] =\mathbb {E}\left[ \sum _{t=1}^{T}\widehat{f}_t(\varvec{\omega }^*)-\sum _{t=1}^{T}\widehat{f}_t(\varvec{\omega }_t)\right] \le \frac{2G}{\alpha }\log T,$$

which ends the proof.

1.2 Proof of Theorem 1

Proof

We first give some basic lemmas from [12] used in the proof.

Lemma 6

For any $\sqrt{\frac{\log (n)}{T}}\le \eta \le \frac{1}{6}$, the following two formulas hold with probability al least $1-O(\frac{1}{n})$

$$\begin{aligned} \max _{i\in [n]}\sum _{t\in [T]}v_t(i)-\sum _{t\in [T]}\left[ \varvec{x}_i^{T }\varvec{\omega }_t+\xi _t(i)\right] \le 4\eta T,\\ \left| \sum _{t\in [T]}\varvec{p}_t^{T }\varvec{v}_t-\sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right| \le 4\eta T. \end{aligned}$$

Lemma 7

With probability al least $\frac{3}{4}$,

$$\sum _{t=1}^{T}\varvec{p}^{\mathrm {T}}_t\varvec{v}^2_t\le 48T.$$

Let $\varvec{\omega }^*,\xi ^*$ be the optimal solutions of problem (7). According to Lemma 3, we have $G=1$ for the optimization objective function of problem (7), so we have

$$ \mathbb {E}_{\{c_t\}}\left[ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right] \ge \mathbb {E}_{\{c_t\}}\left[ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }^*+\varvec{\xi }^*)\right] -\frac{2}{\alpha }\log T. $$

The expectation is taken for updating $\varvec{\omega }_t$, denoted as $c_t$. Let $\gamma ^*$ be the optimal value of problem (7), we have

$$ \mathbb {E}_{\{c_t\}}\left[ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }^*+\varvec{\xi }^*)\right] \ge T\gamma ^*.$$

So we obtain

$$\begin{aligned} \mathbb {E}_{\{c_t\}}\left[ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right] \ge T\gamma ^*-\frac{2}{\alpha }\log T. \end{aligned}$$

(8)

Applying Lemma 2 to the clipped vector $\varvec{v}_t$ we have

$$ \sum _{t\in [T]}\varvec{p}_t^{T }\varvec{v}_t\le \min _{i\in [n]}\sum _{t\in [T]}v_t(i)+\frac{\log n}{\eta }+\eta \sum _{t\in [T]}\varvec{p}_t^{T }\varvec{v}_t^2. $$

From Lemma 6, with probability at least $1-O(1/n)$ we obtain

$$ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\le \min _{i\in [n]}\sum _{t\in [T]}(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)+\frac{\log n}{\eta }+\eta \sum _{t\in [T]}\varvec{p}_t^{T }\varvec{v}_t^2+8\eta T. $$

According to Lemma 7, with probability $\frac{3}{4}-O(1/n)\ge \frac{1}{2}$,

$$ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\le \min _{i\in [n]}\sum _{t\in [T]}(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)+\frac{\log n}{\eta }+56\eta T. $$

Taking expectation with respect to $c_t$, with probability al least $\frac{1}{2}$ we obtain

$$\begin{aligned} \mathbb {E}_{\{c_t\}}\left[ \sum _{t\in [T]}\varvec{p}_t^{T }(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right] \le \mathbb {E}_{\{c_t\}}\left[ \min _{i\in [n]}\sum _{t\in [T]}(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right] +\frac{\log n}{\eta }+56\eta T. \end{aligned}$$

(9)

Combining (8),(9) and dividing by T, with probability al least $\frac{1}{2}$ we have

$$ \frac{1}{T}\mathbb {E}_{\{c_t\}}\left[ \min _{i\in [n]}\sum _{t\in [T]}(\varvec{X} \varvec{\omega }_t+\varvec{\xi }_t)\right] \ge \gamma ^*-\frac{2}{T\alpha }\log T-\frac{\log n}{T\eta }-56\eta , $$

and using our choice for T, $\eta$ and $\alpha$, with probability at least $\frac{1}{2}$,

$$ \mathbb {E}_{\{c_t\}}\left[ \min _{i\in [n]}(\varvec{X} \bar{\varvec{\omega }}_t+\bar{\varvec{\xi }}_t)\right] \ge \gamma ^*-\varepsilon , $$

which implies that $(\bar{\varvec{\omega }},\bar{\varvec{\xi }})$ forms an expected $\varepsilon$-approximate solution to problem (7).

Next, we analyse the runtime.

For the objective function of problem (7), we have $G=1$ in Lemma 3, and combining the time complexity of ISUPDA presented in (4), we can obtain the runtime of ISUPDA-SVM

$$O\left( \varepsilon ^{-2}n\log n+\varepsilon ^{-2}d\sqrt{\frac{\log n}{T}}\log T\right) .$$