A Cost Based Reweighted Scheme of Principal Support Vector Machine

Abstract

Principal Support Vector Machine (PSVM) is a recently proposed method that uses Support Vector Machines to achieve linear and nonlinear sufficient dimension reduction under a unified framework. In this work, a reweighted scheme is used to improve the performance of the algorithm. We present basic theoretical results and demonstrate the effectiveness of the reweighted algorithm through simulations and real data application.

Appendix

Proof of Theorem 1.

Without loss of generality, assume that $E\boldsymbol{X} =\boldsymbol{ 0}$. First we note that for $i = 1,-1$

$$\displaystyle\begin{array}{rcl} E\left (\lambda _{\tilde{Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]^{+}\right ) =& E\{E[\left (\lambda _{\tilde{ Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]^{+}\right )\vert Y,\boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}]\}& {}\\ =& E\{E[\left (\lambda _{\tilde{Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]\right )^{+}\vert Y,\boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}]\}& {}\\ \end{array}$$

where the last equality holds because $\lambda _{\tilde{Y }}$ is positive. Since the function a ↦ a ⁺ is convex, by Jensen’s inequality we have

$$\displaystyle\begin{array}{rcl} E[\left (\lambda _{\tilde{Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]\right )^{+}\vert Y,\boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}] \geq & \{E[\lambda _{\tilde{ Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]\vert Y,\boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}]\}^{+}& {}\\ =& \{\lambda _{\tilde{Y }}[1 -\tilde{ Y }(E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}) - t)]\}^{+} & {}\\ \end{array}$$

where the equality follows from the model assumption (1.1). Thus

$$\displaystyle\begin{array}{rcl} E\left (\lambda _{\tilde{Y }}[1 -\tilde{ Y }(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X} - t)]^{+}\right ) \geq E& \{\lambda _{\tilde{ Y }}[1 -\tilde{ Y }(E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}) - t)]\}^{+}.&{}\end{array}$$

(1.12)

On the first term now we have:

$$\displaystyle\begin{array}{rcl} \boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{\varSigma }\boldsymbol{\psi } =\mathrm{ var}(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}) =& \mathrm{var}[E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X})] + E[\mathrm{var}(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X})]& \\ \geq & \mathrm{var}[E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X})]. &{}\end{array}$$

(1.13)

Combining (1.12) and (1.13),

$$\displaystyle\begin{array}{rcl} L_{R}(\boldsymbol{\psi },t) \geq \mathrm{ var}[E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X})] + E& \{\lambda _{\tilde{ Y }}[1 -\tilde{ Y }(E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}) - t)]\}^{+}.& {}\\ \end{array}$$

By the definition of the linearity condition in the theorem $E(\boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{X}\vert \boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{X}) = \boldsymbol{\psi }^{\mathsf{T}}\boldsymbol{P}_{\boldsymbol{\beta }}^{\mathsf{T}}(\boldsymbol{\varSigma })\boldsymbol{X}$ and therefore the right-hand side of the inequality is equal to $L_{R}(\boldsymbol{P}_{\boldsymbol{\beta }}(\boldsymbol{\varSigma })\boldsymbol{\psi },t)$. If $\boldsymbol{\psi }$ is not in the CDRS then the inequality is strict which implies $\boldsymbol{\psi }$ is not the minimizer. □ 

Proof of Theorem 2.

Following the same argument as in Vapnik [14] it can be shown that minimizing (1.9) is equivalent tov

$$\displaystyle\begin{array}{rcl} \begin{array}{ll} &\mbox{ minimizing}\ \ \boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{\zeta } + \frac{1} {n}(\boldsymbol{\lambda }^{{\ast}})^{\mathsf{T}}\boldsymbol{\xi }\ \ \mbox{ over}\ \ (\boldsymbol{\zeta },t,\boldsymbol{\xi }) \\ &\mbox{ subject to}\ \ \ \boldsymbol{\xi } \geq \boldsymbol{ 0},\ \ \tilde{y} \odot (\boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{Z} - t\boldsymbol{1}) \geq \boldsymbol{ 1} -\boldsymbol{\xi }.\end{array} & &{}\end{array}$$

(1.14)

The Lagrangian function of this problem is

$$\displaystyle\begin{array}{rcl} L(\boldsymbol{c},t,\boldsymbol{\xi },\boldsymbol{\alpha },\boldsymbol{\beta }) = \boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{\zeta } + \frac{1} {n}(\boldsymbol{\lambda }^{{\ast}})^{\mathsf{T}}\boldsymbol{\xi } -\boldsymbol{\alpha }^{\mathsf{T}}[\tilde{y} \odot (\boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{Z} - t\boldsymbol{1}) -\boldsymbol{ 1} + \boldsymbol{\xi }] -\boldsymbol{\beta }^{\mathsf{T}}\boldsymbol{\xi }.& &{}\end{array}$$

(1.15)

where $\boldsymbol{\xi } = (\xi _{1},\ldots,\xi _{n})$. Let $(\boldsymbol{\zeta }^{{\ast}},\boldsymbol{\xi }^{{\ast}},t^{{\ast}})$ be a solution to problem (1.14). Using the Kuhn–Tucker Theorem, one can show that minimizing over $(\boldsymbol{\zeta },t,\boldsymbol{\xi })$ is similar as maximizing over $(\boldsymbol{\alpha },\boldsymbol{\beta })$. So, differentiating with respect to $\boldsymbol{\zeta }$, t, and $\boldsymbol{\xi }$ to obtain the system of equations:

$$\displaystyle\begin{array}{rcl} \left \{\begin{array}{@{}l@{\quad }l@{}} \partial L/\partial \boldsymbol{\zeta } = 2\boldsymbol{\zeta } -\boldsymbol{Z}^{\mathsf{T}}(\boldsymbol{\alpha } \odot \tilde{ y}) =\boldsymbol{ 0}\quad \\ \partial L/\partial t = \boldsymbol{\alpha }^{\mathsf{T}}\tilde{y} =\boldsymbol{ 0} \quad \\ \partial L/\partial \boldsymbol{\xi } = \frac{1} {n}\boldsymbol{\lambda }^{{\ast}}-\boldsymbol{\alpha }-\boldsymbol{\beta } =\boldsymbol{ 0}. \quad \end{array} \right.& &{}\end{array}$$

(1.16)

Substitute the last two equations above into (1.15) to obtain

$$\displaystyle\begin{array}{rcl} \boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{\zeta } -\boldsymbol{\alpha }^{\mathsf{T}}[\tilde{y} \odot (\boldsymbol{\zeta }^{\mathsf{T}}\boldsymbol{Z}) -\boldsymbol{ 1}]& &{}\end{array}$$

(1.17)

Now substitute the first equation in (1.16) ($\boldsymbol{\zeta } = \frac{1} {2}\boldsymbol{Z}^{\mathsf{T}}(\boldsymbol{\alpha } \odot \tilde{ y})$) in the above:

$$\displaystyle\begin{array}{rcl} \boldsymbol{1}^{\mathsf{T}}\boldsymbol{\alpha } -\frac{1} {4}(\boldsymbol{\alpha } \odot \tilde{ y})^{\mathsf{T}}\boldsymbol{Z}\boldsymbol{Z}^{\mathsf{T}}(\boldsymbol{\alpha } \odot \tilde{ y}).& &{}\end{array}$$

(1.18)

Thus to minimize (1.15) we need to maximize (1.18) over the constraints

$$\displaystyle\begin{array}{rcl} \left \{\begin{array}{@{}l@{\quad }l@{}} \boldsymbol{\alpha }^{\mathsf{T}}\tilde{y} =\boldsymbol{ 0} \quad \\ \frac{1} {n}\boldsymbol{\lambda }^{{\ast}}-\boldsymbol{\alpha }-\boldsymbol{\beta } =\boldsymbol{ 0}.\quad \end{array} \right.& &{}\end{array}$$

(1.19)

which are equivalent to the constraints in (1.10). □