Equivalent Lipschitz surrogates for zero-norm and rank optimization problems

Abstract

This paper proposes a mechanism to produce equivalent Lipschitz surrogates for zero-norm and rank optimization problems by means of the global exact penalty for their equivalent mathematical programs with an equilibrium constraint (MPECs). Specifically, we reformulate these combinatorial problems as equivalent MPECs by the variational characterization of the zero-norm and rank function, show that their penalized problems, yielded by moving the equilibrium constraint into the objective, are the global exact penalization, and obtain the equivalent Lipschitz surrogates by eliminating the dual variable in the global exact penalty. These surrogates, including the popular SCAD function in statistics, are also difference of two convex functions (D.C.) if the function and constraint set involved in zero-norm and rank optimization problems are convex. We illustrate an application by designing a multi-stage convex relaxation approach to the rank plus zero-norm regularized minimization problem.

Appendices

Appendix A

The following lemma characterizes an important property of the function family $\Phi$.

Lemma 1

Let $\phi \in \Phi$. Then, there exists $t_0\in [0,1)$ such that $\frac{1}{1-t^*}\in \partial \phi (t_0)$, and for any given $\omega \ge 0$ and $\varrho >0$, the optimal value $\upsilon ^*\!:=\min _{t\in [0,1]}\{\phi (t)+\varrho \omega (1-\!t)\}$ satisfies

$$\begin{aligned} \left\{ \begin{array}{ll} \upsilon ^*=1&{}\quad \mathrm{if}\ \varrho \omega \in (\phi _{-}'(1),+\infty );\\ \upsilon ^*\ge \frac{\varrho \omega (1-t_0)}{\phi _{-}'(1)(1-t^*)}&{}\quad \mathrm{if}\ \varrho \omega \in \big [\frac{1}{1-t^*},\phi _{-}'(1)\big ];\\ \upsilon ^*\ge \varrho \omega (1-t_0) &{}\quad \mathrm{if}\ \varrho \omega \in \big [0,\frac{1}{1-t^*}\big ). \end{array}\right. \end{aligned}$$

Proof

If $\phi '(t)$ is a constant for $t\in [t^*,1]$, then $\phi '(t)=\frac{\phi (1)-\phi (t^*)}{1-t^*}=\frac{1}{1-t^*}$ for all $t\in [t^*,1]$, which means that any $t_0\in [t^*,1)$ satisfies the requirement. Otherwise, there must exist a point $\overline{t}\in (t^*,1)$ such that $\phi _{-}'(\overline{t})<\phi _{-}'(1)$. Together with the convexity of $\phi$ in [0, 1], we have $\phi _{-}'(t)\le \phi _{-}'(\overline{t})$ for $t\in [t^*,\overline{t}]$. By [37, Corollary 24.2.1], it follows that

$$\begin{aligned} 1&=\phi (1)-\phi (t^*) =\int _{t^*}^{1}\phi _{-}'(t)dt =\int _{t^*}^{\overline{t}}\phi _{-}'(t)dt+\int _{\overline{t}}^{1}\phi _{-}'(t)dt\\&<\phi _{-}'(1)(\overline{t}-t^*)+\int _{\overline{t}}^{1}\phi _{-}'(t)dt \le \phi _{-}'(1)(1-t^*). \end{aligned}$$

Also, by the convexity of $\phi$ in [0, 1], $1=\phi (1)\ge \phi (t^*)+\phi _{+}'(t^*)(1-t^*)=\phi _{+}'(t^*)(1-t^*)$. Thus, $a:=\frac{1}{1-t^*}\in [\phi _{+}'(t^*),\phi _{-}'(1))\subseteq [\phi _{+}'(0),\phi _{-}'(1)]$, which further implies that

$$\begin{aligned} (\partial \phi )^{-1}(a)=(\partial \psi )^{-1}(a)=\partial \psi ^*(a)\subseteq [0,1]. \end{aligned}$$

Notice that $(\partial \phi )^{-1}(a)\cap [0,1)\ne \emptyset$ (if not, $a\in \partial \phi (1)=[\phi _{-}'(1),\phi _{+}'(1)]$, which is impossible since $a<\phi _{-}'(1)$). Therefore, $t_0\in (\partial \phi )^{-1}(a)\cap [0,1)$ satisfies the requirement.

When $\varrho \omega \ge \phi _{-}'(1)$, clearly, $\upsilon ^*=\phi (1)=1$ since $\phi (t)+\varrho \omega (1\!-t)$ is nonincreasing in [0, 1]. When $\varrho \omega \in \big [0,\frac{1}{1-t^*}\big )$, since $\phi _{-}'(t)\ge \phi _{+}'(t_0)>\varrho \omega$ for $t>t_0$, the optimal solution $\widehat{t}$ of $\min _{t\in [0,1]}\{\phi (t)+\varrho \omega (1-t)\}$ satisfies $\widehat{t}\le t_0$. Along with the convexity of $\phi$ in [0, 1],

$$\begin{aligned} \phi (t)+\varrho \omega (1\!-t) \ge \phi (\widehat{t})+\varrho \omega (1\!-\widehat{t}) \ge \varrho \omega (1-t_0)\quad \ \forall t\in [0,1]. \end{aligned}$$

This shows that $\upsilon ^*\ge \varrho \omega (1-t_0)$ for this case. When $\varrho \omega \in \big [\frac{1}{1-t^*},\phi _{-}'(1)\big ]$, it follows that

(42)

If $t_0>0$, from the fact that $t_0<1$ and $\frac{1}{1-t^*}\in \partial \phi (t_0)$, it immediately follows that

$$\begin{aligned} \min _{t\in [0,1]}\Big \{\phi (t)+\frac{1}{1-t^*}(1-t)\Big \} =\phi (t_0)+\frac{1}{1-t^*}(1-t_0)\ge \frac{1-t_0}{1-t^*}. \end{aligned}$$

Together with (42) and $\varrho \omega \le \phi _{-}'(1)$, we have $\upsilon ^*\ge \frac{1-t_0}{1-t^*} \ge \frac{\varrho \omega (1-t_0)}{\phi _{-}'(1)(1-t^*)}$. If $t_0=0$, from $\frac{1}{1-t^*}\in \partial \phi (t_0)$ we have $\phi _{+}'(0)\ge \frac{1}{1-t^*}\ge 1=\phi (1)\ge \phi (0)+\phi _{+}'(0)\ge \phi _{+}'(0),$ where the third inequality is due to the convexity of $\phi$ at [0, 1]. Then, for any $t\in [0,1]$,

$$\begin{aligned} \phi (t)+\frac{1}{1-t^*}(1-t) \ge \phi (0)+\phi _{+}'(0)t+\frac{1}{1-t^*}(1-t)\ge \phi (0)+\frac{1}{1-t^*}\ge \frac{1}{1-t^*}, \end{aligned}$$

where the first inequality is using the convexity of $\phi$ at [0, 1]. Together with (42), it follows that $\upsilon ^*\ge \frac{1}{1-t^*}\ge \frac{\varrho \omega }{\phi _{-}'(1)(1-t^*)} \ge \frac{\varrho \omega (1-t_0)}{\phi _{-}'(1)(1-t^*)}$. The proof is completed. $\square$

Lemma 2

Let $\phi \in \Phi$. For any given $\varrho >0$, the function $\Theta _{\varrho }(X):={\textstyle \sum _{i=1}^{n_1}}\psi ^*(\varrho \sigma _i(X))$ is lsc and convex in $\mathbb {R}^{n_1\times n_2}$, where $\psi ^*$ is the conjugate of $\psi$, defined by (1) with $\phi$.

Proof

Let $\widehat{\Psi }(x):=\sum _{i=1}^{n_1}\widehat{\psi }(x_i)$ for $x\in \mathbb {R}^{n_1}$, where $\widehat{\psi }(t):=\psi (|t|)$ for $t\in \mathbb {R}$. Clearly, $\widehat{\Psi }$ is absolutely symmetric, i.e., $\widehat{\Psi }(x)=\widehat{\Psi }(Px)$ for any signed permutation matrix $P\in \mathbb {R}^{n_1\times n_1}$. Moreover, by the definitions of $\psi$ and $\widehat{\Psi }$, the conjugate function $\widehat{\Psi }^*$ of $\widehat{\Psi }$ satisfies

(43)

By the definition of $\Theta _\varrho$, we have $\Theta _\varrho (X)\equiv (\widehat{\Psi }^*\circ \sigma )(\varrho X)$. Notice that $\widehat{\Psi }^*$ is lsc and convex. By [23, Lemma 2.3(b) & Corollary 2.6], $\Theta _\varrho$ is convex and lsc on $\mathbb {R}^{n_1\times n_2}$. $\square$

Appendix B

Example 1

Let $\phi (t):=t$ for $t\in \mathbb {R}$. Clearly, $\phi \in \Phi$ with $t^*=0$. After a calculation,

$$\begin{aligned} \psi ^*(s)=\left\{ \begin{array}{ll} s-1 &{} \quad \text {if}\ s>1;\\ 0 &{} \quad \text {if}\ s\le 1. \end{array}\right. \end{aligned}$$

Example 2

Let $\phi (t):=\frac{\varphi (t)}{\varphi (1)}$ with $\varphi (t)=-t-\frac{q-1}{q}(1-t+\epsilon )^{\frac{q}{q-1}}+\epsilon +\frac{q-1}{q}\,(0<q<1)$ for $t\in (-\infty , 1+\epsilon ]$, where $\epsilon \in (0,1)$ is a fixed constant. Now one has $\psi ^*(s)=\!\frac{h(\varphi (1)s)}{\varphi (1)}$ with

$$\begin{aligned} h(s):=\left\{ \begin{array}{cl} s+\frac{q-1}{q}\epsilon ^{\frac{q}{q-1}}-\epsilon +\frac{1}{q} &{} \quad \text {if}\ s\ge \epsilon ^{\frac{1}{q-1}}-1,\\ (1\!+\epsilon )s-\frac{1}{q}(s+1)^q+\frac{1}{q} &{} \quad \text {if}\ (1\!+\epsilon )^{\frac{1}{q-1}}-1<s<\!\epsilon ^{\frac{1}{q-1}}-1,\\ \frac{q-1}{q}(1+\epsilon )^{\frac{q}{q-1}}-\epsilon -\frac{q-1}{q} &{} \quad \text {if}\ s\le (1\!+\epsilon )^{\frac{1}{q-1}}-1. \end{array}\right. \end{aligned}$$

Example 3

Let $\phi (t):=\frac{\varphi (t)}{\varphi (1)}$ with $\varphi (t)=-t-\ln (1-t+\epsilon )+\epsilon$ for $t\in (-\infty ,1+\epsilon )$, where $\epsilon \in (0,1)$ is a fixed constant. Clearly, $\phi \in \Phi$ with $t^*=\epsilon$. Now $\psi ^*(s)=\frac{1}{\varphi (1)}h(\varphi (1)s)$ with

$$\begin{aligned} h(s):=\left\{ \begin{array}{cl} s+1+\ln (\epsilon )-\epsilon &{} \quad \text {if}\ s\ge \frac{1}{\epsilon }-1,\\ s(1+\epsilon )-\ln (s+1) &{} \quad \text {if}\ \frac{1}{1+\epsilon }-1<s< \frac{1}{\epsilon }-1,\\ \ln (1+\epsilon )-\epsilon &{} \quad \text {if}\ s\le \frac{1}{1+\epsilon }-1. \end{array}\right. \end{aligned}$$

Example 4

Let $\phi (t):=\frac{\varphi (t)}{\varphi (1)}$ with $\varphi (t)=(1+\epsilon )\arctan \left( \sqrt{\frac{t}{1-t+\epsilon }}\right) -\sqrt{t(1-t+\epsilon )}$ for $t\in [0,1]$, where $\epsilon \in (0,1)$ is a fixed constant. Now one has $\psi ^*(s)=\frac{1}{\varphi (1)}h(\varphi (1)s)$ with

$$\begin{aligned} h(s):=\left\{ \begin{array}{cl} s-(1+\epsilon )\arctan (\sqrt{1/\epsilon })+\sqrt{\epsilon } &{} \quad \text {if}\ s\ge \!\sqrt{1/\epsilon },\\ s(1+\epsilon )-(1+\epsilon )\arctan (s) &{} \quad \text {if}\ 0<s<\!\sqrt{1/\epsilon },\\ 0 &{}\quad \text {if}\ s\le 0. \end{array}\right. \end{aligned}$$

Example 5

Let $\phi (t):=\frac{\varphi (t)}{\varphi (1)}$ with $\varphi (t)=\frac{a-1}{2}t^2+t$ for $t\in \mathbb {R}$, where $a>1$ is a fixed constant. Clearly, $\phi \in \Phi$ with $t^*=0$. For such $\phi$, one has $\psi ^*(s)=\frac{1}{\varphi (1)}h(\varphi (1)s)$ with

$$\begin{aligned} h(s):=\left\{ \!\begin{array}{ll} 0 &{}\quad \mathrm{if}\ s\le 1,\\ \frac{(s-1)^2}{2(a-1)}&{}\quad \mathrm{if}\ 1<s\le a,\\ s-\frac{a+1}{2} &{}\quad \mathrm{if}\ s>a. \end{array}\right. \end{aligned}$$

Now the objective function in (9) with $m=n$ and $J_i=\{i\}$, i.e., $\sum _{i=1}^n\big [\varrho |x_i|-\psi ^*(\varrho |x_i|)\big ]$ is exactly the SCAD function [15]. This shows that the minimization problem of the SCAD function is an equivalent surrogate for the zero-norm problem under a mild condition.