A new globally convergent algorithm for non-Lipschitz ℓ_p-ℓ_q minimization

Abstract

We consider the non-Lipschitz ℓ_p-ℓ_q (0 < p < 1 ≤ q < ∞) minimization problem, which has many applications and is a great challenge for optimization. The problem contains a non-Lipschitz regularization term and a possibly nonsmooth fidelity. In this paper, we present a new globally convergent algorithm, which gradually shrinks the variable support and uses linearization and proximal approximations. The subproblem at each iteration is then convex with increasingly fewer unknowns. By showing a lower bound theory for the sequence generated by our algorithm, we prove that the sequence globally converges to a stationary point of the ℓ_p-ℓ_q objective function. Our method can be extended to the ℓ_p-regularized elastic net model. Numerical experiments demonstrate the performances and flexibilities of the proposed algorithm, such as the applicability to measurements with either Gaussian or heavy-tailed noise.

Appendix

We recall some definitions and results here.

Definition A.1 (Subdifferentials 38)

Let $h: \mathbb {R}^{\mathsf {N}} \to \mathbb {R}\cup \{+\infty \}$ be a proper, lower semicontinuous function.

• (i) The regular subdifferential of h at $\bar {\mathbf {x}} \in \text {dom} h = \{ \mathbf {x} \in \mathbb {R}^{\mathsf {N}}: h(\mathbf {x}) < +\infty \}$ is defined as

  $$\widehat{\partial}h(\bar{\mathbf{x}} ) := \left\{\mathbf{v} \in\mathbb{R}^{\mathsf{N}}: \liminf_{\begin{array}{llll} \mathbf{x} \to \bar{\mathbf{x}} \\ \mathbf{x} \neq \bar{\mathbf{x}} \end{array}}\frac{h(\mathbf{x})-h(\bar{\mathbf{x}})- \langle \mathbf{v},\mathbf{x} -\bar{\mathbf{x}} \rangle}{\|\mathbf{x}-\bar{\mathbf{x}}\|}\geq 0 \right\}; $$

• (ii) The (limiting) subdifferential of h at $\bar {\mathbf {x}} \in \text {dom} h$ is defined as

  $$\partial h(\bar{\mathbf{x}} ):=\left\{\mathbf{v} \in\mathbb{R}^{\mathsf{N}}: \exists \mathbf{x}^{(k)} \!\to\! \bar{\mathbf{x}} , h\left( \mathbf{x}^{(k)}\right) \!\to\! h(\mathbf{x}), \mathbf{v}^{(k)}\in \widehat{\partial} h\left( \mathbf{x}^{(k)}\right), \mathbf{v}^{(k)} \!\to\! \mathbf{v} \right\}. $$

Remark 1

From Definition A.1, the following properties hold:

• (i) For any $\bar {\mathbf {x}} \in \text {dom} h$, $\widehat {\partial }h(\bar {\mathbf {x}} ) \subset \partial h(\bar {\mathbf {x}} )$. If h is continuously differentiable at $\bar {\mathbf {x}}$, then $\widehat {\partial }h(\bar {\mathbf {x}} ) = \partial h(\bar {\mathbf {x}} )= \left \{\nabla h(\bar {\mathbf {x}} )\right \}$;

• (ii) For any $\bar {\mathbf {x}} \in \text {dom} h$, the subdifferential set $\partial h(\bar {\mathbf {x}} )$ is closed, i.e.,

  $$\left\{\mathbf{v} \in\mathbb{R}^{\mathsf{N}}:\exists \mathbf{x}^{(k)} \to \bar{\mathbf{x}}, h\left( \mathbf{x}^{(k)}\right) \to h(\bar{\mathbf{x}}), \mathbf{v}^{(k)}\in\partial h\left( \mathbf{x}^{(k)}\right), \mathbf{v}^{(k)} \to \mathbf{v} \right\}\subset \partial h(\bar{\mathbf{x}} ). $$

The fundamental works on the Kurdyka-Łojasiewicz (KL) property are due to Łojasiewicz [34] and Kurdyka [29]. For the development of the applications of KL property in optimization theory, see [1, 2, 5, 7] and references therein.

Definition A.2 (Kurdyka-Łojasiewicz Property 1)

A proper function h is said to have the Kurdyka-Łojasiewicz property at $\bar {\mathbf {x}} \in \text {dom} \partial h = \{ \mathbf {x} \in \mathbb {R}^{\mathsf {N}}: \partial h(\mathbf {x}) \neq \emptyset \}$ if there exist ζ ∈ (0, +∞], a neighborhood U of $\bar {\mathbf {x}}$, and a continuous concave function $\varphi : [0, \zeta ) \to \mathbb {R}_{+}$ such that

• (i) φ(0) = 0;

• (ii) φ(0) is C¹ on (0,ζ);

• (iii) for all s ∈ (0,ζ), φ^′(s) > 0;

• (iv) for all x ∈ U satisfying $h(\bar {\mathbf {x}}) < h(\mathbf {x}) < h(\bar {\mathbf {x}}) + \zeta$, the Kurdyka-Łojasiewicz inequality holds:

  $$\varphi^{\prime}(h(\mathbf{x}) - h(\bar{\mathbf{x}})) \text{dist} (0,\partial h(\mathbf{x})) \geq 1. $$

  where dist(0,∂h(x)) = min{∥v∥ : v ∈ ∂h(x)},

A proper, lower semicontinuous function h satisfying the KL property at all points in dom∂h is called a KL function. Some examples can be found in [2].

A rich class of KL functions used widely in practice applications belongs to a special structure called o-minimal structure, which is introduced in [42]. The following definition is adopted from [1, Definition 4.1]

Definition A.3

Let $\mathcal {O}=\left \{\mathcal {O}_{n}\right \}_{n\in N}$ be such that each $\mathcal {O}_{n}$ is a collection of subsets of $\mathbb {R}^{n}$. The family $\mathcal {O}$ is an o-minimal structure over $\mathbb {R}$, if it satisfies the following axioms:

• (i) Each $\mathcal {O}_{n}$ is a boolean algebra. Namely $\emptyset \in \mathcal {O}_{n}$ and for each $A,B\in \mathcal {O}_{n}, A\cup B, A\cap B,$ and $\mathbb {R}^{n}\setminus A$ belong to $\mathcal {O}_{n}$.

• (ii) For all $A\in \mathcal {O}_{n}, A\times R$ and R × A belong to $\mathcal {O}_{n + 1}$.

• (iii) For all $A\in \mathcal {O}_{n + 1}, \prod (A):=\left \{(x_{1},...,x_{n})\in \mathbb {R}^{n} | (x_{1},...,x_{n},x_{n + 1})\in A\right \}$ belongs to $\mathcal {O}_{n}$.

• (iv) For all i≠j in {1, 2,...,n}, $\left \{(x_{1},...,x_{n})\in \mathbb {R}^{n} | x_{i}=x_{j}\right \}$ belong to $\mathcal {O}_{n}$.

• (v) The set $\left \{(x_{1},x_{2})\in \mathbb {R}^{2} | x_{1},x_{2}\right \}$ belong to $\mathcal {O}_{2}$.

• (vi) The elements of $\mathcal {O}_{1}$ are exactly finite unions of intervals.

Let $\mathcal {O}$ be an o-minimal structure on $\mathbb {R}$. We say that a set $A\subseteq \mathbb {R}^{n}$ is definable on $\mathcal {O}$ if $A\in \mathcal {O}^{n}$ and that a map $f : \mathbb {R}^{n}\rightarrow \mathbb {R}^{m}$ is definable on $\mathcal {O}$ if its graph $\left \{(x,y)\in \mathbb {R}^{n}\times \mathbb {R}^{m}: y \in f(x)\right \}$ is definable on $\mathcal {O}$. Definable functions can be defined like definable maps. The o-minimal structure has very nice properties. We list some known elementary properties of definable functions below [1]:

• (i) Finite sum of definable functions are definable.

• (ii) Compositions of definable functions are definable.

• (iii) Indicator functions of definable sets are definable.

A class of o-minimal structure is the log-exp structure [42, Example 2.5]. By this structure, the following function are all definable:

• 1. semi-algebraic function [7, Definition 5], such as real polynomial functions, and $f : \mathbb {R}\rightarrow \mathbb {R}$ defined by x↦|x|.

• 2. $x^{r}: \mathbb {R}\rightarrow \mathbb {R}$ defined by

  $$a \mapsto\left\{\begin{array}{lll} a^{r}, & a>0\\ 0, &a\leq 0. \end{array}\right. $$

  where $r\in \mathbb {R}$.

• 3. the exponential function: $\mathbb {R}\rightarrow \mathbb {R}$ given by x↦e^x and the logarithm function: $(0, \infty ) \rightarrow \mathbb {R}$ given by x↦ log(x).

It is known that any proper lower semicontinuous function that is definable on an o-minimal structure is a KL function. See [6]and [1, Theorem 4.1]. Combining the elementary properties (i)(ii) of definable functions and examples (1)(2), one can see that the function $\mathcal {E}$ in this paper is a KL function.