A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems

Abstract

We consider a class of nonconvex nonsmooth optimization problems whose objective is the sum of a smooth function and a finite number of nonnegative proper closed possibly nonsmooth functions (whose proximal mappings are easy to compute), some of which are further composed with linear maps. This kind of problems arises naturally in various applications when different regularizers are introduced for inducing simultaneous structures in the solutions. Solving these problems, however, can be challenging because of the coupled nonsmooth functions: the corresponding proximal mapping can be hard to compute so that standard first-order methods such as the proximal gradient algorithm cannot be applied efficiently. In this paper, we propose a successive difference-of-convex approximation method for solving this kind of problems. In this algorithm, we approximate the nonsmooth functions by their Moreau envelopes in each iteration. Making use of the simple observation that Moreau envelopes of nonnegative proper closed functions are continuous difference-of-convex functions, we can then approximately minimize the approximation function by first-order methods with suitable majorization techniques. These first-order methods can be implemented efficiently thanks to the fact that the proximal mapping of each nonsmooth function is easy to compute. Under suitable assumptions, we prove that the sequence generated by our method is bounded and any accumulation point is a stationary point of the objective. We also discuss how our method can be applied to concrete applications such as nonconvex fused regularized optimization problems and simultaneously structured matrix optimization problems, and illustrate the performance numerically for these two specific applications.

A Convergence of an NPG method with majorization

In this appendix, we consider the following optimization problem:

$$\begin{aligned} \min \limits _{{\varvec{x}}} ~~ F({\varvec{x}}) = h({\varvec{x}}) + P({\varvec{x}}) - g({\varvec{x}}), \end{aligned}$$

(44)

where h is an $L_h$-smooth function, P is a proper closed function with $\inf P > -\infty$ and g is a continuous convex function. We assume in addition that there exists ${\varvec{x}}^0\in \mathrm{dom}\,P$ so that F is continuous in ${\varOmega }({\varvec{x}}^0):= \{{\varvec{x}}:\; F({\varvec{x}})\le F({\varvec{x}}^0)\}$ and the set ${\varOmega }({\varvec{x}}^0)$ is compact. As a consequence, it holds that $\inf F > -\infty$.

In Algorithm 2 below, we describe an algorithm, the nonmonotone proximal gradient method with majorization ($\hbox {NPG}_{\mathrm{major}}$), for solving (44). We first show that the line-search criterion is well-defined.

Proposition 1

For each t, the condition (46) is satisfied after at most

$$\begin{aligned} {\tilde{n}} := \max \left\{ \left\lceil \frac{\log (L_h + c) - \log (L_{\min })}{\log \tau }\right\rceil ,1\right\} \end{aligned}$$

inner iterations, which is independent of t. Consequently, $\{{\bar{L}}_t\}$ is bounded.

Proof

For each t and $L > 0$, let ${\varvec{u}}^t_L$ be an arbitrarily fixed element in

$$\begin{aligned} \mathop {\hbox {Argmin}}\limits _{{\varvec{x}}} \left\{ (\nabla h({\varvec{x}}^t) - {\varvec{\zeta }}^t)^\top ({\varvec{x}}-{\varvec{x}}^t) + \frac{L}{2}\Vert {\varvec{x}}-{\varvec{x}}^t\Vert ^2 + P({\varvec{x}})\right\} . \end{aligned}$$

Then we have

$$\begin{aligned} \begin{aligned} F({\varvec{u}}^t_L)&\le h({\varvec{x}}^t) + \nabla h({\varvec{x}}^t)^\top ({\varvec{u}}^t_L - {\varvec{x}}^t) + \frac{L_h}{2}\Vert {\varvec{u}}^t_L - {\varvec{x}}^t\Vert ^2 + P({\varvec{u}}^t_L) - g({\varvec{x}}^t) \\&\qquad - {{\varvec{\zeta }}^t}^\top ({\varvec{u}}^t_L - {\varvec{x}}^t)\\&= h({\varvec{x}}^t) - g({\varvec{x}}^t) + (\nabla h({\varvec{x}}^t)- {{\varvec{\zeta }}^t})^\top ({\varvec{u}}^t_L - {\varvec{x}}^t) + \frac{L_h}{2}\Vert {\varvec{u}}^t_L - {\varvec{x}}^t\Vert ^2 + P({\varvec{u}}^t_L)\\&\le F({\varvec{x}}^t) + \frac{L_h-L}{2}\Vert {\varvec{u}}^t_L - {\varvec{x}}^t\Vert ^2, \end{aligned} \end{aligned}$$

where the first inequality holds because of the $L_h$-smoothness of h, the convexity of g and the fact that ${\varvec{\zeta }}^t\in \partial g({\varvec{x}}^t)$, and the last inequality follows from the definition of ${\varvec{u}}^t_L$ as a minimizer. Thus, at the t-th iteration, the criterion (46) is satisfied by ${\varvec{u}} = {\varvec{u}}^t_L$ whenever $L \ge L_h + c$. Since we have

$$\begin{aligned} \tau ^{{\tilde{n}}} L^0_t \ge \tau ^{{\tilde{n}}} L_{\min } \ge L_h+c, \end{aligned}$$

we conclude that (46) must be satisfied at or before the ${\tilde{n}}$-th inner iteration. Consequently, we have ${\bar{L}}_t \le \tau ^{{\tilde{n}}}L_{\max }$ for all t. $\square$

The convergence of $\hbox {NPG}_{\mathrm{major}}$ can now be proved similarly as in [31, Lemma 4].

Proposition 2

Let $\{{\varvec{x}}^t\}$ be the sequence generated by $\hbox {NPG}_{\mathrm{major}}$. Then $\Vert {\varvec{x}}^{t+1} - {\varvec{x}}^t\Vert \rightarrow 0$.