Convergence Rate of Proximal Inertial Algorithms Associated with Moreau Envelopes of Convex Functions

Abstract

In a Hilbert space setting ${\mathcal H}$, we develop new inertial proximal-based algorithms that aim to rapidly minimize a convex lower-semicontinuous proper function $\varPhi : \mathcal H \rightarrow {\mathbb R} \cup \{+\infty \}$. The guiding idea is to use an accelerated proximal scheme where, at each step, Φ is replaced by its Moreau envelope, with varying approximation parameter. This leads to consider a Relaxed Inertial Proximal Algorithm (RIPA) with variable parameters which take into account the effects of inertia, relaxation, and approximation. (RIPA) was first introduced to solve general maximally monotone inclusions, in which case a judicious adjustment of the parameters makes it possible to obtain the convergence of the iterates towards the equilibria. In the case of convex minimization problems, convergence analysis of (RIPA) was initially addressed by Attouch and Cabot, based on its formulation as an inertial gradient method with varying potential functions. We propose a new approach to this algorithm, along with further developments, based on its formulation as a proximal algorithm associated with varying Moreau envelopes. For convenient choices of the parameters, we show the fast optimization property of the function values, with the order o(k ⁻²), and the weak convergence of the iterates. This is in line with the recent studies of Su-Boyd-Candès, Chambolle-Dossal, Attouch-Peypouquet. We study the impact of geometric assumptions on the convergence rates, and the stability of the results with respect to perturbations and errors. Finally, in the case of structured minimization problems smooth + nonsmooth, based on this approach, we introduce new proximal-gradient inertial algorithms for which similar convergence rates are shown.

Appendix

1.1.1 Some Properties of the Moreau Envelope

For a detailed presentation of the Moreau envelope, we refer the reader to [20, 25, 35, 36]. We merely point out the following properties, of constant use here:

• (i) the function λ ∈ ]0, +∞[↦Φ _λ(x) is nonincreasing for each $x\in {\mathcal H}$;

• (ii) the equality $\inf _{\mathcal H}\varPhi =\inf _{\mathcal H}\varPhi _\lambda$ holds in ${\mathbb R}\cup \{-\infty \}$ for all λ > 0;

• (iii) $\operatorname *{\mbox{arg min}}\varPhi = \operatorname *{\mbox{arg min}}\varPhi _\lambda$ for all λ > 0.

It will be convenient to consider the Moreau envelope as a function of the two variables $x \in {\mathcal H}$ and λ ∈ ]0, +∞[. Its differentiability with respect to (x, λ) plays a crucial role in our analysis.

1.1.1.1 a

Let us first recall some classical facts concerning the differentiability of the function x↦Φ _λ(x) for fixed λ > 0. The infimum in (1.2) is attained at a unique point

$$\displaystyle \begin{aligned} \mbox{prox}_{\lambda\varPhi}(x) = \mbox{{argmin}}_{ \xi \in \mathcal H}\left\lbrace \varPhi (\xi) + \frac{1}{2\lambda}\| x -\xi \|{}^2\right\rbrace, \end{aligned} $$

(1.68)

which gives

$$\displaystyle \begin{aligned} \varPhi_{\lambda}(x) = \varPhi (\mbox{prox}_{\lambda\varPhi} (x)) + \frac{1}{2\lambda} \|x - \mbox{prox}_{\lambda\varPhi}(x) \|{}^2 . \end{aligned} $$

(1.69)

Writing the optimality condition for (1.68), we get $\mbox{prox}_{\lambda \varPhi }(x) + \lambda \partial \varPhi \left ( \mbox{prox}_{\lambda \varPhi }(x) \right ) \ni x,$ that is

$$\displaystyle \begin{aligned}\mbox{prox}_{\lambda\varPhi}(x) = \left( I + \lambda\partial \varPhi \right)^{-1} (x). \end{aligned}$$

Thus, prox_λΦ is the resolvent of index λ > 0 of the maximal monotone operator ∂Φ. As a consequence, the mapping $\mbox{prox}_{\lambda \varPsi }: {\mathcal H} \to {\mathcal H}$ is firmly nonexpansive. The function x↦Φ _λ(x) is continuously differentiable, with

$$\displaystyle \begin{aligned} \nabla \varPhi_{\lambda}(x) = \frac{1}{\lambda} \left( x- \mbox{prox}_{\lambda\varPhi}(x) \right) . \end{aligned} $$

(1.70)

Equivalently

$$\displaystyle \begin{aligned} \nabla \varPhi_{\lambda} = \frac{1}{\lambda} \left( I- \left( I + \lambda\partial \varPhi \right)^{-1} \right)= \left( \partial \varPhi \right)_{\lambda} \end{aligned} $$

(1.71)

which is the Yosida approximation of the maximal monotone operator ∂Φ. As such, ∇Φ _λ is Lipschitz continuous, with Lipschitz constant $\frac {1}{\lambda }$, and $\varPhi _{\lambda } \in \mathcal C^{1,1}$.

1.1.1.2 b

A less known result is the ${\mathcal C}^1$-regularity of the function λ↦Φ _λ(x), for each $x \in {\mathcal H}$. Its derivative is given by

$$\displaystyle \begin{aligned} \frac{d}{d\lambda}\varPhi_\lambda(x)=-\frac{1}{2}\|\nabla\varPhi_{\lambda}(x)\|{}^2 . \end{aligned} $$

(1.72)

This result is known as the Lax-Hopf formula for the above first-order Hamilton-Jacobi equation, see [4, Remark 3.2; Lemma 3.27], [8, Lemma A.1], and [29].

Lemma 1.7

For each $x\in {\mathcal H}$ , the real-valued function λ↦Φ _λ(x) is continuously differentiable on ]0, +∞[, with

$$\displaystyle \begin{aligned} \frac{d}{d\lambda}\varPhi_\lambda(x)=-\frac{1}{2}\|\nabla\varPhi_{\lambda}(x)\|{}^2. \end{aligned} $$

(1.73)

As a consequence, for any $x \in {\mathcal H}$, λ > 0 and μ > 0,

$$\displaystyle \begin{aligned} (\varPhi_\lambda)_{\mu}(x)= \varPhi_{(\lambda +\mu)}(x). \end{aligned} $$

(1.74)

Indeed, (1.74) is the semi-group property satisfied by the orbits of the autonomous evolution equation (1.72). Differentiating (1.74) with respect to x, and using (1.71) gives the classical resolvent equation

$$\displaystyle \begin{aligned} (A_\lambda)_{\mu}= A_{(\lambda +\mu)}, \end{aligned} $$

(1.75)

where A = ∂Φ. Indeed, (1.75) is valid for a general maximally monotone operator A, see, for example, [20, Proposition 23.6] or [25, Proposition 2.6].

1.1.2 Auxiliary Results

Theorem 1.6.2

Let λ : [t ₀, +∞[→]0, +∞[ be continuous and nondecreasing. Let $\varPhi :{\mathcal H}\to \mathbb R\cup \{+\infty \}$ be convex, lower-semicontinuous, and proper. Then, given any x ₀ and v ₀ in ${\mathcal H}$ , system (1.1) has a unique twice continuously differentiable global solution $x:[t_0,+\infty [\to {\mathcal H}$ verifying x(t ₀) = x ₀ , $\dot x(t_0)=v_0$.

Proof

The assertion appeals to the most elementary form of the Cauchy-Lipschitz theorem (see any textbook) and hinges on the (t, x)-continuity of ∇Φ _λ and on its Lipschitz continuity with respect to x, uniform with respect to t.

Indeed, for t ∈ [t ₀, +∞[ and $(x,x')\in {\mathcal H}\times {\mathcal H}$ we have

$$\displaystyle \begin{aligned} \|\nabla\varPhi_{\lambda(t)}(x')-\nabla\varPhi_{\lambda(t)}(x)\|\leq \frac{1}{\lambda(t)}\|x'-x\|\leq \frac{1}{\lambda(t_0)}\|x'-x\|. \end{aligned}$$

Next, the continuity of $\nabla \varPhi _{\lambda (t)}(x)=\frac {1}{\lambda (t)}(x-\mbox{prox}_{\lambda (t)\varPhi }x)$ boils down to the continuity of the mapping $(t,x)\in [t_0,+\infty [\times {\mathcal H}\to \mbox{prox}_{\lambda (t)\varPhi }x\in {\mathcal H}$. For (t, x) and (t′, x′) in $[t_0,+\infty [\times {\mathcal H}$ we have

$$\displaystyle \begin{aligned} \|\mbox{prox}_{\lambda(t')\varPhi}x'-\mbox{prox}_{\lambda(t)\varPhi}x\|\leq \|\mbox{prox}_{\lambda(t')\varPhi}x'-\mbox{prox}_{\lambda(t')\varPhi}x\|+\|\mbox{prox}_{\lambda(t')\varPhi}x-\mbox{prox}_{\lambda(t)\varPhi}x\|. \end{aligned}$$

But, since prox_λΦ is nonexpansive

$$\displaystyle \begin{aligned} \|\mbox{prox}_{\lambda(t')\varPhi}x'-\mbox{prox}_{\lambda(t')\varPhi}x\|\leq\|x'-x\|; \end{aligned}$$

and also (see [20, Prop. 23.28(iii)])

$$\displaystyle \begin{aligned} \|\mbox{prox}_{\lambda(t')\varPhi}x-\mbox{prox}_{\lambda(t)\varPhi}x\|\leq \left|\frac{\lambda(t')}{\lambda(t)}-1\right|\|\mbox{prox}_{\lambda(t)\varPhi}x-x\|. \end{aligned}$$

Therefore

$$\displaystyle \begin{aligned} \|\mbox{prox}_{\lambda(t')\varPhi}x'-\mbox{prox}_{\lambda(t)\varPhi}x\|\leq \|x'-x\|+|\lambda(t')-\lambda(t)|\|\nabla\varPhi_{\lambda(t)}(x)\|, \end{aligned}$$

which proves the continuity of prox_λΦ at point (t, x).

Let us state the discrete version of Opial’s lemma.

Lemma 1.8

Let S be a nonempty subset of $\mathcal H$ , and (x _k) a sequence of elements of $\mathcal H$ . Assume that

• (i) for every z ∈ S, lim_k→+∞∥x _k − z∥ exists;

• (ii) every weak sequential cluster point of (x _k), as k →∞, belongs to S.

Then x _k converges weakly as k →∞ to a point in S.

We shall also make use of the following discrete version of the Gronwall lemma:

Lemma 1.9

Let (a _k) be a sequence of nonnegative numbers such that, for all $k\in \mathbb N$

$$\displaystyle \begin{aligned}a_k^2 \leq c^2 + \sum_{j=1}^k \beta_j a_j,\end{aligned} $$

where (β _j) is a summable sequence of nonnegative numbers, and c ≥ 0. Then, $\displaystyle a_k \leq c + \sum _{j=1}^{\infty } \beta _j$ for all $k\in \mathbb N$.

Proof

For $k\in \mathbb N$, set A _k :=max_1≤m≤k a _m. Then, for 1 ≤ m ≤ k, we have

$$\displaystyle \begin{aligned}a_m^2 \leq c^2 + \sum_{j=1}^m \beta_j a_j \leq c^2 + A_k \sum_{j=1}^{\infty} \beta_j.\end{aligned} $$

Taking the maximum over 1 ≤ m ≤ k, we obtain

$$\displaystyle \begin{aligned}A_k^2 \leq c^2 + A_k \sum_{j=1}^{\infty} \beta_j. \end{aligned}$$

Bounding by the roots of the corresponding quadratic equation, we obtain the result.

The next lemma provides an estimate of the convergence rate of a sequence that is summable with respect to weights.

Lemma 1.10 ([7, Lemma 22])

Let (τ _k) be a nonnegative sequence such that $\sum _{k=1}^{+\infty } \tau _{k}=+\infty$ . Assume that (𝜖 _k) is a nonnegative and nonincreasing sequence satisfying $\sum _{k=1}^{+\infty } \tau _{k}\,\epsilon _k<+\infty$ . Then we have $\epsilon _k=o\left (\frac {1}{\sum _{i=1}^k \tau _i}\right ) \quad \mathit{\mbox{as }} k\to +\infty .$