Regularized Non-local Total Variation and Application in Image Restoration

Abstract

In the usual non-local variational models, such as the non-local total variations, the image is regularized by minimizing an energy term that penalizes gray-levels discrepancy between some specified pairs of pixels; a weight value is computed between these two pixels to penalize their dissimilarity. In this paper, we impose some regularity to those weight values. More precisely, we minimize a function involving a regularization term, analogous to an $H^1$ term, on weights. Doing so, the finite differences defining the image regularity depend on their environment. When the weights are difficult to define, they can be restored by the proposed stable regularization scheme. We provide all the details necessary for the implementation of a PALM algorithm with proved convergence. We illustrate the ability of the model to restore relevant unknown edges from the neighboring edges on an image inpainting problem. We also argue on inpainting, zooming and denoising problems that the model better recovers thin structures.

Appendices

Appendices

1.1 TV Under a Max Form

We want to prove that for any $u\in {\mathbb {R}}^{{\mathcal {P}}}$ and any (fixed) $v\in {{{\mathcal {U}}}^{{\mathcal {P}}}}$

$$\begin{aligned} \mathrm{TV}\left( v, u\right) = \max _{\Vert w\Vert _{\infty ,2} \le 1} \left\langle \mathbf {D}_vu, w\right\rangle - \frac{\mu }{2} \Vert w\Vert ^2, \end{aligned}$$

(51)

where we remind that, for $w\in {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}$, the norm defining the constraint takes the form

$$\begin{aligned} \Vert w \Vert _{\infty ,2} = \max _{p\in {{\mathcal {P}}}} \Vert w_p\Vert , \quad \text{ for }\quad w_p=(w_{p,q})_{q\in {{\mathcal {B}}}} \in {\mathbb {R}}^{{\mathcal {B}}}. \end{aligned}$$

(52)

First, notice that if for all $p\in {{\mathcal {P}}}$ we know $w^*_p\in {\mathbb {R}}^{{\mathcal {B}}}$ such that

$$\begin{aligned} w^*_p = {{\mathrm{argmax}}}_{w\in {\mathbb {R}}^{{\mathcal {B}}}: \Vert w\Vert \le 1} \left\langle \mathbf {D}_vu_p, w \right\rangle - \frac{\mu }{2} \Vert w\Vert ^2, \end{aligned}$$

(53)

where we denote $\mathbf {D}_vu_p = (\mathbf {D}_vu_{p,q})_{q\in {{\mathcal {B}}}} \in {\mathbb {R}}^{{\mathcal {B}}}$; we can deduce from the optimality of all its components $w^*_p$ that $w^* = (w^*_p)_{p\in {{\mathcal {P}}}}\in {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}$ satisfies

$$\begin{aligned} w^* = {{\mathrm{argmax}}}_{w\in {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}:\Vert w\Vert _{\infty ,2} \le 1} \left\langle \mathbf {D}_vu, w\right\rangle - \frac{\mu }{2} \Vert w\Vert ^2. \end{aligned}$$

(54)

In order to calculate $w^*_p$, for a given $p\in {{\mathcal {P}}}$, we first remark that there exists $\alpha _p^*\ge 0$ such that

$$\begin{aligned} w^*_p = \alpha _p^* \mathbf {D}_vu_p. \end{aligned}$$

(55)

Then, if we use this expression in (53) we have that

$$\begin{aligned} \alpha _p^*= & {} {{\mathrm{argmax}}}_{\alpha \Vert \mathbf {D}_vu_p\Vert \le 1} \alpha \Vert \mathbf {D}_vu_p\Vert ^2 - \frac{\mu }{2} \alpha ^2 \Vert \mathbf {D}_vu_p\Vert ^2, \\= & {} {{\mathrm{argmax}}}_{\alpha \Vert \mathbf {D}_vu_p\Vert \le 1} ~\alpha \left( 1-\frac{\mu \alpha }{2}\right) ,\\= & {} \left\{ \begin{array}{ll} \frac{1}{ \Vert \mathbf {D}_vu_p\Vert }, &{} \text{ if } \frac{1}{\mu } >\frac{1}{ \Vert \mathbf {D}_vu_p\Vert }, \\ \frac{1}{\mu },&{} \text{ otherwise. } \end{array}\right. \end{aligned}$$

We finally obtain that

$$\begin{aligned} w^*_p = \left\{ \begin{array}{ll} \frac{\mathbf {D}_vu_p}{\mu }, &{} \text{ if } \Vert (\mathbf {D}_vu)_p\Vert \le \mu ,\\ \frac{\mathbf {D}_vu_p}{\Vert (\mathbf {D}_vu)_p\Vert }, &{} \text{ otherwise. } \end{array}\right. \end{aligned}$$

(56)

This corresponds to the expression of $w^*(u)_p$ in Proposition 1.

If we now use the expression for $w^*_p$ to calculate the objective function, we find that

$$\begin{aligned} \begin{aligned}&\max _{w\in {\mathbb {R}}^{{\mathcal {B}}}: \Vert w\Vert \le 1} \left\langle \mathbf {D}_vu_p, w \right\rangle - \frac{\mu }{2} \Vert w\Vert ^2\\&\quad = \left\langle \mathbf {D}_vu_p, w^*_p\right\rangle - \frac{\mu }{2} \Vert w^*_p\Vert ^2\\&\quad = \left\{ \begin{array}{ll} \frac{\Vert \mathbf {D}_vu_p\Vert ^2}{2\mu }, &{} \text{ if } \Vert \mathbf {D}_vu_p\Vert \le \mu ,\\ \Vert \mathbf {D}_vu_p\Vert -\frac{\mu }{2}, &{} \text{ otherwise. } \end{array}\right. \\&\quad = \varPsi _\mu (\Vert \mathbf {D}_vu_p \Vert ). \end{aligned} \end{aligned}$$

(57)

As a consequence,

$$\begin{aligned} \begin{aligned}&\max _{\Vert w\Vert _{\infty ,2} \le 1} \left\langle \mathbf {D}_vu, w\right\rangle - \frac{\mu }{2} \Vert w\Vert ^2\\&\quad = \sum _{p\in {{\mathcal {P}}}} \left\langle \mathbf {D}_vu_p, w^*_p\right\rangle - \frac{\mu }{2} \Vert w^*_p\Vert ^2\\&\quad = TV(v,u). \end{aligned} \end{aligned}$$

(58)

1.2 Proof of Proposition 1

Let us first remind and adapt to the context of this paper theorem 1 stated in [33]. This result considers finite-dimensional real vector spaces $V_1$ and $V_2$, a linear operator $A:V_1\rightarrow V_2$, a parameter $\mu >0$ and a function

$$\begin{aligned} f_\mu (x) = \max _{u\in Q_2} \left\langle Ax, u\right\rangle - \frac{\mu }{2} \Vert u\Vert ^2,\qquad \forall x\in V_1 \end{aligned}$$

(59)

where $Q_2\subset V_2$ is a closed convex bounded set.

It is stated and proved in [33] that function $f_\mu$ is continuously differentiable at any $x\in V_1$. Moreover, if we denote $u_\mu (x)$ the unique solution of the maximization problem defining $f_\mu$, we have:

$$\begin{aligned} \nabla f_\mu (x) = A^* u_\mu (x), \end{aligned}$$

(60)

where $A^*$ is the adjoint of A. Moreover, $x\longmapsto \ f_\mu (x)$ is Lipschitz continuous with constant

$$\begin{aligned} \frac{\Vert A\Vert _{V_1\rightarrow V_2}}{\mu }, \end{aligned}$$

(61)

where

$$\begin{aligned} \Vert A\Vert _{V_1\rightarrow V_2} = \max _{\Vert x\Vert \le 1} \Vert Ax\Vert . \end{aligned}$$

(62)

Proposition 1 is a straightforward application of this statement to the function

$$\begin{aligned} TV(v,u)=\max _{\Vert w\Vert _{\infty ,2} \le 1} \left\langle \mathbf {D}_vu, w\right\rangle - \frac{\mu }{2} \Vert w\Vert ^2. \end{aligned}$$

(63)

In order to compute the Lipschitz constant, we need however to find a bound for $\Vert \mathbf {D}_v\Vert _{{\mathbb {R}}^{{\mathcal {P}}}\rightarrow {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}}$. In order to compute it, we consider $u\in {\mathbb {R}}^{{\mathcal {P}}}$. We have for any $v\in {{\mathcal {U}}}^{{\mathcal {P}}}$

$$\begin{aligned} \begin{aligned} \Vert \mathbf {D}_vu\Vert ^2&= \sum _{p\in {{\mathcal {P}}}}\sum _{q\in {{\mathcal {B}}}} v^p_q (u_p - u_{p+q})^2 \\&\le 2 \sum _{p\in {{\mathcal {P}}}}\sum _{q\in {{\mathcal {B}}}} v^p_q \left( u_p^2 + u_{p+q}^2\right) \\&\le 2 \sum _{p\in {{\mathcal {P}}}}u_p^2 + 2 \sum _{q\in {{\mathcal {B}}}} \sum _{p\in {{\mathcal {P}}}}u_{p+q}^2 \\&\le 2 \Vert u\Vert ^2 + 2 |{{\mathcal {B}}}| \Vert u\Vert ^2, \end{aligned} \end{aligned}$$

(64)

where $|{{\mathcal {B}}}|$ denotes the cardinality of ${{\mathcal {B}}}$. We then deduce⁹ that for any $v\in {{\mathcal {U}}}^{{\mathcal {P}}}$

$$\begin{aligned} \Vert \mathbf {D}_v\Vert _{{\mathbb {R}}^{{\mathcal {P}}}\rightarrow {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}} \le \sqrt{2} \sqrt{|{{\mathcal {B}}}| +1}. \end{aligned}$$

(65)

Finally, it is standard that (32) is a consequence of the fact that $l''=\frac{\sqrt{2} \sqrt{|{{\mathcal {B}}}| +1}}{\mu }$ is a Lipschitz bound for $u\longmapsto \nabla _u TV(v,u)$.

1.3 Proof of Proposition 2

Considering $v\in {{\mathcal {U}}}^{{\mathcal {P}}}, u \in {\mathbb {R}}^{{\mathcal {P}}}$ and a small variation $h\in {\mathbb {R}}^{{{\mathcal {P}}}\times {{\mathcal {B}}}}$, we denote ${{\mathcal {P}}}_1= \{p\in {{\mathcal {P}}}| (\mathbf {A}_uv)_p \ge \frac{\mu }{2} \}$ and ${{\mathcal {P}}}_2 = {{\mathcal {P}}}\setminus {{\mathcal {P}}}_1$. For h small enough, we have

$$\begin{aligned} TV(v+h,u)= & {} \sum _{p\in {{\mathcal {P}}}_1} \varPsi _\mu \left( \sqrt{(\mathbf {A}_uv)_p + (\mathbf {A}_uh)_p} \right) \nonumber \\&+\sum _{p\in {{\mathcal {P}}}_2} \frac{ (\mathbf {A}_uv)_p + (\mathbf {A}_uh)_p }{ 2\mu }. \end{aligned}$$

(66)

Moreover,

$$\begin{aligned} \begin{aligned}&\sum _{p\in {{\mathcal {P}}}_1} \varPsi _\mu \left( \sqrt{(\mathbf {A}_uv)_p + (\mathbf {A}_uh)_p} \right) \\&\quad = \sum _{p\in {{\mathcal {P}}}_1} \varPsi _\mu \left( \sqrt{(\mathbf {A}_uv)_p}+ \frac{(\mathbf {A}_uh)_p}{2\sqrt{(\mathbf {A}_uv)_p}} +o\left( |(\mathbf {A}_uh)_p|\right) \right) \\&\quad = \sum _{p\in {{\mathcal {P}}}_1} \varPsi _\mu \left( \sqrt{(\mathbf {A}_uv)_p}\right) \\&\qquad + \varPsi '_\mu \left( \sqrt{(\mathbf {A}_uv)_p} \right) \frac{(\mathbf {A}_uh)_p}{2\sqrt{(\mathbf {A}_uv)_p}} +o\left( |(\mathbf {A}_uh)_p|\right) \\ \end{aligned} \end{aligned}$$

(67)

Denoting for all $p\in {{\mathcal {P}}}_1$

$$\begin{aligned} (u^*(v))_p = \frac{\varPsi '_\mu \left( \sqrt{(\mathbf {A}_uv)_p} \right) }{2 \sqrt{(\mathbf {A}_uv)_p}} \end{aligned}$$

(68)

and for all $p\in {{\mathcal {P}}}_2$

$$\begin{aligned} (u^*(v))_p = \frac{1}{2 \mu } \end{aligned}$$

(69)

and using the simple closed-form expression for the derivative $\varPsi '_\mu$, we get for all $p\in {{\mathcal {P}}}$

$$\begin{aligned} (u^*(v))_p = \left\{ \begin{array}{ll} \frac{1}{2 \sqrt{(\mathbf {A}_uv)_p}}, &{} \text{ if } \sqrt{(\mathbf {A}_uv)_p} \ge \mu ,\\ \frac{1}{2\mu },&{} \text{ if } \mu \ge \sqrt{(\mathbf {A}_uv)_p}\ge 0. \end{array}\right. \end{aligned}$$

(70)

Using this notation in the previous calculations we obtain that

$$\begin{aligned} TV(v+h,u) = TV(v,u) +\left\langle u^*(v), \mathbf {A}_uh\right\rangle + o(\Vert h\Vert ). \end{aligned}$$

(71)

We finally conclude that

$$\begin{aligned} \nabla _v TV(v,u) = \mathbf {A}_u^* u^*(v). \end{aligned}$$

(72)

This proves the first part of Proposition 2.

Let us now show that $v\longmapsto TV(v,u)$ is concave over ${\mathbb {R}}_+^{{{\mathcal {P}}}\times {{\mathcal {B}}}}$. In order to do so, we rewrite the latter formula under the form $(u^*(v))_p = \varphi ((\mathbf {A}_uv)_p)$ where the function $\varphi$ is defined for all $t\ge 0$ by

$$\begin{aligned} \varphi (t) = \left\{ \begin{array}{ll} \frac{1}{2\sqrt{t}}, &{} \text{ if } \sqrt{t} \ge \mu , \\ \frac{1}{2\mu },&{} \text{ if } \mu \ge \sqrt{t}\ge 0. \end{array}\right. \end{aligned}$$

(73)

Notice that function $\varphi$ is non-increasing and therefore

$$\begin{aligned} (\varphi (t_1) - \varphi (t_2) ) (t_1-t_2) \le 0,\ \forall (t_1,t_2)\in {\mathbb {R}}^2. \end{aligned}$$

(74)

We now consider v and $v'\in {\mathbb {R}}_+^{{{\mathcal {P}}}\times {{\mathcal {B}}}}$ and $u\in {\mathbb {R}}^{{\mathcal {P}}}$. Using Taylor’s theorem, we know there exists $t\in [0,1]$ such that $v''=tv'+(1-t)v$ satisfies

$$\begin{aligned} \begin{aligned}&TV(v',u)- TV(v,u)-\left\langle \nabla _vTV(v,u), v'-v\right\rangle \\&\quad = \left\langle \nabla _vTV(v'',u)-\nabla _vTV(v,u), v'-v\right\rangle \\&\quad = \left\langle u^*(v'')-u^*(v), \mathbf {A}_u(v'-v)\right\rangle \\&\quad = \sum _{p\in {{\mathcal {P}}}} \left( \varphi (\mathbf {A}_uv''_p) - \varphi (\mathbf {A}_uv_p) \right) \left( \mathbf {A}_uv'_p - \mathbf {A}_uv_p \right) . \end{aligned} \end{aligned}$$

(75)

However, for any $p\in {{\mathcal {P}}}, \mathbf {A}_uv''_p - \mathbf {A}_uv_p = t (\mathbf {A}_uv'_p- \mathbf {A}_uv_p)$ and $\mathbf {A}_uv''_p - \mathbf {A}_uv_p$ and $\mathbf {A}_uv'_p- \mathbf {A}_uv_p$ have the same sign. Using (74), we then get

$$\begin{aligned} \left( \varphi (\mathbf {A}_uv''_p) - \varphi (\mathbf {A}_uv_p) \right) \left( \mathbf {A}_uv'_p - \mathbf {A}_uv_p \right) \le 0 \end{aligned}$$

(76)

and finally

$$\begin{aligned} TV(v',u)- TV(v,u)-\left\langle \nabla _vTV(v,u), v'-v\right\rangle \le 0. \end{aligned}$$

(77)

This concludes the proof.

1.4 Proof of Proposition 3

For any $v\in {{{\mathcal {U}}}^{{\mathcal {P}}}}$, given the expression

$$\begin{aligned} R(v) = \gamma ~\Vert Bv\Vert ^2, \end{aligned}$$

(78)

we immediately have

$$\begin{aligned} \nabla R (v) = 2 \gamma B^*B v. \end{aligned}$$

(79)

We only need to calculate the Lipschitz constant $l'$ provided in Proposition 3. In order to do so, we consider v and $v'\in {{{\mathcal {U}}}^{{\mathcal {P}}}}$ and denote $w=v'-v$. We have

$$\begin{aligned} \Vert \nabla R(v') - \nabla R(v)\Vert ^2 = 4\gamma ^2 \Vert B^*Bw\Vert ^2. \end{aligned}$$

(80)

Moreover, using the formula for $B^*$, we get

$$\begin{aligned}&\Vert B^*Bw\Vert ^2 \nonumber \\&\,\, =\sum _{(p,q)\in {{{\mathcal {P}}}\times {{\mathcal {B}}}}} \left| \sum _{p'\in {{\mathcal {N}}}} ((Bw)_{p,q,p'} - (Bw)_{p-p',q,p'})\right| ^2. \end{aligned}$$

(81)

The term inside the absolute value can be rewritten, using the definition of B, under the form

$$\begin{aligned} \begin{aligned}&\sum _{p'\in {{\mathcal {N}}}} ((Bw)_{p,q,p'} - (Bw)_{p-p',q,p'}) \\&\quad = \sum _{p'\in {{\mathcal {N}}}} (2w_{p,q} - w_{p+p',q}- w_{p-p',q}),\\&\quad = 2|{{\mathcal {N}}}| w_{p,q} - \sum _{p'\in {{\mathcal {N}}}}w_{p+p',q}- \sum _{p'\in {{\mathcal {N}}}}w_{p-p',q}. \end{aligned} \end{aligned}$$

(82)

Therefore,

$$\begin{aligned} \begin{aligned}&\Vert B^*Bw\Vert ^2 \\&\quad \le \sum _{(p,q)\in {{{\mathcal {P}}}\times {{\mathcal {B}}}}}3\left( 4|{{\mathcal {N}}}|^{2}w_{p,q}^{2}+\left( \sum _{p'\in {{\mathcal {N}}}}w_{p+p',q}\right) ^{2}\right. \\&\qquad \left. +\left( \sum _{p'\in {{\mathcal {N}}}}w_{p-p',q}\right) ^{2}\right) \\&\quad \le 3\left( 4|{{\mathcal {N}}}|^{2}\Vert w\Vert ^{2}+|{{\mathcal {N}}}|\sum _{(p,q)\in {{{\mathcal {P}}}\times {{\mathcal {B}}}}}\sum _{p'\in {{\mathcal {N}}}}w_{p+p',q}^{2}\right. \\&\qquad \left. +\,\,\,|{{\mathcal {N}}}|\sum _{(p,q)\in {{{\mathcal {P}}}\times {{\mathcal {B}}}}}\sum _{p'\in {{\mathcal {N}}}}w_{p-p',q}^{2}\right) \\&\quad \le 3\times 6 |{{\mathcal {N}}}|^2\Vert w\Vert ^2. \end{aligned} \end{aligned}$$

(83)

Finally,

$$\begin{aligned} \Vert \nabla R(v') - \nabla R(v)\Vert ^2 \le 2\times 6^2 \gamma ^2 |{{\mathcal {N}}}|^2\Vert v'-v\Vert ^2. \end{aligned}$$

(84)

We then conclude that $v\longmapsto \nabla R(v)$ is Lipschitz with Lipschitz constant $6\sqrt{2} \gamma |{{\mathcal {N}}}|$. This concludes the proof.