Optimization of structural similarity in mathematical imaging

Abstract

It is now generally accepted that Euclidean-based metrics may not always adequately represent the subjective judgement of a human observer. As a result, many image processing methodologies have been recently extended to take advantage of alternative visual quality measures, the most prominent of which is the Structural Similarity Index Measure (SSIM). The superiority of the latter over Euclidean-based metrics have been demonstrated in several studies. However, being focused on specific applications, the findings of such studies often lack generality which, if otherwise acknowledged, could have provided a useful guidance for further development of SSIM-based image processing algorithms. Accordingly, instead of focusing on a particular image processing task, in this paper, we introduce a general framework that encompasses a wide range of imaging applications in which the SSIM can be employed as a fidelity measure. Subsequently, we show how the framework can be used to cast some standard as well as original imaging tasks into optimization problems, followed by a discussion of a number of novel numerical strategies for their solution.

Appendix

Proof of Theorem 4

Let $f:X\subset {\mathbb {R}}^n\rightarrow {\mathbb {R}}$ be defined as in Eq. (23). Then, its Jacobian is Lipschitz continuous on any open convex set $\Omega \subset X$; that is, there exists a constant $L>0$ such that for any $x,w\in \Omega$,

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le L\Vert x-w\Vert _2 \, . \end{aligned}$$

(95)

Here, $\Vert \cdot \Vert _F$ denotes the Frobenius norm, and

$$\begin{aligned} L = C_1\Vert \Phi ^T\Phi \Vert _F+\lambda (C_2\Vert z\Vert _2+C_3),\,C_1,C_2,C_3>0. \end{aligned}$$

(96)

$\square$

Proof

Without loss of generality, and for the sake of simplicity, let the stability constant C of the dissimilarity measure be zero. Also, let y be a non-zero vector in ${\mathbb {R}}^m$. Let us define

$$\begin{aligned} s(x):=\frac{2x^Ty}{\Vert Dx\Vert _2^2+\Vert y\Vert _2^2}, \end{aligned}$$

(97)

and

$$\begin{aligned} r(x):=\Vert Dx\Vert _2^2+\Vert y\Vert _2^2. \end{aligned}$$

(98)

Therefore, we have that $\Vert J_f(x)-J_f(w)\Vert _F$ is bounded by

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le & {} \Vert \Phi ^T\Phi \Vert _F\Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F+\nonumber \\&\lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F+\nonumber \\&\lambda \Vert x\nabla r(x)^T-w\nabla r(w)^T\Vert _F+\nonumber \\&|s(x)-s(w)|\Vert \Phi ^T\Phi \Vert _F+\lambda |r(x)-r(w)|. \end{aligned}$$

(99)

To show that $J_f$ is Lipschitz continuous on $\Omega$, we have to show that each term is Lipschitz continuous on $\Omega$ as well. Let us begin with the term $|r(x)-r(w)|$. By using the mean-value theorem for real-valued functions of several variables, we have that

$$\begin{aligned} |r(x)-r(w)|\le \Vert 2\Phi ^T\Phi (\alpha x+(1-\alpha )w)\Vert _2\Vert x-w\Vert _2, \end{aligned}$$

(100)

for some $\alpha \in [0,1]$ and all $x,w\in \Omega$. Thus,

$$\begin{aligned} |r(x)-r(w)|\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\alpha \Vert x-w\Vert _2+\Vert w\Vert _2)\Vert x-w\Vert _2\end{aligned}$$

(101)

$$\begin{aligned}\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\Vert x-w\Vert _2+\Vert w\Vert _2)\Vert x-w\Vert _2. \end{aligned}$$

(102)

Let $\sigma (\Omega )$ be the diameter of the set $\Omega$, that is,

$$\begin{aligned} \sigma (\Omega )=\sup _{x,v\in \Omega }\Vert x-v\Vert _2. \end{aligned}$$

(103)

Also, let $\rho (\Omega )$ be the $\ell _2$ norm of the largest element of the set $\Omega$, i.e.,

$$\begin{aligned} \rho (\Omega )=\sup _{x\in \Omega }\Vert x\Vert _2. \end{aligned}$$

(104)

Then,

$$\begin{aligned} |r(x)-r(w)|\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\sigma (\Omega )+\rho (\Omega ))\Vert x-w\Vert _2 \end{aligned}$$

(105)

$$\begin{aligned}\le & {} K_1\Vert x-w\Vert _2, \end{aligned}$$

(106)

where $K_1=2\Vert \Phi ^T\Phi \Vert _2(\sigma (\Omega )+\rho (\Omega ))$.

As for $|s(x)-s(w)|$, in a similar fashion, we obtain that

$$\begin{aligned} |s(x)-s(w)|\le \Vert \nabla s(\alpha x+(1-\alpha )w)\Vert _2\Vert x-w\Vert _2. \end{aligned}$$

(107)

In fact, it can be shown that for any vector $v\in {\mathbb {R}}^n$, the norm of the gradient of s is bounded by

$$\begin{aligned} \Vert \nabla s(v)\Vert \le (\sqrt{2}+1)\frac{\Vert D\Vert _2}{\Vert y\Vert _2}. \end{aligned}$$

(108)

Let $K_2=(\sqrt{2}+1)\frac{\Vert D\Vert _2}{\Vert y\Vert _2}$. Thus, $|s(x)-s(w)|\le K_2\Vert x-w\Vert _2$.

Regarding the term $\Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F$, we have that the ij-th each entry of the $n\times n$ matrix $x\nabla s(x)^T-w\nabla s(w)^T$ is given by

$$\begin{aligned} \nabla _js(x)x_i-\nabla _js(w)w_i, \end{aligned}$$

(109)

where $\nabla _js(\cdot )$ is the j-th component of the gradient of $s(\cdot )$. By employing the mean-value theorem for functions of one variable we obtain that

$$\begin{aligned} |\nabla _js(x)x_i-\nabla _js(w)w_i|=\left| \frac{\partial }{\partial x_i}(\nabla _js(x(v)))\right| |x_i-w_i|, \end{aligned}$$

(110)

for some $v\in {\mathbb {R}}$. Here, $x(v)=[x_1,\dots ,x_{i-1},v,\dots ,x_n]$. The partial derivative of the previous equation is bounded, which can be proved using the classical triangle inequality and differential calculus. Given this, we have that

$$\begin{aligned} \left| \frac{\partial }{\partial x_i}(\nabla _js(x))(v)\right|\le & {} (\sqrt{2}+3)\frac{\Vert \Phi _i^T\Vert _2\Vert \Phi _j^T\Vert _2}{\Vert y\Vert _2^2}+(2\sqrt{3}+2)\frac{\Vert \Phi _j^T\Vert _2}{\Vert y\Vert _2^3} \end{aligned}$$

(111)

$$\begin{aligned}= & {} K_{ij}, \end{aligned}$$

(112)

where $\Phi _k^T$ is the k-th row of the the transpose of the matrix $\Phi$. Therefore,

$$\begin{aligned} |\nabla _js(x)x_i-\nabla _js(w)w_i,|\le K_{ij}|x_i-w_i|. \end{aligned}$$

(113)

Using this result, we can conclude that

$$\begin{aligned} \Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F\le K_3\Vert x-w\Vert _2, \end{aligned}$$

(114)

where $K_3$ is equal to

$$\begin{aligned} K_3=n\max _{1\le i,j\le n}K_{ij}; \end{aligned}$$

(115)

that is, $K_3$ is equal to the largest $K_{ij}$ times n.

In a similar manner, it can be shown that

$$\begin{aligned} \Vert x\nabla r(x)^T-w\nabla r(w)^T\Vert _F\le K_4\Vert x-w\Vert _2, \end{aligned}$$

(116)

where $K_4$ is given by

$$\begin{aligned} K_4=\max _{1\le i,j\le n}\{2nK_1\Vert \Phi _j^T\Vert _2(\Vert \Phi _i^T\Vert _2+\Vert \Phi \Vert _2)\}. \end{aligned}$$

(117)

As for the term $\lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F$, this is equal to

$$\begin{aligned} \lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F=2\lambda \Vert z(\Phi ^T\Phi (x-w))^T\Vert _F. \end{aligned}$$

(118)

Each ij-th entry of the matrix $z(\Phi ^T\Phi (x-w))^T$ is given by $z_i(\Phi ^T\Phi _j(x-w))^T$, where $\Phi ^T\Phi _j$ is the j-th row of $\Phi ^T\Phi$. Then, we have that

$$\begin{aligned} 2\lambda \Vert z(\Phi ^T\Phi (x-w))^T\Vert _F=2\lambda \sqrt{\sum _i^n\sum _j^n|z_i(\Phi ^T\Phi _j(x-w))^T|^2}. \end{aligned}$$

(119)

Therefore,

$$\begin{aligned} 2\lambda \sqrt{\sum _i^n\sum _j^n|z_i(\Phi ^T\Phi _j(x-w))^T|^2}\le & {} 2\lambda \sqrt{\sum _i^n\sum _j^n|z_i|^2\Vert \Phi ^T\Phi _j\Vert _2^2\Vert x-w\Vert _2^2}\\\le & {} 2\lambda \sqrt{\sum _i^n|z_i|^2\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\Vert x-w\Vert _2\\= & {} 2\lambda \Vert z\Vert _2\sqrt{\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\Vert x-w\Vert _2\\= & {} \lambda K_5\Vert z\Vert _2\Vert x-w\Vert _2, \end{aligned}$$

where $K_5=2\sqrt{\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}$.

Finally, we obtain that

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le [(K_2+K_3)\Vert \Phi ^T\Phi \Vert _F+\lambda (K_5\Vert z\Vert _2+K_1+K_4)]\Vert x-z\Vert _2, \end{aligned}$$

which completes the proof. $\square$