Optimization of structural similarity in mathematical imaging

Abstract

It is now generally accepted that Euclidean-based metrics may not always adequately represent the subjective judgement of a human observer. As a result, many image processing methodologies have been recently extended to take advantage of alternative visual quality measures, the most prominent of which is the Structural Similarity Index Measure (SSIM). The superiority of the latter over Euclidean-based metrics have been demonstrated in several studies. However, being focused on specific applications, the findings of such studies often lack generality which, if otherwise acknowledged, could have provided a useful guidance for further development of SSIM-based image processing algorithms. Accordingly, instead of focusing on a particular image processing task, in this paper, we introduce a general framework that encompasses a wide range of imaging applications in which the SSIM can be employed as a fidelity measure. Subsequently, we show how the framework can be used to cast some standard as well as original imaging tasks into optimization problems, followed by a discussion of a number of novel numerical strategies for their solution.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    The Frobenius norm of an \(m\times n\) matrix A is defined as \(\Vert A\Vert _F=\sqrt{\sum _{i=1}^m\sum _{j=1}^n|a_{ij}|^2}\).

References

  1. Bach F, Jenatton R, Mairal J, Obozinski G (2012) Convex optimization with sparsity-inducing norms. In: Sra S, Nowozin S, Wright SJ (eds) Optimization for machine learning. MIT Press, Cambridge, pp 19–53

    Google Scholar 

  2. Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci Arch 2:183–202

    MathSciNet  Article  Google Scholar 

  3. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

    Google Scholar 

  4. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3:1–122

    Article  Google Scholar 

  5. Brunet D (2012) A study of the structural similarity image quality measure with applications to image processing. University of Waterloo, Waterloo Ph.D. Thesis

    Google Scholar 

  6. Brunet D, Vrscay ER, Wang Z (2011) Structural similarity-based approximation of signals and images using orthogonal bases. In: proceedings of the international conference on image analysis and recognition. Springer, pp 11–22

  7. Brunet D, Vrscay ER, Wang Z (2012) On the mathematical properties of the structural similarity index. IEEE Trans Image Process 21:1488–1499

    MathSciNet  Article  Google Scholar 

  8. Brunet D, Channappayya S, Wang Z, Vrscay ER, Bovik A (2017) Optimizing image quality. In: Monga V (ed) Handbook of convex optimization methods in imaging science. Springer, Cham, pp 15–41

    Google Scholar 

  9. Chambolle A (2004) An algorithm for total variation minimization and applications. J Math Imaging Vis 20:89–97

    MathSciNet  Article  Google Scholar 

  10. Chambolle A, Pock T (2011) A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40:120–145

    MathSciNet  Article  Google Scholar 

  11. Chambolle A, Caselles V, Cremers D, Novaga M, Pock T (2010) An introduction to total variation for image analysis. Theor Found Numer Methods Sparse Recovery 9:263–340

    MathSciNet  MATH  Google Scholar 

  12. Channappayya S, Bovik AC, Caramanis C, Heath RW Jr (2008) Design of linear equalizers optimized for the structural similarity index. IEEE Trans Image Process 17:857–872

    MathSciNet  Article  Google Scholar 

  13. Combettes PL, Pesquet J-C (2004) Image restoration subject to a total variation constraint. IEEE Trans Image Process 13:1213–1222

    Article  Google Scholar 

  14. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32:407–451

    MathSciNet  Article  Google Scholar 

  15. Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15:3736–3745

    MathSciNet  Article  Google Scholar 

  16. Fadili J, Peyré G (2011) Total variation projection with first order schemes. IEEE Trans Image Process 20:657–669

    MathSciNet  Article  Google Scholar 

  17. Fuhry M, Reichel L (2012) A new Tikhonov regularization method. Numer Algorithms 59:433–445

    MathSciNet  Article  Google Scholar 

  18. Hestenes MR (2012) Conjugate direction methods in optimization, vol 12. Springer, Berlin

    Google Scholar 

  19. Hochstenbach ME, Reichel L (2010) An iterative method for Tikhonov regularization with a general linear regularization operator. J Integral Equ Appl 22:465–482

    MathSciNet  Article  Google Scholar 

  20. Morigi S, Reichel L, Sgallari F (2007) Orthogonal projection regularization operators. Numer Algorithms 44:99–114

    MathSciNet  Article  Google Scholar 

  21. Ortega JM (1968) The Newton–Kantorovich theorem. Am Math Mon 75:658–660

    MathSciNet  Article  Google Scholar 

  22. Otero D (2015) Function-valued mappings and SSIM-based optimization in imaging. University of Waterloo, Waterloo Ph.D. Thesis

    Google Scholar 

  23. Otero D, Vrscay ER (2014a) Unconstrained structural similarity-based optimization. In: Proceedings of the international conference on image analysis and recognition. Springer, pp 167–176

  24. Otero D, Vrscay ER (2014b) Solving optimization problems that employ structural similarity as the fidelity measure. In: Proceedings of the international on image processing, computer vision and pattern recognition. CSREA Press, pp 474–479

  25. Parikh N, Boyd S (2013) Proximal algorithms. Found Trends Optim 1:123–231

    Google Scholar 

  26. Rehman A, Rostami M, Wang Z, Brunet D, Vrscay ER (2012) SSIM-inspired image restoration using sparse representation. EURASIP J Adv Signal Process 2012:1–12

    Article  Google Scholar 

  27. Rehman A, Gao Y, Wang J, Wang Z (2013) Image classification based on complex wavelet structural similarity. Signal Process Image Commun 28:984–992

    Article  Google Scholar 

  28. Shao Y, Sun F, Li H, Liu Y (2012) Structural similarity-optimal total variation algorithm for image denoising. In: Proceedings of the first international conference on cognitive systems and information processing. Springer, pp 833–843

  29. Turlach BA (2005) On algorithms for solving least squares problems under an $\ell ^1$ penalty or an $\ell ^1$ constraint. In: Proceedings of the American Statistical Association, statistical computing section. American Statistical Association, pp 2572–2577

  30. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9:81–84

    Article  Google Scholar 

  31. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

    Article  Google Scholar 

  32. Wang S, Rehman A, Wang Z, Ma S, Gao W (2012) SSIM-motivated rate-distortion optimization for video coding. IEEE Trans Circuits Syst Video Technol 22:516–529

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported in part by Discovery Grants (ERV and OM) from the Natural Sciences and Engineering Research Council of Canada (NSERC). Financial support from the Faculty of Mathematics and the Department of Applied Mathematics, University of Waterloo (DO) is also gratefully acknowledged.

Author information

Affiliations

Authors

Corresponding author

Correspondence to D. La Torre.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Proof of Theorem 4

Let \(f:X\subset {\mathbb {R}}^n\rightarrow {\mathbb {R}}\) be defined as in Eq. (23). Then, its Jacobian is Lipschitz continuous on any open convex set \(\Omega \subset X\); that is, there exists a constant \(L>0\) such that for any \(x,w\in \Omega\),

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le L\Vert x-w\Vert _2 \, . \end{aligned}$$
(95)

Here, \(\Vert \cdot \Vert _F\) denotes the Frobenius norm, and

$$\begin{aligned} L = C_1\Vert \Phi ^T\Phi \Vert _F+\lambda (C_2\Vert z\Vert _2+C_3),\,C_1,C_2,C_3>0. \end{aligned}$$
(96)

\(\square\)

Proof

Without loss of generality, and for the sake of simplicity, let the stability constant C of the dissimilarity measure be zero. Also, let y be a non-zero vector in \({\mathbb {R}}^m\). Let us define

$$\begin{aligned} s(x):=\frac{2x^Ty}{\Vert Dx\Vert _2^2+\Vert y\Vert _2^2}, \end{aligned}$$
(97)

and

$$\begin{aligned} r(x):=\Vert Dx\Vert _2^2+\Vert y\Vert _2^2. \end{aligned}$$
(98)

Therefore, we have that \(\Vert J_f(x)-J_f(w)\Vert _F\) is bounded by

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le & {} \Vert \Phi ^T\Phi \Vert _F\Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F+\nonumber \\&\lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F+\nonumber \\&\lambda \Vert x\nabla r(x)^T-w\nabla r(w)^T\Vert _F+\nonumber \\&|s(x)-s(w)|\Vert \Phi ^T\Phi \Vert _F+\lambda |r(x)-r(w)|. \end{aligned}$$
(99)

To show that \(J_f\) is Lipschitz continuous on \(\Omega\), we have to show that each term is Lipschitz continuous on \(\Omega\) as well. Let us begin with the term \(|r(x)-r(w)|\). By using the mean-value theorem for real-valued functions of several variables, we have that

$$\begin{aligned} |r(x)-r(w)|\le \Vert 2\Phi ^T\Phi (\alpha x+(1-\alpha )w)\Vert _2\Vert x-w\Vert _2, \end{aligned}$$
(100)

for some \(\alpha \in [0,1]\) and all \(x,w\in \Omega\). Thus,

$$\begin{aligned} |r(x)-r(w)|\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\alpha \Vert x-w\Vert _2+\Vert w\Vert _2)\Vert x-w\Vert _2\end{aligned}$$
(101)
$$\begin{aligned}\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\Vert x-w\Vert _2+\Vert w\Vert _2)\Vert x-w\Vert _2. \end{aligned}$$
(102)

Let \(\sigma (\Omega )\) be the diameter of the set \(\Omega\), that is,

$$\begin{aligned} \sigma (\Omega )=\sup _{x,v\in \Omega }\Vert x-v\Vert _2. \end{aligned}$$
(103)

Also, let \(\rho (\Omega )\) be the \(\ell _2\) norm of the largest element of the set \(\Omega\), i.e.,

$$\begin{aligned} \rho (\Omega )=\sup _{x\in \Omega }\Vert x\Vert _2. \end{aligned}$$
(104)

Then,

$$\begin{aligned} |r(x)-r(w)|\le & {} 2\Vert \Phi ^T\Phi \Vert _2(\sigma (\Omega )+\rho (\Omega ))\Vert x-w\Vert _2 \end{aligned}$$
(105)
$$\begin{aligned}\le & {} K_1\Vert x-w\Vert _2, \end{aligned}$$
(106)

where \(K_1=2\Vert \Phi ^T\Phi \Vert _2(\sigma (\Omega )+\rho (\Omega ))\).

As for \(|s(x)-s(w)|\), in a similar fashion, we obtain that

$$\begin{aligned} |s(x)-s(w)|\le \Vert \nabla s(\alpha x+(1-\alpha )w)\Vert _2\Vert x-w\Vert _2. \end{aligned}$$
(107)

In fact, it can be shown that for any vector \(v\in {\mathbb {R}}^n\), the norm of the gradient of s is bounded by

$$\begin{aligned} \Vert \nabla s(v)\Vert \le (\sqrt{2}+1)\frac{\Vert D\Vert _2}{\Vert y\Vert _2}. \end{aligned}$$
(108)

Let \(K_2=(\sqrt{2}+1)\frac{\Vert D\Vert _2}{\Vert y\Vert _2}\). Thus, \(|s(x)-s(w)|\le K_2\Vert x-w\Vert _2\).

Regarding the term \(\Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F\), we have that the ij-th each entry of the \(n\times n\) matrix \(x\nabla s(x)^T-w\nabla s(w)^T\) is given by

$$\begin{aligned} \nabla _js(x)x_i-\nabla _js(w)w_i, \end{aligned}$$
(109)

where \(\nabla _js(\cdot )\) is the j-th component of the gradient of \(s(\cdot )\). By employing the mean-value theorem for functions of one variable we obtain that

$$\begin{aligned} |\nabla _js(x)x_i-\nabla _js(w)w_i|=\left| \frac{\partial }{\partial x_i}(\nabla _js(x(v)))\right| |x_i-w_i|, \end{aligned}$$
(110)

for some \(v\in {\mathbb {R}}\). Here, \(x(v)=[x_1,\dots ,x_{i-1},v,\dots ,x_n]\). The partial derivative of the previous equation is bounded, which can be proved using the classical triangle inequality and differential calculus. Given this, we have that

$$\begin{aligned} \left| \frac{\partial }{\partial x_i}(\nabla _js(x))(v)\right|\le & {} (\sqrt{2}+3)\frac{\Vert \Phi _i^T\Vert _2\Vert \Phi _j^T\Vert _2}{\Vert y\Vert _2^2}+(2\sqrt{3}+2)\frac{\Vert \Phi _j^T\Vert _2}{\Vert y\Vert _2^3} \end{aligned}$$
(111)
$$\begin{aligned}= & {} K_{ij}, \end{aligned}$$
(112)

where \(\Phi _k^T\) is the k-th row of the the transpose of the matrix \(\Phi\). Therefore,

$$\begin{aligned} |\nabla _js(x)x_i-\nabla _js(w)w_i,|\le K_{ij}|x_i-w_i|. \end{aligned}$$
(113)

Using this result, we can conclude that

$$\begin{aligned} \Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F\le K_3\Vert x-w\Vert _2, \end{aligned}$$
(114)

where \(K_3\) is equal to

$$\begin{aligned} K_3=n\max _{1\le i,j\le n}K_{ij}; \end{aligned}$$
(115)

that is, \(K_3\) is equal to the largest \(K_{ij}\) times n.

In a similar manner, it can be shown that

$$\begin{aligned} \Vert x\nabla r(x)^T-w\nabla r(w)^T\Vert _F\le K_4\Vert x-w\Vert _2, \end{aligned}$$
(116)

where \(K_4\) is given by

$$\begin{aligned} K_4=\max _{1\le i,j\le n}\{2nK_1\Vert \Phi _j^T\Vert _2(\Vert \Phi _i^T\Vert _2+\Vert \Phi \Vert _2)\}. \end{aligned}$$
(117)

As for the term \(\lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F\), this is equal to

$$\begin{aligned} \lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F=2\lambda \Vert z(\Phi ^T\Phi (x-w))^T\Vert _F. \end{aligned}$$
(118)

Each ij-th entry of the matrix \(z(\Phi ^T\Phi (x-w))^T\) is given by \(z_i(\Phi ^T\Phi _j(x-w))^T\), where \(\Phi ^T\Phi _j\) is the j-th row of \(\Phi ^T\Phi\). Then, we have that

$$\begin{aligned} 2\lambda \Vert z(\Phi ^T\Phi (x-w))^T\Vert _F=2\lambda \sqrt{\sum _i^n\sum _j^n|z_i(\Phi ^T\Phi _j(x-w))^T|^2}. \end{aligned}$$
(119)

Therefore,

$$\begin{aligned} 2\lambda \sqrt{\sum _i^n\sum _j^n|z_i(\Phi ^T\Phi _j(x-w))^T|^2}\le & {} 2\lambda \sqrt{\sum _i^n\sum _j^n|z_i|^2\Vert \Phi ^T\Phi _j\Vert _2^2\Vert x-w\Vert _2^2}\\\le & {} 2\lambda \sqrt{\sum _i^n|z_i|^2\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\Vert x-w\Vert _2\\= & {} 2\lambda \Vert z\Vert _2\sqrt{\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\Vert x-w\Vert _2\\= & {} \lambda K_5\Vert z\Vert _2\Vert x-w\Vert _2, \end{aligned}$$

where \(K_5=2\sqrt{\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\).

Finally, we obtain that

$$\begin{aligned} \Vert J_f(x)-J_f(w)\Vert _F\le [(K_2+K_3)\Vert \Phi ^T\Phi \Vert _F+\lambda (K_5\Vert z\Vert _2+K_1+K_4)]\Vert x-z\Vert _2, \end{aligned}$$

which completes the proof. \(\square\)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Otero, D., La Torre, D., Michailovich, O. et al. Optimization of structural similarity in mathematical imaging. Optim Eng (2020). https://doi.org/10.1007/s11081-020-09525-8

Download citation

Keywords

  • Structural Similarity Index (SSIM)
  • Mathematical imaging
  • Visual quality
  • Numerical optimization