## Abstract

It is now generally accepted that Euclidean-based metrics may not always adequately represent the subjective judgement of a human observer. As a result, many image processing methodologies have been recently extended to take advantage of alternative visual quality measures, the most prominent of which is the Structural Similarity Index Measure (SSIM). The superiority of the latter over Euclidean-based metrics have been demonstrated in several studies. However, being focused on specific applications, the findings of such studies often lack generality which, if otherwise acknowledged, could have provided a useful guidance for further development of SSIM-based image processing algorithms. Accordingly, instead of focusing on a particular image processing task, in this paper, we introduce a general framework that encompasses a wide range of imaging applications in which the SSIM can be employed as a fidelity measure. Subsequently, we show how the framework can be used to cast some standard as well as original imaging tasks into optimization problems, followed by a discussion of a number of novel numerical strategies for their solution.

This is a preview of subscription content, log in to check access.

## Notes

- 1.
The Frobenius norm of an \(m\times n\) matrix

*A*is defined as \(\Vert A\Vert _F=\sqrt{\sum _{i=1}^m\sum _{j=1}^n|a_{ij}|^2}\).

## References

Bach F, Jenatton R, Mairal J, Obozinski G (2012) Convex optimization with sparsity-inducing norms. In: Sra S, Nowozin S, Wright SJ (eds) Optimization for machine learning. MIT Press, Cambridge, pp 19–53

Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci Arch 2:183–202

Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2010) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends Mach Learn 3:1–122

Brunet D (2012) A study of the structural similarity image quality measure with applications to image processing. University of Waterloo, Waterloo Ph.D. Thesis

Brunet D, Vrscay ER, Wang Z (2011) Structural similarity-based approximation of signals and images using orthogonal bases. In: proceedings of the international conference on image analysis and recognition. Springer, pp 11–22

Brunet D, Vrscay ER, Wang Z (2012) On the mathematical properties of the structural similarity index. IEEE Trans Image Process 21:1488–1499

Brunet D, Channappayya S, Wang Z, Vrscay ER, Bovik A (2017) Optimizing image quality. In: Monga V (ed) Handbook of convex optimization methods in imaging science. Springer, Cham, pp 15–41

Chambolle A (2004) An algorithm for total variation minimization and applications. J Math Imaging Vis 20:89–97

Chambolle A, Pock T (2011) A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40:120–145

Chambolle A, Caselles V, Cremers D, Novaga M, Pock T (2010) An introduction to total variation for image analysis. Theor Found Numer Methods Sparse Recovery 9:263–340

Channappayya S, Bovik AC, Caramanis C, Heath RW Jr (2008) Design of linear equalizers optimized for the structural similarity index. IEEE Trans Image Process 17:857–872

Combettes PL, Pesquet J-C (2004) Image restoration subject to a total variation constraint. IEEE Trans Image Process 13:1213–1222

Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32:407–451

Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15:3736–3745

Fadili J, Peyré G (2011) Total variation projection with first order schemes. IEEE Trans Image Process 20:657–669

Fuhry M, Reichel L (2012) A new Tikhonov regularization method. Numer Algorithms 59:433–445

Hestenes MR (2012) Conjugate direction methods in optimization, vol 12. Springer, Berlin

Hochstenbach ME, Reichel L (2010) An iterative method for Tikhonov regularization with a general linear regularization operator. J Integral Equ Appl 22:465–482

Morigi S, Reichel L, Sgallari F (2007) Orthogonal projection regularization operators. Numer Algorithms 44:99–114

Ortega JM (1968) The Newton–Kantorovich theorem. Am Math Mon 75:658–660

Otero D (2015) Function-valued mappings and SSIM-based optimization in imaging. University of Waterloo, Waterloo Ph.D. Thesis

Otero D, Vrscay ER (2014a) Unconstrained structural similarity-based optimization. In: Proceedings of the international conference on image analysis and recognition. Springer, pp 167–176

Otero D, Vrscay ER (2014b) Solving optimization problems that employ structural similarity as the fidelity measure. In: Proceedings of the international on image processing, computer vision and pattern recognition. CSREA Press, pp 474–479

Parikh N, Boyd S (2013) Proximal algorithms. Found Trends Optim 1:123–231

Rehman A, Rostami M, Wang Z, Brunet D, Vrscay ER (2012) SSIM-inspired image restoration using sparse representation. EURASIP J Adv Signal Process 2012:1–12

Rehman A, Gao Y, Wang J, Wang Z (2013) Image classification based on complex wavelet structural similarity. Signal Process Image Commun 28:984–992

Shao Y, Sun F, Li H, Liu Y (2012) Structural similarity-optimal total variation algorithm for image denoising. In: Proceedings of the first international conference on cognitive systems and information processing. Springer, pp 833–843

Turlach BA (2005) On algorithms for solving least squares problems under an $\ell ^1$ penalty or an $\ell ^1$ constraint. In: Proceedings of the American Statistical Association, statistical computing section. American Statistical Association, pp 2572–2577

Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9:81–84

Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

Wang S, Rehman A, Wang Z, Ma S, Gao W (2012) SSIM-motivated rate-distortion optimization for video coding. IEEE Trans Circuits Syst Video Technol 22:516–529

## Acknowledgements

This work has been supported in part by Discovery Grants (ERV and OM) from the Natural Sciences and Engineering Research Council of Canada (NSERC). Financial support from the Faculty of Mathematics and the Department of Applied Mathematics, University of Waterloo (DO) is also gratefully acknowledged.

## Author information

### Affiliations

### Corresponding author

## Additional information

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Appendix

### Appendix

### Proof of Theorem 4

Let \(f:X\subset {\mathbb {R}}^n\rightarrow {\mathbb {R}}\) be defined as in Eq. (23). Then, its Jacobian is Lipschitz continuous on any open convex set \(\Omega \subset X\); that is, there exists a constant \(L>0\) such that for any \(x,w\in \Omega\),

Here, \(\Vert \cdot \Vert _F\) denotes the Frobenius norm, and

\(\square\)

### Proof

Without loss of generality, and for the sake of simplicity, let the stability constant *C* of the dissimilarity measure be zero. Also, let *y* be a non-zero vector in \({\mathbb {R}}^m\). Let us define

and

Therefore, we have that \(\Vert J_f(x)-J_f(w)\Vert _F\) is bounded by

To show that \(J_f\) is Lipschitz continuous on \(\Omega\), we have to show that each term is Lipschitz continuous on \(\Omega\) as well. Let us begin with the term \(|r(x)-r(w)|\). By using the mean-value theorem for real-valued functions of several variables, we have that

for some \(\alpha \in [0,1]\) and all \(x,w\in \Omega\). Thus,

Let \(\sigma (\Omega )\) be the diameter of the set \(\Omega\), that is,

Also, let \(\rho (\Omega )\) be the \(\ell _2\) norm of the largest element of the set \(\Omega\), i.e.,

Then,

where \(K_1=2\Vert \Phi ^T\Phi \Vert _2(\sigma (\Omega )+\rho (\Omega ))\).

As for \(|s(x)-s(w)|\), in a similar fashion, we obtain that

In fact, it can be shown that for any vector \(v\in {\mathbb {R}}^n\), the norm of the gradient of *s* is bounded by

Let \(K_2=(\sqrt{2}+1)\frac{\Vert D\Vert _2}{\Vert y\Vert _2}\). Thus, \(|s(x)-s(w)|\le K_2\Vert x-w\Vert _2\).

Regarding the term \(\Vert x\nabla s(x)^T-w\nabla s(w)^T\Vert _F\), we have that the *ij*-th each entry of the \(n\times n\) matrix \(x\nabla s(x)^T-w\nabla s(w)^T\) is given by

where \(\nabla _js(\cdot )\) is the *j*-th component of the gradient of \(s(\cdot )\). By employing the mean-value theorem for functions of one variable we obtain that

for some \(v\in {\mathbb {R}}\). Here, \(x(v)=[x_1,\dots ,x_{i-1},v,\dots ,x_n]\). The partial derivative of the previous equation is bounded, which can be proved using the classical triangle inequality and differential calculus. Given this, we have that

where \(\Phi _k^T\) is the *k*-th row of the the transpose of the matrix \(\Phi\). Therefore,

Using this result, we can conclude that

where \(K_3\) is equal to

that is, \(K_3\) is equal to the largest \(K_{ij}\) times *n*.

In a similar manner, it can be shown that

where \(K_4\) is given by

As for the term \(\lambda \Vert z(\nabla r(x)^T-\nabla r(w)^T)\Vert _F\), this is equal to

Each *ij*-th entry of the matrix \(z(\Phi ^T\Phi (x-w))^T\) is given by \(z_i(\Phi ^T\Phi _j(x-w))^T\), where \(\Phi ^T\Phi _j\) is the *j*-th row of \(\Phi ^T\Phi\). Then, we have that

Therefore,

where \(K_5=2\sqrt{\sum _j^n\Vert \Phi ^T\Phi _j\Vert _2^2}\).

Finally, we obtain that

which completes the proof. \(\square\)

## Rights and permissions

## About this article

### Cite this article

Otero, D., La Torre, D., Michailovich, O. *et al.* Optimization of structural similarity in mathematical imaging.
*Optim Eng* (2020). https://doi.org/10.1007/s11081-020-09525-8

Received:

Revised:

Accepted:

Published:

### Keywords

- Structural Similarity Index (SSIM)
- Mathematical imaging
- Visual quality
- Numerical optimization