Matrix forms of iterative algorithms to solve large-scale discrete ill-posed problems with an application to image restoration

  • Fernando PazosEmail author
  • Amit Bhaya
Original Research


Iterative methods to solve linear large-scale discrete problems are well known in the literature. When the linear system is ill-posed and contaminated by noise, some kind of regularization must be applied in order to achieve a feasible solution. In the first part of this paper, we revisit briefly some known methods to solve large-scale ill-posed discrete linear problems which are easy to implement and have low computational cost, formulating them in a unified manner and also proposing simple modifications in order to improve their performances. Matrix forms of iterative algorithms can be formulated depending on certain conditions on the blurring process, and have the advantage of avoiding the formation and storage in memory of the matrix that represents the blurring process, which is generally of very large dimension. As an original contribution, in the final part of this paper we present the matrix forms of the iterative algorithms revisited and test them in the problem of restoration of an image degraded by blurring and noise.


Discrete ill-posed linear problems Iterative algorithms Tikhonov regularization Krylov subspaces Image restoration 

Mathematics Subject Classification

65F10 65F22 



The first author would like to thank Prof Teresa Regińska for the interesting papers provided by her.


  1. 1.
    Ascher, U., van den Doel, K., Huang, H., Svaiter, B.: On fast integration to steady state and earlier times. Math. Model. Numer. Anal. 43, 689–708 (2009)CrossRefzbMATHGoogle Scholar
  2. 2.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988). MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bauer, F., Lukas, M.A.: Comparing parameter choice methods for regularization of ill-posed problems. Math. Comput. Simul. 81, 1795–1841 (2011). MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bhaya, A., Bliman, P.-A., Pazos, F.: Control-theoretic design of iterative methods for symmetric linear systems of equations. In: Proceedings of the 48th IEEE Conference on Decision and Control, Shanghai, China (2009).
  5. 5.
    Calvetti, D., Reichel, L., Zhang, Q.: Iterative Solution Methods for Large Linear Discrete Ill-Posed Problems, vol. 1, pp. 313–367. Birkhäuser, Boston (1999). zbMATHGoogle Scholar
  6. 6.
    Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer Academic Publishers, London (1996)CrossRefzbMATHGoogle Scholar
  7. 7.
    Wang, Y.-F., Xiao, T.-Y.: Fast realization algorithms for determining regularization parameters in linear inverse problems. Inverse Probl. 17, 281–291 (2001). MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Frommer, A., Maass, P.: Fast CG-based methods for Tikhonov–Phillips regularization. SIAM J. Sci. Comput. 20, 1831–1850 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Gazzola, S., Novati, P., Russo, M.R.: On Krylov projection methods and Tikhonov regularization. Electron. Trans. Numer. Anal. 44, 83–123 (2015)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Golub, G., Heath, M., Wahba, G.: Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21, 215–223 (1979). MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Golub, G., Van Loan, C.: Matrix Computation. The Johns Hopkins University Press, Baltimore (1989)zbMATHGoogle Scholar
  12. 12.
    Golub, G., von Matt, U.: Quadratically constrained least squares and quadratic problems. Numer. Math. 59, 561–580 (1991). MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Golub, G., von Matt, U.: Tikhonov regularization for large scale problems. In: Golub, G.H., Lui, S.H., Luk, F., Plemmons, R. (eds.) Workshop on Scientific Computing, pp. 3–26. Springer, New York (1997)Google Scholar
  14. 14.
    Greenbaum, A.: Iterative Methods for Solving Linear Systems. SIAM, Philadelphia (1997)CrossRefzbMATHGoogle Scholar
  15. 15.
    Güler, O.: Fundations of Optimization. Graduate Texts in Mathematics. Springer, Berlin (2010)Google Scholar
  16. 16.
    Hanke, M.: Iterative regularization techniques in image reconstruction. In: Colton, D., Engl, H.W., Louis, A.K., McLaughlin, J.R., Rundell, W. (eds.) Surveys on Solution Methods for Inverse Problems, pp. 35–52. Springer, Vienna (2000). CrossRefGoogle Scholar
  17. 17.
    Hanke, M., Nagy, J.G.: Restoration of atmospherically blurred images by symmetric indefinite conjugate gradient techniques. Inverse Probl. 12, 157–173 (1996). MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Hansen, P.C.: The L-curve and its use in the numerical treatment of inverse problems. In: Johnston, P. (ed.) Computational Inverse Problems in Electrocardiology, Advances in Computational Bioengineering. IMM, Department of Mathematical Modelling, Technical University of Denmark (2000)Google Scholar
  19. 19.
    Hansen, P.C.: Regularization tools: a MATLAB package for analysis and solution of discrete ill-posed problems. Numer. Algorithms 46, 189–194 (2007). MathSciNetCrossRefGoogle Scholar
  20. 20.
    Hansen, P.C., Nagy, J., O’Leary, D.P.: Deblurring Images. Matrices, Spectra and Filtering, Fundamentals of Algorithms. SIAM, Philadelphia (2006)CrossRefzbMATHGoogle Scholar
  21. 21.
    Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Natl. Bur. Stand. 49, 409–436 (1952)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Hochstenbach, M.E., Reichel, L., Rodriguez, G.: Regularization parameter determination for discrete ill-posed problems. J. Comput. Appl. Math. 273, 132–149 (2015). MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Jain, A.: Fundamentals of Digital Image Processing. Prentice-Hall, Englewood Cliffs, NJ (1989)zbMATHGoogle Scholar
  24. 24.
    Jensen, T.K., Hansen, P.C.: Iterative regularization with minimum-residual methods. BIT Numer. Math. 47, 103–120 (2007). MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Kilmer, M., O’Leary, D.P.: Choosing regularization parameters in iterative methods for ill-posed problems. SIAM J. Matrix Anal. Appl. 22, 1204–1221 (2001). MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Kunisch, K., Zou, J.: Iterative choices of regularization parameters in linear inverse problems. Inverse Probl. 14, 1247–1264 (1998). MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Morozov, V.A.: Regularization Methods for Ill-Posed Problems. CRC Press, Albany, NY (1993)zbMATHGoogle Scholar
  28. 28.
    Nagy, J.G., Palmer, K.M.: Steepest descent, CG and iterative regularization of ill posed problems. BIT Numer. Math. 43, 1003–1017 (2003). MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Nagy, J.G., Palmer, K.M.: Quasi-Newton methods for image restoration. In: Proceedings of SPIE—The International Society for Optical Engineering, vol. 5559, pp. 412–422 (2004).
  30. 30.
    Nagy, J.G., O’Leary, D.P.: Fast iterative image restoration with a spatially-varying PSF. In: Luk, F.T. (ed.) Proceedings of SPIE—The International Society for Optical Engineering, vol. 3162, pp. 388–399. San Diego, CA, USA (1997).
  31. 31.
    Nagy, J.G., O’Leary, D.P.: Restoring images degraded by spatially variant blur. SIAM J. Sci. Comput. 19, 1063–1082 (1998). MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Novati, P., Russo, M.R.: Adaptive Arnoldi–Tikhonov regularization for image restoration. Numer. Algorithms 65, 745–757 (2014). MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Paige, C.C., Saunders, M.A.: LSQR: sparse linear equations and least squares problems. ACM Trans. Math. Softw. 8, 195–209 (1982). CrossRefzbMATHGoogle Scholar
  34. 34.
    Pazos, F., Bhaya, A.: Matrix forms of gradient descent algorithms applied to restoration of blurred images. Int. J. Signal Process. Image Process. Pattern Recognit. 7, 17–28 (2014). Google Scholar
  35. 35.
    Pazos, F., Bhaya, A.: Adaptive choice of the Tikhonov regularization parameter to solve ill-posed linear algebraic equations via Liapunov optimizing control. J. Comput. Appl. Math. 279, 123–132 (2015). MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Regińska, T.: A regularization parameter in discrete ill-posed problems. SIAM J. Sci. Comput. 17, 740–749 (1996). MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Reichel, L., Rodriguez, G.: Old and new parameter choice rules for discrete ill-posed problems. Numer. Algorithms 63, 65–87 (2013). MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Reichel, L., Shyshkov, A.: A new zero-finder for Tikhonov regularization. BIT Numer. Math. 48, 627–643 (2008). MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. Winston & Sons, Washignton, DC (1977)zbMATHGoogle Scholar
  40. 40.
    Xie, J., Zou, J.: An improved model function method for choosing regularization parameters in linear inverse problems. Inverse Probl. 18, 631–643 (2002). MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    You, Y.-L., Kaveh, M.: Regularization and image restoration using differential equations. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. 5, pp. 285–288 (1993).
  42. 42.
    Zhang, D., Huang, T.-Z.: Generalized Tikhonov regularization method for large-scale linear inverse problems. J. Comput. Anal. Appl. 15, 1317–1331 (2013)MathSciNetzbMATHGoogle Scholar

Copyright information

© Korean Society for Computational and Applied Mathematics 2018

Authors and Affiliations

  1. 1.Department of Electronics and Telecommunication EngineeringState University of Rio de JaneiroRio de JaneiroBrazil
  2. 2.Department of Electrical EngineeringFederal University of Rio de Janeiro, PEE/COPPE/UFRJRio de JaneiroBrazil

Personalised recommendations