Iterative Solution Methods

  • Martin Burger
  • Barbara Kaltenbacher
  • Andreas Neubauer


This chapter deals with iterative methods for nonlinear ill-posed problems. We present gradient and Newton type methods as well as nonstandard iterative algorithms such as Kaczmarz, expectation maximization, and Bregman iterations.Our intention here is to cite convergence results in the sense of regularization and to provide further references to the literature.


Discrepancy Principle Newton Method Source Condition Leibler Divergence Newton Type Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References and Further Reading

  1. 1.
    Akcelik V, Biros G, Draganescu A, Hill J, Ghattas O, Van Bloemen Waanders B (2005) Dynamic data-driven inversion for terascale simulations: Real-time identification of airborne contaminants. In: Proceedings of SC05. IEEE/ACM, SeattleGoogle Scholar
  2. 2.
    Bachmayr M, Burger M (2009) Iterative total variation schemes for nonlinear inverse problems. Inverse Prob 25, 105004MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bakushinsky AB (1992) The problem of the convergence of the iteratively regularized Gauss-Newton method. Comput Math Math Phys 32:1353–1359MathSciNetGoogle Scholar
  4. 4.
    Bakushinsky AB (1995) Iterative methods without degeneration for solving degenerate nonlinear operator equations. Dokl Akad Nauk 344:7–8MathSciNetGoogle Scholar
  5. 5.
    Bakushinsky AB, Kokurin MY (2004) Iterative methods for approximate solution of inverse problems. Vol. 577 of Mathematics and its applications. Springer, DordrechtGoogle Scholar
  6. 6.
    Bauer F, Hohage T (2005) A Lepskij-type stopping rule for regularized Newton methods. Inverse Prob 21:1975–1991MathSciNetMATHCrossRefGoogle Scholar
  7. 7.
    Bauer F, Hohage T, Munk A (2009) Iteratively regularized Gauss-Newton method for nonlinear inverse problems with random noise. SIAM J Numer Anal 47:1827–1846MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    Baumeister J, De Cezaro A, Leitao A (2009) On iterated Tikhonov-Kaczmarz regularization methods for ill-posed problems, ICJV (2010), doi: 10.1007/s11263-010-0339-5Google Scholar
  9. 9.
    Baumeister J, Kaltenbacher B, Leitao A (2010) On Levenberg-Marquardt-Kaczmarz iterative methods for solving systems of nonlinear ill-posed equations. Inverse Problems and Imaging, (to appear)Google Scholar
  10. 10.
    Bertero M, Boccacci P (1998) Introduction to inverse problems in imaging. Institute of Physics, BristolMATHCrossRefGoogle Scholar
  11. 11.
    Bissantz N, Hohage T, Munk A, Ruymgaart F (2007) Convergence rates of general regularization methods for statistical inverse problems and applications. SIAM J Numer Anal 45:2610–2636MathSciNetMATHCrossRefGoogle Scholar
  12. 12.
    Bissantz N, Mair B, Munk A (2008) A statistical stopping rule for mlem reconstructions in pet. IEEE Nucl Sci Symp Conf Rec 8:4198–4200Google Scholar
  13. 13.
    Blaschke B, Neubauer A, Scherzer O (1997) On convergence rates for the iteratively regularized Gauss–Newton method. IMA J Numer Anal 17:421–436MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Bregman LM (1967) The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comp Math Math Phys 7:200–217CrossRefGoogle Scholar
  15. 15.
    Brune C, Sawatzky A, Burger M (2009) Bregman-EM-TV methods with application to optical nanoscopy. In: Tai X-C et al (ed) Proceedings of the 2nd International Conference on Scale Space and Variational Methods in Computer Vision. Vol. 5567 of LNCS, Springer, pp. 235–246Google Scholar
  16. 16.
    Brune C, Sawatzky A, Burger M (2009) Primal and dual Bregman methods with application to optical nanoscopy. Preprint, Submitted to IJCV: Special Issue SSVM09, Institute of Computational and Applied MathematicsGoogle Scholar
  17. 17.
    Burger M, Kaltenbacher B (2006) Regularizing Newton–Kaczmarz methods for nonlinear ill-posed problems. SIAM J Numer Anal 44:153–182MathSciNetMATHCrossRefGoogle Scholar
  18. 18.
    Cai JF, Osher S, Shen Z (2009) Convergence of the linearized Bregman iteration for l1-norm minimization. Math Comp 78:2127–2136MathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Cai JF, OSher S, Shen Z (2009) Linearized Bregman iterations for compressed sensing. Math Comp 78:1515–1536Google Scholar
  20. 20.
    Cai JF, OSher S, Shen Z (2009) Linearized Bregman iterations for frame-based image deblurring. SIAM J Imaging Sci 2:226–252Google Scholar
  21. 21.
    Ciarlet PG (1978) The finite element method for elliptic problems. North Holland, AmsterdamMATHGoogle Scholar
  22. 22.
    Colonius F, Kunisch K (1989) Output least squares stability in elliptic systems. Appl Math Optim 19:33–63MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Combettes PL, Pesquet J-C (2008) A proximal decomposition method for solving convex variational inverse problems, Inverse Prob 24, 065014 (27pp)Google Scholar
  24. 24.
    Daubechies I, Defrise M, De Mol C (2004) An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm Pure Appl Math 57:1413–1457MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    De Cezaro A, Haltmeier M, Leitao A, Scherzer O (2008) On steepest-descent-Kaczmarz methods for regularizing systems of nonlinear ill-posed equations. Appl Math Comp 202:596–607MATHCrossRefGoogle Scholar
  26. 26.
    Dembo RS, Eisenstat SC, Steihaug T (1982) Inexact Newton’s method. SIAM J Numer Anal 14:400–408MathSciNetCrossRefGoogle Scholar
  27. 27.
    Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B 39Google Scholar
  28. 28.
    Deuflhard P, Engl HW, Scherzer O (1998) A convergence analysis of iterative methods for the solution of nonlinear ill-posed problems under affinely invariant conditions. Inverse Prob 14:1081–1106MathSciNetMATHCrossRefGoogle Scholar
  29. 29.
    Douglas J, Rachford HH (1956) On the numerical solution of heat conduction problems in two and three space variables. Trans Am Math Soc 82:421–439MathSciNetMATHCrossRefGoogle Scholar
  30. 30.
    Egger H (2007) Fast fully iterative Newton-type methods for inverse problems. J Inverse Ill-Posed Prob 15:257–275MathSciNetMATHCrossRefGoogle Scholar
  31. 31.
    Egger H (2008) Y-Scale regularization. SIAM J Numer Anal 46:419–436MathSciNetMATHCrossRefGoogle Scholar
  32. 32.
    Egger H, Neubauer A (2005) Preconditioning Landweber iteration in Hilbert scales. Numer Math 101:643–662MathSciNetMATHCrossRefGoogle Scholar
  33. 33.
    Eicke B, Louis AK, Plato R (1990) The instability of some gradient methods for ill-posed problems. Numer Math 58:129–134MathSciNetMATHCrossRefGoogle Scholar
  34. 34.
    Engl HW, Hanke M, Neubauer A (1996) Regularization of inverse problems. Kluwer, DordrechtMATHCrossRefGoogle Scholar
  35. 35.
    Engl HW, Zou J (2000) A new approach to convergence rate analysis of Tiknonov regularization for parameter identification in heat conduction. Inverse Prob 16:1907–1923MathSciNetMATHCrossRefGoogle Scholar
  36. 36.
    Glowinski R, Le Tallec P (1989) Augmented lagrangian and operator splitting methods in nonlinear mechanics. SIAM, PhiladelphiaMATHCrossRefGoogle Scholar
  37. 37.
    Green PJ (1990) Bayesian reconstructions from emission tomography data using a modified EM algorithm. IEEE Trans Med Imaging 9:84–93CrossRefGoogle Scholar
  38. 38.
    Green PJ (1990) On use of the EM algorithm for penalized likelihood estimation. J R Stat Soc Ser B (Methodological) 52:443–452MATHGoogle Scholar
  39. 39.
    Haber E (2005) Quasi-Newton methods for large-scale electromagnetic inverse problems. Inverse Prob 21:305–323MathSciNetMATHCrossRefGoogle Scholar
  40. 40.
    Haber E, Ascher U (2001) A multigrid method for distributed parameter estimation problems. Inverse Prob 17:1847–1864MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    Haltmeier M, Kowar R, Leitao A, Scherzer O (2007) Kaczmarz methods for regularizing nonlinear ill-posed equations II: applications. Inverse Prob Imaging 1:507–523MathSciNetMATHCrossRefGoogle Scholar
  42. 42.
    Haltmeier M, Leitao A, Scherzer O (2007) Kaczmarz methods for regularizing nonlinear ill-posed equations I: convergence analysis. Inverse Prob Imaging 1:289–298MathSciNetMATHCrossRefGoogle Scholar
  43. 43.
    Hanke M (1997) A regularization Levenberg–arquardt scheme, with applications to inverse groundwater filtration problems. Inverse Prob 13:79–95MathSciNetMATHCrossRefGoogle Scholar
  44. 44.
    Hanke M (1997) Regularizing properties of a truncated Newton-CG algorithm for nonlinear inverse problems. Numer Funct Anal Optim 18:971–993MathSciNetMATHCrossRefGoogle Scholar
  45. 45.
    Hanke M (2009) The regularizing Levenberg-Marquardt scheme is of optimal order, J. Integral Equations Appl. 22, (2010), 259–283MathSciNetGoogle Scholar
  46. 46.
    Hanke M, Neubauer A, Scherzer O (1995) A convergence analysis of the Landweber iteration for nonlinear ill-posed problems. Numer Math 72:21–37MathSciNetMATHCrossRefGoogle Scholar
  47. 47.
    He L, Burger M, Osher S (2006) Iterative total variation regularization with non-quadratic fidelity. J Math Imaging Vision 26:167–184MathSciNetCrossRefGoogle Scholar
  48. 48.
    Hochbruck M, Hönig M, Ostermann A (2009) A convergence analysis of the exponential Euler iteration for nonlinear ill-posed problems. Inverse Prob 25:075009 (18pp)Google Scholar
  49. 49.
    Hochbruck M, Hönig M, Ostermann A (2009) Regularization of nonlinear ill-posed problems by exponential integrators. Math Mod Numer Anal 43:709–720MATHCrossRefGoogle Scholar
  50. 50.
    Hohage T (1997) Logarithmic convergence rates of the iteratively regularized Gauß-Newton method for an inverse potential and an inverse scattering problem. Inverse Prob 13: 1279–1299MathSciNetMATHCrossRefGoogle Scholar
  51. 51.
    Hohage T (1999) Iterative methods in inverse obstacle scattering: regularization theory of linear and nonlinear exponentially ill-posed problems. PhD thesis, University of LinzGoogle Scholar
  52. 52.
    Hohage T (2000) Regularization of exponentially ill-posed problems. Numer Funct Anal Optim 21:439–464MathSciNetMATHCrossRefGoogle Scholar
  53. 53.
    Jin Q, Tautenhahn U (2009) On the discrepancy principle for some Newton type methods for solving nonlinear ill-posed problems. Numer Math 111:509–558MathSciNetMATHCrossRefGoogle Scholar
  54. 54.
    Kaltenbacher B (1997) Some Newton type methods for the regularization of nonlinear ill-posed problems. Inverse Prob 13:729–753MathSciNetMATHCrossRefGoogle Scholar
  55. 55.
    Kaltenbacher B (1998) On Broyden’s method for ill-posed problems. Numer Funct Anal Optim 19:807–833MathSciNetMATHCrossRefGoogle Scholar
  56. 56.
    Kaltenbacher B (1998) A posteriori parameter choice strategies for some Newton type methods for the regularization of nonlinear ill-posed problems. Numer Math 79:501–528MathSciNetMATHCrossRefGoogle Scholar
  57. 57.
    Kaltenbacher B (2000) A projection-regularized Newton method for nonlinear ill-posed problems and its application to parameter identification problems with finite element discretization. SIAM J Numer Anal 37:1885–1908MathSciNetMATHCrossRefGoogle Scholar
  58. 58.
    Kaltenbacher B (2001) On the regularizing properties of a full multigrid method for ill-posed problems. Inverse Prob 17:767–788MathSciNetMATHCrossRefGoogle Scholar
  59. 59.
    Kaltenbacher B, Hofmann B (2009) Convergence rates for the iteratively regularized Gauss-Newton method in Banach spaces, Inverse Problems 26 (2010), 035007MathSciNetGoogle Scholar
  60. 60.
    Kaltenbacher B, Neubauer A (2006) Convergence of projected iterative regularization methods for nonlinear problems with smooth solutions. Inverse Prob 22:1105–1119MathSciNetMATHCrossRefGoogle Scholar
  61. 61.
    Kaltenbacher B, Neubauer A, Ramm AG (2002) Convergence rates of the continuous regularized Gauss–Newton method. J Inverse Ill-Posed Prob 10:261–280MathSciNetMATHGoogle Scholar
  62. 62.
    Kaltenbacher B, Neubauer A, Scherzer O (2008) Iterative regularization methods for nonlinear ill-posed problems. Radon Series on Computational and Applied Mathematics, de Gruyter, BerlinGoogle Scholar
  63. 63.
    Kaltenbacher B, Schicho J (2002) A multi-grid method with a priori and a posteriori level choice for the regularization of nonlinear ill-posed problems. Numer Math 93:77–107MathSciNetMATHGoogle Scholar
  64. 64.
    Kaltenbacher B, Schöpfer F, Schuster T (2009) Iterative methods for nonlinear ill-posed problems in Banach spaces: convergence and applications to parameter identification problems. Inverse Prob 25:065003 (19pp)Google Scholar
  65. 65.
    King JT (1992) Multilevel algorithms for ill-posed problems. Numer Math 61:311–334MathSciNetMATHCrossRefGoogle Scholar
  66. 66.
    Kowar R, Scherzer O (2002) Convergence analysis of a Landweber-Kaczmarz method for solving nonlinear ill-posed problems. In: Romanov VG, Kabanikhin SI, Anikonov YuE, Bukhgeim AL (eds) Ill-posed and inverse problems. Zeist, VSP, pp 69–90Google Scholar
  67. 67.
    Krein SG, Petunin JI (1966) Scales of Banach spaces. Russian Math Surveys 21:85–160MathSciNetCrossRefGoogle Scholar
  68. 68.
    Kügler P (2003) A derivative free Landweber iteration for parameter identification in certain elliptic PDEs. Inverse Prob 19:1407–1426MATHCrossRefGoogle Scholar
  69. 69.
    Kügler P (2003) A derivative free landweber method for parameter identification in elliptic partial differential equations with application to the manufacture of car wind-shields. PhD thesis, Johannes Kepler University, Linz, AustriaGoogle Scholar
  70. 70.
    Langer S (2007) Preconditioned Newton methods for ill-posed problems. PhD thesis, University of GöttingenGoogle Scholar
  71. 71.
    Langer S, Hohage T (2007) Convergence analysis of an inexact iteratively regularized Gauss-Newton method under general source conditions. J Inverse Ill-Posed Prob 15:19–35MathSciNetCrossRefGoogle Scholar
  72. 72.
    Lions P-L, Mercier B (1979) Splitting algorithms for the sum of two nonlinear operators. SIAM J Numer Anal 16:964–979MathSciNetMATHCrossRefGoogle Scholar
  73. 73.
    Mülthei HN, Schorr B (1989) On properties of the iterative maximum likelihood reconstruction method. Math Methods Appl Sci 11:331–342MathSciNetMATHCrossRefGoogle Scholar
  74. 74.
    Natterer F, Wübbeling F (2001) Mathematical methods in image reconstruction. Society for Industrial and Applied Mathematics (SIAM), PhiladelphiaGoogle Scholar
  75. 75.
    Neubauer A (1992) Tikhonov regularization of nonlinear ill-posed problems in Hilbert scales. Appl Anal 46:59–72MathSciNetMATHCrossRefGoogle Scholar
  76. 76.
    Neubauer A (2000) On Landweber iteration for nonlinear ill-posed problems in Hilbert scales. Numer Math 85:309–328MathSciNetMATHCrossRefGoogle Scholar
  77. 77.
    Neubauer A, Scherzer O (1995) A convergent rate result for a steepest descent method and a minimal error method for the solution of nonlinear ill-posed problems. ZAA 14:369–377MathSciNetMATHGoogle Scholar
  78. 78.
    Osher S, Burger M, Goldfarb D, Xu J, Yin W (2005) An iterative regularization method for total variation based image restoration. SIAM Multiscale Mod Simul 4:460–489MathSciNetMATHCrossRefGoogle Scholar
  79. 79.
    Press WH, Teukolsky SA, Vetterling WT, Flannery BP (2007) Numerical recipes: the art of scientific computing, 3rd edn. Cambridge University Press, CambridgeMATHGoogle Scholar
  80. 80.
    Resmerita E, Engl HW, Iusem AN (2007) The expectation-maximization algorithm for ill-posed integral equations: a convergence analysis. Inverse Prob 23:2575–2588MathSciNetMATHCrossRefGoogle Scholar
  81. 81.
    Rieder A (1999) On the regularization of nonlinear ill-posed problems via inexact Newton iterations. Inverse Prob 15:309–327MathSciNetMATHCrossRefGoogle Scholar
  82. 82.
    Rieder A (2001) On convergence rates of inexact Newton regularizations. Numer Math 88: 347–365MathSciNetMATHCrossRefGoogle Scholar
  83. 83.
    Rieder A (2005) Inexact Newton regularization using conjugate gradients as inner iteration. SIAM J Numer Anal 43:604–622MathSciNetMATHCrossRefGoogle Scholar
  84. 84.
    Sawatzky A, Brune C, Wübbeling F, Kösters T, Schäfers K, Burger M (2008) Accurate EM-TV algorithm in PET with low SNR. Nuclear Science Symposium Conference Record. NSS’08. IEEE, pp 5133–5137Google Scholar
  85. 85.
    Scherzer O (1998) A modified Landweber iteration for solving parameter estimation problems. Appl Math Optim 38:45–68MathSciNetMATHCrossRefGoogle Scholar
  86. 86.
    Schöpfer F, Louis AK, Schuster T (2006) Nonlinear iterative methods for linear ill-posed problems in Banach spaces. Inverse Prob 22:311–329MATHCrossRefGoogle Scholar
  87. 87.
    Schöpfer F, Schuster T, Louis AK (2008) An iterative regularization method for the solution of the split feasibility problem in banach spaces. Inverse Prob 24:055008 (20pp)Google Scholar
  88. 88.
    Vardi Y, Shepp LA, Kaufman L (1985) A statistical model for positron emission tomography with discussion. J Am Stat Assoc 80:8–37MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Martin Burger
    • 1
  • Barbara Kaltenbacher
    • 2
  • Andreas Neubauer
    • 3
  1. 1.University of MainzMainzGermany
  2. 2.University of GrazGrazAustria
  3. 3.Karlsruhe Institute of Technology (KIT)KarlsruheGermany

Personalised recommendations