Abstract
Gradient descent methods for large positive definite linear and nonlinear algebraic systems arise when integrating a PDE to steady state and when regularizing inverse problems. However, these methods may converge very slowly when utilizing a constant step size, or when employing an exact line search at each step, with the iteration count growing proportionally to the condition number. Faster gradient descent methods must occasionally resort to significantly larger step sizes, which in turn yields a strongly nonmonotone decrease pattern in the residual vector norm.
In fact, such faster gradient descent methods generate chaotic dynamical systems for the normalized residual vectors. Very little is required to generate chaos here: simply damping steepest descent by a constant factor close to 1 will do. The fastest practical methods of this family in general appear to be the chaotic, two-step ones. Despite their erratic behavior, these methods may also be used as smoothers, or regularization operators. Our results also highlight the need for better theory for these methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Let us assume throughout for simplicity that λ 1>λ 2 and λ m−1>λ m .
- 2.
The vector ℓ 2-norm is utilized here and elsewhere, unless otherwise specified.
References
Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Stat. Math. Tokyo 11, 1–16 (1959)
Ascher, U.: Numerical Methods for Evolutionary Differential Equations. SIAM, Philadelphia (2008)
Ascher, U., Mattheij, R., Russell, R.: Numerical Solution of Boundary Value Problems for Ordinary Differential Equations. SIAM, Philadelphia (1995)
Ascher, U., Huang, H., van den Doel, K.: Artificial time integration. BIT Numer. Math. 47, 3–25 (2007)
Ascher, U., van den Doel, K., Huang, H., Svaiter, B.: Gradient descent and fast artificial time integration. Modél. Math. Anal. Numér. 43, 689–708 (2009)
Barzilai, J., Borwein, J.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)
Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996)
Fletcher, R.: On the Barzilai–Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications. Kluwer Series in Applied Optimization, vol. 96, pp. 235–256 (2005)
Friedlander, A., Martinez, J., Molina, B., Raydan, M.: Gradient method with retard and generalizations. SIAM J. Numer. Anal. 36, 275–289 (1999)
Hairer, E., Norsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (1993)
Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Springer, Berlin (2002)
Huang, H., Ascher, U.: Faster gradient descent and the efficient recovery of images. Math. Program. (2013, to appear)
Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (1999)
Pronzato, L., Wynn, H., Zhigljavsky, A.: Dynamical Search: Applications of Dynamical Systems in Search and Optimization. Chapman & Hall/CRC, Boca Raton (2000)
Raydan, M., Svaiter, B.: Relaxed steepest descent and Cauchy–Barzilai–Borwein method. Comput. Optim. Appl. 21, 155–167 (2002)
Shimada, I., Nagashima, T.: A numerical approach to ergodic problems of dissipative dynamical systems. Prog. Theor. Phys. 61(6), 1605–1616 (1979)
van den Doel, K., Ascher, U.: The chaotic nature of faster gradient descent methods. J. Sci. Comput. 48 (2011). doi:10.1007/s10915-011-9521-3
Vogel, C.: Computational Methods for Inverse Problem. SIAM, Philadelphia (2002)
Acknowledgements
The first author thanks IMPA, Rio de Janeiro, for support and hospitality during several months in 2011 when this work was completed.
U. Ascher supported in part by NSERC Discovery Grant 84306.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Ascher, U., van den Doel, K. (2013). Fast Chaotic Artificial Time Integration. In: de Moura, C., Kubrusly, C. (eds) The Courant–Friedrichs–Lewy (CFL) Condition. Birkhäuser, Boston. https://doi.org/10.1007/978-0-8176-8394-8_10
Download citation
DOI: https://doi.org/10.1007/978-0-8176-8394-8_10
Publisher Name: Birkhäuser, Boston
Print ISBN: 978-0-8176-8393-1
Online ISBN: 978-0-8176-8394-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)