Fast Chaotic Artificial Time Integration

  • Uri AscherEmail author
  • Kees van den Doel


Gradient descent methods for large positive definite linear and nonlinear algebraic systems arise when integrating a PDE to steady state and when regularizing inverse problems. However, these methods may converge very slowly when utilizing a constant step size, or when employing an exact line search at each step, with the iteration count growing proportionally to the condition number. Faster gradient descent methods must occasionally resort to significantly larger step sizes, which in turn yields a strongly nonmonotone decrease pattern in the residual vector norm.

In fact, such faster gradient descent methods generate chaotic dynamical systems for the normalized residual vectors. Very little is required to generate chaos here: simply damping steepest descent by a constant factor close to 1 will do. The fastest practical methods of this family in general appear to be the chaotic, two-step ones. Despite their erratic behavior, these methods may also be used as smoothers, or regularization operators. Our results also highlight the need for better theory for these methods.


Gradient descent Artificial time integration Dynamical system Stability Chaos Regularization 



The first author thanks IMPA, Rio de Janeiro, for support and hospitality during several months in 2011 when this work was completed.

U. Ascher supported in part by NSERC Discovery Grant 84306.


  1. 1.
    Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Stat. Math. Tokyo 11, 1–16 (1959) MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Ascher, U.: Numerical Methods for Evolutionary Differential Equations. SIAM, Philadelphia (2008) zbMATHCrossRefGoogle Scholar
  3. 3.
    Ascher, U., Mattheij, R., Russell, R.: Numerical Solution of Boundary Value Problems for Ordinary Differential Equations. SIAM, Philadelphia (1995) zbMATHCrossRefGoogle Scholar
  4. 4.
    Ascher, U., Huang, H., van den Doel, K.: Artificial time integration. BIT Numer. Math. 47, 3–25 (2007) zbMATHCrossRefGoogle Scholar
  5. 5.
    Ascher, U., van den Doel, K., Huang, H., Svaiter, B.: Gradient descent and fast artificial time integration. Modél. Math. Anal. Numér. 43, 689–708 (2009) zbMATHCrossRefGoogle Scholar
  6. 6.
    Barzilai, J., Borwein, J.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988) MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996) zbMATHCrossRefGoogle Scholar
  8. 8.
    Fletcher, R.: On the Barzilai–Borwein method. In: Qi, L., Teo, K., Yang, X. (eds.) Optimization and Control with Applications. Kluwer Series in Applied Optimization, vol. 96, pp. 235–256 (2005) CrossRefGoogle Scholar
  9. 9.
    Friedlander, A., Martinez, J., Molina, B., Raydan, M.: Gradient method with retard and generalizations. SIAM J. Numer. Anal. 36, 275–289 (1999) MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hairer, E., Norsett, S.P., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (1993) zbMATHGoogle Scholar
  11. 11.
    Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Springer, Berlin (2002) zbMATHGoogle Scholar
  12. 12.
    Huang, H., Ascher, U.: Faster gradient descent and the efficient recovery of images. Math. Program. (2013, to appear) Google Scholar
  13. 13.
    Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (1999) zbMATHCrossRefGoogle Scholar
  14. 14.
    Pronzato, L., Wynn, H., Zhigljavsky, A.: Dynamical Search: Applications of Dynamical Systems in Search and Optimization. Chapman & Hall/CRC, Boca Raton (2000) zbMATHGoogle Scholar
  15. 15.
    Raydan, M., Svaiter, B.: Relaxed steepest descent and Cauchy–Barzilai–Borwein method. Comput. Optim. Appl. 21, 155–167 (2002) MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Shimada, I., Nagashima, T.: A numerical approach to ergodic problems of dissipative dynamical systems. Prog. Theor. Phys. 61(6), 1605–1616 (1979) MathSciNetzbMATHCrossRefGoogle Scholar
  17. 17.
    van den Doel, K., Ascher, U.: The chaotic nature of faster gradient descent methods. J. Sci. Comput. 48 (2011). doi: 10.1007/s10915-011-9521-3
  18. 18.
    Vogel, C.: Computational Methods for Inverse Problem. SIAM, Philadelphia (2002) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of British ColumbiaVancouverCanada

Personalised recommendations