Dynamics of Gradient-Based Learning and Applications to Hyperparameter Estimation

  • K. Y. Michael Wong
  • Peixun Luo
  • Fuli Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2690)


We analyse the dynamics of gradient-based learning algorithms using the cavity method, considering the cases of batch learning with non-vanishing rates, and on-line learning. It has an an excellent agreement with simulations. Applications to efficient and precise estimation of hyperparameters are proposed.


Learning Rate Training Error Generalization Error Cavity Method Cavity Activation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Saad, D. (ed.): On-line learning in Neural Networks. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  2. 2.
    Wong, K.Y.M., Li, S., Tong, Y.W.: Many-body approach to the dynamics of batch learning. Phys. Rev. E 62, 4036–4042 (2000)CrossRefGoogle Scholar
  3. 3.
    Luo, P., Wong, K.Y.M.: Dynamical and stationary properties of on-line learning from finite training sets. Phys. Rev. E 67, 11906 (2003) (to appear)Google Scholar
  4. 4.
    Bös, S., Opper, M.: Dynamics of batch learning in a perceptron. J. Phys. A 31, 4835–4850 (1998)zbMATHCrossRefGoogle Scholar
  5. 5.
    Barber, D., Sollich, P.: Online learning from finite training sets, in [1], 279– 302 (1998)Google Scholar
  6. 6.
    Heimel, J.A.F., Coolen, A.C.C.: Supervised learning with restricted training sets: a generating functional analysis. J. Phys. A: Math. Gen. 34, 9009–9026 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Amari, S., Murata, N., Müller, K.-R., Finke, M., Yang, H.H.: Asymptotic statisitcal theory of overtraining and cross-validation. IEEE Trans. on Neural Networks 8, 985–996 (1997)CrossRefGoogle Scholar
  8. 8.
    Wong, K.Y.M., Li, F.: Fast parameter estimation using Green’s functions. In: Dieterrich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems 14. MIT Press, Cambridge (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • K. Y. Michael Wong
    • 1
  • Peixun Luo
    • 1
  • Fuli Li
    • 2
  1. 1.Department of PhysicsHong Kong University of Science and TechnologyHong KongChina
  2. 2.Department of Applied PhysicsXian Jiaotong UniversityXianChina

Personalised recommendations