Rates of Learning in Gradient and Genetic Training of Recurrent Neural Networks

  • Ricardo Riaza
  • Pedro J. Zufiria
Conference paper


In this paper, gradient descent and genetic techniques are used for on-line training of recurrent neural networks. A singular perturbation model for gradient learning of fixed points introduces the problem of the rate of learning formulated as the relative speed of evolution of the network and the adaptation process, and motivates an analogous study when genetic training is used. The existence of bounds for the rate of learning in order to guarantee convergence is obtained in both gradient and genetic training. Some computer simulations confirm theoretical predictions.


Genetic Algorithm Adaptation Process Recurrent Neural Network Recurrent Network Stable Equilibrium Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Baldi, P.: Gradient Descent Learning Algorithm Overview: A General Dynamical Systems Perspective, IEEE Trans, on Neural Networks 6, 182–195 (1995).CrossRefGoogle Scholar
  2. [2]
    Kokotovic, P. V., Khalil, H. K., O’Reilly J.: Singular Perturbation Methods in Control: Analysis and Design. Academic Press 1986.Google Scholar
  3. [3]
    Kushner, H. J., Clark, D. S.: Stochastic Approximation Methods for Constrained and Unconstrained Systems. Springer-Verlag 1978.Google Scholar
  4. [4]
    Ljung, L.: Analysis of recursive stochastic algorithms, IEEE Tr. Aut. Cont. 22 551–575 (1977).MathSciNetMATHCrossRefGoogle Scholar
  5. [5]
    Pineda, F. J.: Generalization of Back-Propagation to Recurrent Neural Networks, Physical Review Letters 59 2229–2232 (1987).MathSciNetCrossRefGoogle Scholar
  6. [6]
    Riaza, R., Zufiria, P. J.: A singular perturbation approach to fixed point learning in dynamical systems and neural networks, Proc. 2nd World Multiconf. Systemics, Cybernetics and Informatics, Orlando, USA, 1 616–623 (1998).Google Scholar
  7. [7]
    Saberi, A., Khalil, H., Quadratic-type Lya-punov functions for singularly perturbed systems, IEEE Tr. Aut. Cont. 29 542–550 (1984).MathSciNetMATHCrossRefGoogle Scholar
  8. [8]
    Whitley, D., Genetic Algorithms and Neural Networks, in “Genetic Algorithms in Engineering and Computer Science”, Winter, G., Périaux, J., Galán, M., Cuesta, P., eds. John Wiley & Sons, 203–216 (1995).Google Scholar
  9. [9]
    Wieland, A. P., Evolving Neural Networks Controllers for Unstable Systems, IEEE Intl. J. Conf. on Neural Networks, Seattle, USA, 2 667–673 (1991).CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Wien 1999

Authors and Affiliations

  • Ricardo Riaza
    • 1
  • Pedro J. Zufiria
    • 1
  1. 1.Grupo de Redes Neuronales Dept. Matemática Aplicada a las Tecnologías de la Informaci00F3;n Escuela Técnica Superior de Ingenieros de TelecomunicaciónUniversidad Politécnica de Madrid Ciudad Universitaria s/nMadridSpain

Personalised recommendations