Global Convergence Property of Error Back-Propagation Method for Recurrent Neural Networks
Error Back-Propagation (BP) method and its variations are popular methods for the supervised learning of neural networks. BP method can be regarded as an approximate steepest descent method for minimizing the sum of error functions, which uses exact derivatives of each error function. Thus, they have the global convergence property under some natural conditions. On the other hand, Real Time Recurrent Learning method (RTRL) is also one of variations of BP method for the recurrent neural network (RNN) which is suited for handling time sequences. Since, for the real-time learning, this method cannot utilize exact outputs from the network, approximate derivatives of each error function are used to update the weights. Therefore, although RTRL is widely used in practice, its global convergence property is not known yet. In this paper, we show that RTRL has the global convergence property under almost the same conditions as other variations of BP.
Keywordsreal time recurrent learning global convergence property recurrent neural network
Unable to display preview. Download preview PDF.
- Bertsekas, D.P.. Nonlinear Programming: 2nd Edition, Athena Scientific, Belmont, MA, 1999.Google Scholar
- Hertz, J., Krogh, A., Palmer, R. G.. Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City, 1991.Google Scholar
- Ortega, J. M.. Numerical Analysis: A Second Course, Academic Press, NewYork, NY, 1972.Google Scholar
- Rumelhart, D. E., Hinton, G. E., Williams, R. J.. Learning internal representations by error propagation, in D. E. Rumelhart, J. L. McClelland and the PDP Research Group Eds., Parallel Distributed Processing MIT Press, Cambridge, MA, 1986; 318–362.Google Scholar