Abstract
Recurrent Neural Networks (RNNs) possess an implicit internal memory and are well adapted for time series forecasting. Unfortunately, the gradient descent algorithms which are commonly used for their training have two main weaknesses: the slowness and the difficulty of dealing with long-term dependencies in time series. Adding well chosen connections with time delays to the RNNs often reduces learning times and allows gradient descent algorithms to find better solutions. In this article, we demonstrate that the principle of time delay learning by gradient descent, although efficient for feed-forward neural networks and theoretically adaptable to RNNs, shown itself to be difficult to use in this latter case.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/11550907_163 .
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Jin, L., Nikiforuk, N., Gupta, M.M.: Uniform Approximation of Nonlinear Dynamic Systems Using Dynamic Neural Networks. In: International Conference on Artificial Neural Networks, pp. 191–196 (1995)
Bengio, Y., Simard, P., Frasconi, P.: Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks 5, 157–166 (1994)
Lin, T., Horne, B.G., Tino, P., Giles, C.L.: Learning Long-Term Dependencies in NARX Recurrent Neural Networks. IEEE Transactions on Neural Networks 7, 13–29 (1996)
Campolucci, P., Uncini, A., Piazza, F., Rao, B.D.: On-Line Learning Algorithms for Locally Recurrent Neural Networks. IEEE Transactions on Neural Networks 10, 253–271 (1999)
Tsoi, A.C., Back, A.D.: Locally Recurrent Globally Feedforward Networks: A Critical Review of Architectures. IEEE Transactions on Neural Networks 5, 229–239 (1994)
Guignot, J., Gallinari, P.: Recurrent Neural Networks with Delays. In: International Conference on Artificial Neural Networks, pp. 389–392 (1994)
Boné, R., Crucianu, M., Asselin de Beauville, J.-P.: Learning Long-Term Dependencies by the Selective Addition of Time-Delayed Connections to Recurrent Neural Networks. NeuroComputing 48, 251–266 (2002)
Duro, R.J., Santos Reyes, J.: Discrete-Time Backpropagation for Training Synaptic Delay-Based Artificial Neural Networks. IEEE Transactions on Neural Networks 10, 779–789 (1999)
Pearlmutter, B.A.: Dynamic Recurrent Neural Networks. Research Report CMU-CS-90-196, Carnegie Mellon University School of Computer Science (1990)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representations by Error Propagation. In: Rumelhart, D.E., McClelland, J. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, pp. 318–362. MIT Press, Cambridge (1986)
Weigend, A.S., Huberman, B.A., Rumelhart, D.E.: Predicting the Future: A Connectionist Approach. International Journal of Neural Systems 1, 193–209 (1990)
Mackey, M., Glass, L.: Oscillations and chaos in physiological control systems. Science, 197–287 (1977)
Boné, R., Crucianu, M., Verley, G., Asselin de Beauville, J.-P.: A Bounded Exploration Approach to Constructive Algorithms for Recurrent Neural Networks. In: International Joint Conference on Neural Networks, 6 p (2000)
Back, A., Wan, E.A., Lawrence, S., Tsoi, A.C.: A Unifying View of some Training Algorithms for Multilayer Perceptrons with FIR Filter Synapses. Neural Networks for Signal Processing IV, 146–154 (1994)
Aussem, A.: Nonlinear Modeling of Chaotic Processes with Dynamical Recurrent Neural Networks. Neural Networks and Their Applications, 425–433 (1998)
Gers, F., Eck, D., Schmidhuber, J.: Applying LSTM to Time Series Predictable Through Time-Window Approaches. In: Int. Conference on Artificial Neural Networks, pp. 669–675 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Boné, R., Cardot, H. (2005). Time Delay Learning by Gradient Descent in Recurrent Neural Networks. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds) Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005. ICANN 2005. Lecture Notes in Computer Science, vol 3697. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11550907_29
Download citation
DOI: https://doi.org/10.1007/11550907_29
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28755-1
Online ISBN: 978-3-540-28756-8
eBook Packages: Computer ScienceComputer Science (R0)