Evolutionary Learning of Recurrent Networks by Successive Orthogonal Inverse Approximations
Recurrent networks have proved to be more powerful than feedforward neural networks in terms of classes of functions they can compute. But, because training of recurrent networks is a difficult task, it is not clear that these networks provide an advantage over feedforward networks for learning from examples. This communication proposes a general computation model that lays the foundations for characterizing the classes of functions computed by feedforward nets and convergent recurrent nets. Then a mathematical statement proves that convergent nets outperform feedforward nets on data fitting problems. It provides the basis to devise a new learning procedure that constraints the attractor set of a recurrent net and assures a convergent dynamic by using orthogonal inverse tools. The learning algorithm is based on an evolutionary selection mechanism. Using the previous procedure as evaluation function, it has been shown to be robust and well adapted to train convergent recurrent nets when feedforward nets cannot approximate a real parameter mapping.
KeywordsGradient Descent Recurrent Network Front Axle Good Initialization Synchronous Dynamic
Unable to display preview. Download preview PDF.
- R.P. Agaxwal. Difference Equations and Inequalities: Theory, Methods and Applications. Pure and Applied Mathematics. Springer, E.J. Taft and Z. Nashed ed. edition, 1991.Google Scholar
- American Institute of Physics. Neural Information Processing Systems Conference, volume 1, 1987.Google Scholar
- Y. Bengio, P. Simard, and P. Frasconi. Learning Long-Term Dependencies with gradient is difficult. IEEE Trans, on Neural Networks, 1993. Special issue on Recurrent Networks.Google Scholar
- O. Christensen. Frames and pseudo-inverses. J. Math. Anal. Appl., 2, 1995.Google Scholar
- C. Gégout. Improvement of Multilayer Perceptron Trainings with an Evolutionary Initialization. In ICANN’95, volume 2, pages 153–158. EC2, 1995.Google Scholar
- C. Gégout. Stable and Convergent Dynamics for Discrete-time Recurrent Networks. In World Congress of Nonlinear Analysts. IFNA, July 1996. to be published in 1997.Google Scholar
- M. Schoenauer, E. Ronald, and S. Damour. Evolving Networks for Control. In Neuronîmes 93, Paris, 1993. EC2.Google Scholar
- H.T. Siegelmann, B.G. Horne, and C.L. Giles. Computational capabilities of recurrent narx neural networks. Technical Report 95–78, University of Maryland, College Park, Md, 1995. Accepted also IEEE Trans, on Syst., Man and Cybern.Google Scholar
- P.Y. Simard. Learning State Space Dynamics in Recurrent Networks. PhD thesis, University of Rochester, New York, mars 1991. 383.Google Scholar