Abstract
In this paper, we present a formulation of the learning problem that allows deterministic nonmonotone learning behaviour to be generated, i.e. the values of the error function are allowed to increase temporarily although learning behaviour is progressively improved. This is achieved by introducing a nonmonotone strategy on the error function values. We present four training algorithms which are equipped with nonmonotone strategy and investigate their performance in symbolic sequence processing problems. Experimental results show that introducing nonmonotone mechanism can improve traditional learning strategies and make them more effective in the sequence problems tested.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Antunes, C.M., Oliveira, A.L.: Temporal data mining: an overview. In: Proc. KDD Workshop on Temporal Data Mining, San Francisco, CA, August 26, 2001, pp. 1–13 (2001)
Gill, P.E., Murray, W., Wright, M.H.: Practical Optimization. Academic Press, London (1981)
Elman, J.L., Bates, E.A., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., Plunkett, K.: The shape of change. In: Rethinking Innateness: A Connectionist Perspective on Development, ch. 6. MIT Press, Cambridge (1997)
Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numerical Analysis 23, 707–716 (1986)
Grippo, L., Lampariello, F., Lucidi, S.: A quasi-discrete Newton algorithm with a nonmonotone stabilization technique. J. Optimization Theory and Applications 64(3), 495–510 (1990)
Grippo, L., Lampariello, F., Lucidi, S.: A class of nonmonotone stabilization methods in unconstrained optimization. Numerische Mathematik 59, 779–805 (1991)
Grippo, L., Sciandrone, M.: Nonmonotone globalization techniques for the Barzilai-Borwein gradient method. Computational Optimization and Applications 23, 143–169 (2002)
Fasano, G., Lampariello, F., Sciandrone, M.: A truncated nonmonotone Gauss-Newton method for large-scale nonlinear least-squares problems. Computational Optimization and Applications 34, 343–358 (2006)
Plagianakos, V.P., Magoulas, G.D., Vrahatis, M.N.: Deterministic nonmonotone strategies for effective training of multi-layer perceptrons. IEEE Trans. Neural Networks 13(6), 1268–1284 (2002)
Medsker, L.R., Jain, L.C.: Recurrent neural networks: design and applications. CRC Press, Boca Raton (2000)
Nelles, O.: Nonlinear System Identification. Springer, Berlin (2000)
Riedmiller, M., Braun, H.: Rprop – a fast adaptive learning algorithm. In: Proc. Int’l Symposium on Computer and Information Sciences, Antalya, Turkey, pp. 279–285 (1992)
Igel, C., Hüsken, M.: Empirical evaluation of the improved Rprop learning algorithms. Neurocomputing 50, 105–123 (2003)
Anastasiadis, A., Magoulas, G.D., Vrahatis, M.N.: Sign-based Learning Schemes for Pattern Classification. Pattern Recognition Letters 26, 1926–1936 (2005)
Peng, C.-C., Magoulas, G.D.: Advanced Adaptive Nonmonotone Conjugate Gradient Training Algorithm for Recurrent Neural Networks. Int’l J. Artificial Intelligence Tools (IJAIT) 17(5), 963–984 (2008)
Peng, C.-C., Magoulas, G.D.: Adaptive Self-scaling Non-monotone BFGS Training Algorithm for Recurrent Neural Networks. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D.P. (eds.) ICANN 2007. LNCS, vol. 4668, pp. 259–268. Springer, Heidelberg (2007)
Levenberg, K.: A method for the solution of certain problems in least squares. Quart. Applied Mathematics 5, 164–168 (1944)
Marquardt, D.: An algorithm for least squares estimation of nonlinear parameters. J. Society for Industrial and Applied Mathematics 11(2), 431–441 (1963)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Networks 5, 989–993 (1994)
Ampazis, N., Perantonis, S.J.: Two highly efficient second-order algorithms for training feedforward networks. IEEE Trans. Neural Networks 13, 1064–1074 (2002)
Magoulas, G.D., Chen, S.Y., Dimakopoulos, D.: A personalised interface for web directories based on cognitive styles. In: Stary, C., Stephanidis, C. (eds.) UI4ALL 2004. LNCS, vol. 3196, pp. 159–166. Springer, Heidelberg (2004)
McLeod, P., Plunkett, K., Rolls, E.T.: Introduction to connectionist modelling of cognitive processes, pp. 148–151. Oxford University Press, Oxford (1998)
Plaut, D., McClelland, J., Seidenberg, M., Patterson, K.: Understanding normal and impaired reading: computational principles in quasi-regular domains. Psychological Review 103, 56–115 (1996)
Waibel, A.: Modular construction of time-delay neural networks for speech recognition. Neural Computation 1(1), 39–46 (1989)
Waibel, A., Hanazawa, T., Hilton, G., Shikano, K., Lang, K.J.: Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing 37, 328–339 (1989)
Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)
Hagan, M.T., Demuth, H.B., Beale, M.H.: Neural Network Design. PWS Publishing, Boston (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Peng, CC., Magoulas, G.D. (2009). Nonmonotone Learning of Recurrent Neural Networks in Symbolic Sequence Processing Applications. In: Palmer-Brown, D., Draganova, C., Pimenidis, E., Mouratidis, H. (eds) Engineering Applications of Neural Networks. EANN 2009. Communications in Computer and Information Science, vol 43. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03969-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-03969-0_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-03968-3
Online ISBN: 978-3-642-03969-0
eBook Packages: Computer ScienceComputer Science (R0)