Abstract
Elman neural network is a typical class of recurrent network model. Gradient descent method is the popular strategy to train Elman neural networks. However, the gradient descent method is inefficient owing to its linear convergence property. Based on the Generalized Armijo search technique, we propose a novel conjugate gradient method which speeds up the convergence rate in training Elman networks in this paper. A conjugate gradient coefficient is proposed in the algorithm, which constructs conjugate gradient direction with sufficient descent property. Numerical results demonstrate that this method is more stable and efficient than the existing training methods. In addition, simulation shows that, the error function has a monotonically decreasing property and the gradient norm of the corresponding function tends to zero.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Ooi, S.Y., Tan, S.C., Cheah, W.P.: Experimental Study of Elman Network in Temporal Classification. Springer, Singapore (2017)
Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)
Ahalt, S.C., Liu, X.M., Wang, D.L.: On temporal generalization of simple recurrent networks. Neural Netw. Official J. Int. Neural Network Soc. 9(7), 1099–1118 (1996)
Williams, R.J., Zisper, D.: A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. MIT Press 1(2), 270–280 (1989)
Wu, W., Xu, D.P., Li, Z.X.: Convergence of gradient method for Elman networks. Appl. Math. Mech. 29(9), 1231–1238 (2008)
Xu, D.P., Li, Z.X., Wu, W.: Convergence of approximated gradient method for Elman network. Neural Network World 18(3), 171–180 (2008)
Güntürkün, R.: Using elman recurrent neural networks with conjugate gradient algorithm in determining the anesthetic the amount of anesthetic medicine to be applied. J. Med. Syst. 34(4), 479–484 (2010)
Wang, J., Wu, W., Zurada, J.M.: Deterministic convergence of conjugate gradient method for feedforward neural networks. Neurocomputing 74(14), 2368–2376 (2011)
Wang, J., Zhang, B.J., Sun, Z.Q., Hao, W.X., Sun, Q.Y.: A novel conjugate gradient method with generalized Armijo search for efficient training of feedforward neural networks. Neurocomputing 275, 308–316 (2018)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (2006)
Hestenes, M.R., Steifel, E.: Method of Conjugate Gradients for Solving Linear Systems. National Bureau of Standards, Washington (1952)
Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7(2), 149–154 (1964)
Polak, E., Ribiere, G.: Note sur la convergence de methodes de directions conjures. Revue Francaise Information Recherche Operationnelle 16(16), 35–43 (1969)
Rivaie, M., Fauzi, M., Mamat, M., Mohd, I.: A new class of nonlinear conjugate gradient coefficients with global convergence properties. AIP Conf. Proc. 1482, 486–491 (2012)
Sun, Q.Y., Liu, X.H.: Global convergence results of a new three terms conjugate gradient method with generalized Armijo step size rule. Math. Numer. Sin. 26(1), 25–36 (2004)
Dong, X., Yang, X., Huang, Y.: Global convergence of a new conjugate gradient method with Armijo search. J. Henan Normal Univ. 6, 25–29 (2015)
UCI Machine Learning Repository. http://archive.ics.uci.edu/ml
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (No. 61305075), the Natural Science Foundation of Shandong Province (Nos. ZR2015AL014, ZR201709220208) and the Fundamental Research Funds for the Central Universities (Nos. 15CX08011A, 18CX02036A).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Zhu, M., Gao, T., Zhang, B., Sun, Q., Wang, J. (2018). An Efficient Elman Neural Networks Based on Improved Conjugate Gradient Method with Generalized Armijo Search. In: Huang, DS., Bevilacqua, V., Premaratne, P., Gupta, P. (eds) Intelligent Computing Theories and Application. ICIC 2018. Lecture Notes in Computer Science(), vol 10954. Springer, Cham. https://doi.org/10.1007/978-3-319-95930-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-95930-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95929-0
Online ISBN: 978-3-319-95930-6
eBook Packages: Computer ScienceComputer Science (R0)