Abstract
A survey is presented on some recent developments on the convergence of online gradient methods for feedforward neural networks such as BP neural networks. Unlike most of the convergence results which are of probabilistic and non-monotone nature, the convergence results we show here have a deterministic and monotone nature. Also considered are the cases where a momentum or a penalty term is added to the error function to improve the performance of the training procedure.
Partly supported by the National Natural Science Foundation of China.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ellacott, S.W.: The Numerical Analysis Approach of Neural Networks. In: Taylor, J.G. (ed.) Mathematical Approaches to Neural Networks, pp. 103–138. North-Holland, Amsterdam (1993)
Fine, T.L., Mukherjee, S.: Parameter Convergence and Learning Curves for Neural Networks. Neural Computation 11, 747–769 (1999)
Finnoff, W.: Diffusion Approximations for the Constant Learning Rate Backpropagation Algorithm and Resistance to Locol Minima. Neural Computation 6, 285–295 (1994)
Haykin, S.: Neural Networks, 2nd edn. Prentice-Hall, Englewood Cliffs (1999)
Liang, Y.C., et al.: Successive Approximation Training Algorithm for Feedforward Neural Networks. Neurocomputing 42, 11–322 (2002)
Li, Z., Wu, W., Tian, Y.: Convergence of an Online Gradient Method for Feedforward Neural Networks with Stochastic Inputs. Journal of Computational and Applied Mathematics 163(1), 165–176 (2004)
Liu, W.B., Dai, Y.H.: Minimization Algorithms Based on Supervisor and Searcher Cooperation: I-Fast and Robust Gradient Algorithms for Minimization Problems with Strong Noise. Journal of Optimization Theory and Application 111, 359–379 (2001)
Rumelhart, D.E., Hinton, G.E.: Learning Representations of Back-propagation Errors. Nature 323, 533–536 (1986)
Wu, W., Feng, G.R., Li, X.: Training Multylayer Perceptrons via Minimization of Sum of Ridge Functions. Advances in Computational Mathematics 17, 331–347 (2003)
Wu, W., Feng, G., Li, Z., Xu, Y.: Deterministic Convergence of an Online Gradient Method for BP Neural Networks. Accepted by IEEE Transactions on Neural Networks (2004)
Wu, W., Shao, Z.: Convergence of Online Gradient Methods for Continuous Perceptrons with Linearly Separable Training Patterns. Applied Mathematics Letters 16, 999–1002 (2003)
Wu, W., Xu, Y.S.: Deterministic Convergence of an On-line Gradient Method for Neural Networks. Journal of Computational and Applied Mathematics 144, 335–347 (2002)
Zhang, L., Wu, W.: Online Gradient Method with a Penalty Term for Feedforward Neural Networks (to appear)
Zhang, L., Wu, W.: Online Gradient Method with a Penalty Term for BP Nneural Networks (to appear)
Zhang, N., Wu, W., Zheng, G.: Convergence of Gradient Method with a Momentum Term for Feedforward Neural Networks (to appear)
Zhang, N., Wu, W.: Convergence of Gradient Method with a Momentum Term for BP Neural Networks (to appear)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wu, W. et al. (2004). Recent Developments on Convergence of Online Gradient Methods for Neural Network Training. In: Yin, FL., Wang, J., Guo, C. (eds) Advances in Neural Networks – ISNN 2004. ISNN 2004. Lecture Notes in Computer Science, vol 3173. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-28647-9_40
Download citation
DOI: https://doi.org/10.1007/978-3-540-28647-9_40
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22841-7
Online ISBN: 978-3-540-28647-9
eBook Packages: Springer Book Archive