Abstract
Updating steps in a backpropagation neural network with multiplicative factors u > 1 and d < 1 has been presented by several authors. The istatistics field of Stochastic Approximation has a close relation with back-propagation algorithms. Recent theoretical results in this field show that for functions of one variable, different values of u and d can produce very different results: fast convergence at the cost of a poor solution, slow convergence with a better solution, or produce a fast move towards a solution but without converging. To speed up backpropagation in a simple manner we propose a batch step adaptation technique for the online backpropagation algorithm based on theoretical results on simple cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hertz, J., Krogh, A., Palmer, R. G., (1991) Introduction to the theory of neural computation, ISBN 0-201-50395-6, Addison-Wesley Longman Publishing Co., Inc.
Kushner, Harold J., Yin, G. George (1997) Stochastic approximation algorithms and applications, Applications of Mathematics. 35. Berlin: Springer, xxi, 417 p.
LeCun, Y., Bottou, L., Bengio Y, Haffner, P. (1998) Gradient-based learning applied to document recognition. Proc. of the IEEE, vol. 86, nr. 11, 1998.
Spall, J. C, (2000) Adaptive stochastic approximation by the simultaneous perturbation method., IEEE Trans. Autom. Control, vol.45, nr. 10: 1839–1853.
Silva, F. M., Almeida, L. B. (1990), Speeding up backpropagation, Advanced Neural Computers, ed. R. Eckmiller, Elsevier Science Publishers, Amsterdam, pp151–158.
Salomon, R., Hemmen, van J. L., (1996), Accelerating Backpropagation through Dynamic Self-Adaptation, Neural Networks, vol. 9, nr. 4: 589–601.
Plakhov, A., Cruz, P. (to be published), A stochastic approximation algorithm with multiplicative step size adaptation.
Battiti, R., (1992) First and second order methods for learning: between steepest descent and Newton’s method, Neural Computation, vol. 4, nr. 2: 141–166, The MIT Press, Cambridge-Massachusetts.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag/Wien
About this paper
Cite this paper
Cruz, P. (2005). Speeding up backpropagation with Multiplicative Batch Update Step. In: Ribeiro, B., Albrecht, R.F., Dobnikar, A., Pearson, D.W., Steele, N.C. (eds) Adaptive and Natural Computing Algorithms. Springer, Vienna. https://doi.org/10.1007/3-211-27389-1_6
Download citation
DOI: https://doi.org/10.1007/3-211-27389-1_6
Publisher Name: Springer, Vienna
Print ISBN: 978-3-211-24934-5
Online ISBN: 978-3-211-27389-0
eBook Packages: Computer ScienceComputer Science (R0)