Abstract
The standard backpropagation learning algorithm for feedforward networks aims to minimise the mean square error defined over a set of training data. This form of error measure can lead to the problem of over-fitting in which the network stores individual data points from the training set, but fails to generalise satisfactorily for new data points. In this paper we propose a modified error measure which can reduce the tendency to over-fit and whose properties can be controlled by a single scalar parameter. The new error measure depends both on the function generated by the network and on its derivatives. A new learning algorithm is derived which can be used to minimise such error measures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cater J P (1987) Successfully Using Peak Learning Rate of 10 (and greater) in Back- propagation Networks with the Heuristic Learning Algorithm. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II 645–651.
Dahl E D (1987) Accelerated Learning using the Generalised Delta Rule. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, C A Vol II, 523–530.
Denker J, Schwarz D, Wittner B, Solla S, Howard R, Jackel L and Hopfield J (1987) Large Automatic Learning, Rule Extraction and Generalisation Complex Systems 1 877–922.
Farhat N H, and Fai, B (1989) Echo Inversion and Target Shape Estimation by Neuromorphic Processing Neural Networks 2 No 2, 117–125.
Funahashi K (1989) On the Approximate Realisation of Continuous Mappings by Neural Networks. Neural Networks 2 No 3, 183–192.
Hornik K, Stinchcombe M, White H (1989) Multilayer Feedforward Networks are Universal Approxmations Neural Networks 2 No 5, 359–366.
Hush D R, Salas J M (1988) Improving the Learning Rate of Backpropagation with the Gradient Reuse Algorithm. Proceedings of the IEEE International Conference on Neural Networks San Diego, CA Vol I, 441–446.
Jacobs R A (1988) Increased Rates of Convergence through Learning Rate Adaptation. Neural Networks Vol I, No 4 295–307.
Korn G A, Korn T M (1968) Mathematical Handbook for Scientists and Engineers 2nd Ed 564.
Rumelhart D E, McClelland J L (1986) Parallel Distributed Processing: Explorations in the Micro structure of Cognition Vol 1: Foundations. Cambridge, MA MIT Press.
Stornetta W S, Huberman B A (1987) An Improved Three-Layer, Backpropagation Algorithm Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II, 637–643.
Tikhonov A N, Arsenin V Y (1977) Solutions of III Posed Problems. New York, Wiley.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1992 Springer-Verlag London Limited
About this paper
Cite this paper
Bishop, C.M. (1992). Curvature-Driven Smoothing in Backpropagation Neural Networks. In: Taylor, J.G., Mannion, C.L.T. (eds) Theory and Applications of Neural Networks. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-1833-6_8
Download citation
DOI: https://doi.org/10.1007/978-1-4471-1833-6_8
Publisher Name: Springer, London
Print ISBN: 978-3-540-19650-1
Online ISBN: 978-1-4471-1833-6
eBook Packages: Springer Book Archive