Curvature-Driven Smoothing in Backpropagation Neural Networks

  • C M Bishop
Part of the Perspectives in Neural Computing book series (PERSPECT.NEURAL)

Abstract

The standard backpropagation learning algorithm for feedforward networks aims to minimise the mean square error defined over a set of training data. This form of error measure can lead to the problem of over-fitting in which the network stores individual data points from the training set, but fails to generalise satisfactorily for new data points. In this paper we propose a modified error measure which can reduce the tendency to over-fit and whose properties can be controlled by a single scalar parameter. The new error measure depends both on the function generated by the network and on its derivatives. A new learning algorithm is derived which can be used to minimise such error measures.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Cater J P (1987) Successfully Using Peak Learning Rate of 10 (and greater) in Back- propagation Networks with the Heuristic Learning Algorithm. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II 645–651.Google Scholar
  2. Dahl E D (1987) Accelerated Learning using the Generalised Delta Rule. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, C A Vol II, 523–530.Google Scholar
  3. Denker J, Schwarz D, Wittner B, Solla S, Howard R, Jackel L and Hopfield J (1987) Large Automatic Learning, Rule Extraction and Generalisation Complex Systems 1 877–922.MathSciNetMATHGoogle Scholar
  4. Farhat N H, and Fai, B (1989) Echo Inversion and Target Shape Estimation by Neuromorphic Processing Neural Networks 2 No 2, 117–125.CrossRefGoogle Scholar
  5. Funahashi K (1989) On the Approximate Realisation of Continuous Mappings by Neural Networks. Neural Networks 2 No 3, 183–192.CrossRefGoogle Scholar
  6. Hornik K, Stinchcombe M, White H (1989) Multilayer Feedforward Networks are Universal Approxmations Neural Networks 2 No 5, 359–366.CrossRefGoogle Scholar
  7. Hush D R, Salas J M (1988) Improving the Learning Rate of Backpropagation with the Gradient Reuse Algorithm. Proceedings of the IEEE International Conference on Neural Networks San Diego, CA Vol I, 441–446.CrossRefGoogle Scholar
  8. Jacobs R A (1988) Increased Rates of Convergence through Learning Rate Adaptation. Neural Networks Vol I, No 4 295–307.CrossRefGoogle Scholar
  9. Korn G A, Korn T M (1968) Mathematical Handbook for Scientists and Engineers 2nd Ed 564.Google Scholar
  10. Rumelhart D E, McClelland J L (1986) Parallel Distributed Processing: Explorations in the Micro structure of Cognition Vol 1: Foundations. Cambridge, MA MIT Press.Google Scholar
  11. Stornetta W S, Huberman B A (1987) An Improved Three-Layer, Backpropagation Algorithm Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II, 637–643.Google Scholar
  12. Tikhonov A N, Arsenin V Y (1977) Solutions of III Posed Problems. New York, Wiley.Google Scholar

Copyright information

© Springer-Verlag London Limited 1992

Authors and Affiliations

  • C M Bishop

There are no affiliations available

Personalised recommendations