Curvature-Driven Smoothing in Backpropagation Neural Networks

  • C M Bishop
Part of the Perspectives in Neural Computing book series (PERSPECT.NEURAL)


The standard backpropagation learning algorithm for feedforward networks aims to minimise the mean square error defined over a set of training data. This form of error measure can lead to the problem of over-fitting in which the network stores individual data points from the training set, but fails to generalise satisfactorily for new data points. In this paper we propose a modified error measure which can reduce the tendency to over-fit and whose properties can be controlled by a single scalar parameter. The new error measure depends both on the function generated by the network and on its derivatives. A new learning algorithm is derived which can be used to minimise such error measures.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Cater J P (1987) Successfully Using Peak Learning Rate of 10 (and greater) in Back- propagation Networks with the Heuristic Learning Algorithm. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II 645–651.Google Scholar
  2. Dahl E D (1987) Accelerated Learning using the Generalised Delta Rule. Proceedings of the IEEE First International Conference on Neural Networks. San Diego, C A Vol II, 523–530.Google Scholar
  3. Denker J, Schwarz D, Wittner B, Solla S, Howard R, Jackel L and Hopfield J (1987) Large Automatic Learning, Rule Extraction and Generalisation Complex Systems 1 877–922.MathSciNetMATHGoogle Scholar
  4. Farhat N H, and Fai, B (1989) Echo Inversion and Target Shape Estimation by Neuromorphic Processing Neural Networks 2 No 2, 117–125.CrossRefGoogle Scholar
  5. Funahashi K (1989) On the Approximate Realisation of Continuous Mappings by Neural Networks. Neural Networks 2 No 3, 183–192.CrossRefGoogle Scholar
  6. Hornik K, Stinchcombe M, White H (1989) Multilayer Feedforward Networks are Universal Approxmations Neural Networks 2 No 5, 359–366.CrossRefGoogle Scholar
  7. Hush D R, Salas J M (1988) Improving the Learning Rate of Backpropagation with the Gradient Reuse Algorithm. Proceedings of the IEEE International Conference on Neural Networks San Diego, CA Vol I, 441–446.CrossRefGoogle Scholar
  8. Jacobs R A (1988) Increased Rates of Convergence through Learning Rate Adaptation. Neural Networks Vol I, No 4 295–307.CrossRefGoogle Scholar
  9. Korn G A, Korn T M (1968) Mathematical Handbook for Scientists and Engineers 2nd Ed 564.Google Scholar
  10. Rumelhart D E, McClelland J L (1986) Parallel Distributed Processing: Explorations in the Micro structure of Cognition Vol 1: Foundations. Cambridge, MA MIT Press.Google Scholar
  11. Stornetta W S, Huberman B A (1987) An Improved Three-Layer, Backpropagation Algorithm Proceedings of the IEEE First International Conference on Neural Networks. San Diego, CA Vol II, 637–643.Google Scholar
  12. Tikhonov A N, Arsenin V Y (1977) Solutions of III Posed Problems. New York, Wiley.Google Scholar

Copyright information

© Springer-Verlag London Limited 1992

Authors and Affiliations

  • C M Bishop

There are no affiliations available

Personalised recommendations