Abstract
In this paper we present a method for eliminating overtraining during learning on small and noisy data sets. The key idea is to reduce the complexity of the neural network by increasing the stochasticity of the information transmission from the input layer to the hidden-layer. The architecture of the network is a stochastic multilayer perceptron the hidden layer of which behaves like a Pott-Spin. The stochasticity is increased by penalizing the mutual information between the input and its internal representation in the hidden layer.Theoretical and empirical studies validate the usefulness of this novel approach to the problem of overtraining.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Deco G, Finnoff W., and Zimmermann H.G., 1993, “Elimination of Overtraining by a Mutual Information Network”, ICANN’93, Amsterdam, Proc. p. 744–749.
Le Cun Y., Denker J. and Solla S., 1990, “Optimal Brain Damage”, in Proceedings of the Neural Information Processing Systems, Denver, 598–605.
Nowlan, S. and Hinton, G., 1991, “Adaptive Soft Weight Tying using Gaussian Mixtures”, Neural Information Proccesing Systems, VoL 4, 847–854, San Mateo,C.A. Morgan Kaufmann.
Peterson C. and Soederberg B., 1989, “A new method for mapping optimization problems onto neural networks”, Int. J. Neural Syst., 1, 68.
Weigend A., Rumelhart D. and Hubennan B., 1991, “Generalization by weight elimination with application to forecasting”, in Advances in Neural Information Proccesing, Ш, Ed. R. P. Lippman and J. Moody, Morgan Kaufman, 1991.
Friedman J.H., “Multivariate adaptive regression splines”, 1991, Annals of Statistics, 19, 1–141.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1994 Springer-Verlag London Limited
About this paper
Cite this paper
Deco, G., Martinetz, T. (1994). Regularizing Stochastic Pott Neural Networks by Penalizing Mutual Information. In: Marinaro, M., Morasso, P.G. (eds) ICANN ’94. ICANN 1994. Springer, London. https://doi.org/10.1007/978-1-4471-2097-1_163
Download citation
DOI: https://doi.org/10.1007/978-1-4471-2097-1_163
Published:
Publisher Name: Springer, London
Print ISBN: 978-3-540-19887-1
Online ISBN: 978-1-4471-2097-1
eBook Packages: Springer Book Archive