Abstract
The Bayesian evidence approach to regularisation, derived in the previous chapter, is applied to the stochastic time series generated from the logistic-kappa map. The scheme is found to prevent overfitting and lead to a stabilisation of the training process with respect to changes in the length of training time. For a small training set, it is also found to include an automatic pruning scheme: the network complexity is reduced until all remaining parameters are well-determined by the data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Reference
For the simulations reported here, this algorithm was slightly modified as follows. The order of steps 3 and 5 was inverted, and the hyperparameters α k were updated by finding the root of (10.75). Note that this equation is nonlinear in α k since the number of well-determined parameters λ k on the right-hand side of (10.75) depends on α k via (10.72). In practice, a solution can easily be obtained by invocation of a root-finding algorithm, like Brent’s method ([51], Chapter 9). In this way the speed of the standard algorithm can be slightly improved.
The cross-validation ‘error’ was estimated on an independent test set of the same size as the training set.
This terminology is slightly imprecise, because the actual weight decay ‘constant’ is given by the ratio α k /β k
Note that with a diverging weight-decay ‘constant’, λ k , all the weights in the respective weight group decay to zero and leave the mapping implemented in the kth network branch completely misplaced. Consequently, the posterior probability for the kth component in the mixture will always be small, leading to a decay of the prior α k when updated with the EM algorithm.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 1999 Springer-Verlag London Limited
About this chapter
Cite this chapter
Husmeier, D. (1999). Demonstration of the Bayesian Evidence Scheme for Regularisation. In: Neural Networks for Conditional Probability Estimation. Perspectives in Neural Computing. Springer, London. https://doi.org/10.1007/978-1-4471-0847-4_12
Download citation
DOI: https://doi.org/10.1007/978-1-4471-0847-4_12
Publisher Name: Springer, London
Print ISBN: 978-1-85233-095-8
Online ISBN: 978-1-4471-0847-4
eBook Packages: Springer Book Archive