A Penalization Criterion Based on Noise Behaviour for Model Selection
Complexity-penalization strategies are one way to decide on the most appropriate network size in order to address the trade-off between overfitted and underfitted models. In this paper we propose a new penalty term derived from the behaviour of candidate models under noisy conditions that seems to be much more robust against catastrophic overfitting errors that standard techniques. This strategy is applied to several regression problems using polynomial functions, univariate autoregressive models and RBF neural networks. The simulation study at the end of the paper will show that the proposed criterion is extremely competitive when compared to state-of-the-art criteria.
KeywordsAkaike Information Criterion Candidate Model Penalty Term Generalization Error Model Selection Criterion
Unable to display preview. Download preview PDF.
- 2.Akaike, H Information theory and an extension of the maximum likelihood principle. In B.N. Petrov and C saki ed. 2nd Intl. Symp. Inform. Theory 267–281, Budapest. (1973).Google Scholar
- 3.Amari, S. (1995), Learning and statistical inference, in M.A. Arbib, ed., ‘The Handbook of Brain Theory and Neural Networks’, MIT Press, Cambridge, Massachusetts.Google Scholar
- 4.Foster, P.D. Characterizing the generalization performance of model selection strategies. Proceeding of the 14th Intl. Conf. on Machine Learning (ICML-97) Nashville, (1997).Google Scholar
- 7.Moody, J. (1992), The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems, in J. Moody et Al. eds, ‘Advances in Neural Information Processing Systems’, Vol. 4, Morgan Kaufmann, pp. 847–854.Google Scholar
- 8.Ripley, B. (1995), Statistical ideas for selecting network architectures, Invited Presentation, Neural Information Processing Systems 8.Google Scholar