Abstract
In this paper we describe a new penalty-based model selection criterion for nonlinear models which is based on the influence of the noise in the fitting. According to Occam’s razor we should seek simpler models over complex ones and optimize the trade-off between model complexity and the accuracy of a model’s description to the training data. An empirical derivation is developed and computer simulations for multilayer perceptron with weight decay regularization are made in order to show the efficiency and robustness of the method in comparison with other well-known criteria for nonlinear systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Akaike, H. Statistical predictor identification. Ann. Inst. Statist. Math. 22 (1970) 203–217
Akaike, H. Information theory and an extension of the maximum likelihood principle. In B.N. Petrov and Csaki ed. 2nd Intl. Symp. Inform. Theory (1973) 267–281, Budapest.
Barron, A. Predicted squared error: a criterion for automatic model selection. Self-Organizing Methods in Modeling, S. Farlow, ed., Marcel Dekker, New York (1984)
Cherkassky V., Mulier F.: Learning from data: concepts, theory and methods. New York, Wiley (1998)
Craven P., Wahba G. Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation. Numer. Math., 31:377–403(1979)
Efron B. & Tibshirani R. J., An Introduction to the Bootstrap. London, U.K.: Chapman & Hall (1993)
Larsen J., Hansen L.K.: Generalization performance of regularized neural network models. Proc. IEEE Workshop: Neural Networks for Signal Processing IV, Piscataway, New Jersey (1994) 42–51
Mallows, C. L. Some comments on CP. Technometrics 15 (1973):661–675
Moody, J.: The effective number of parameters: an analysis of generalization and regularization in nonlinear learning systems. NIPS (1992) 847–854
Moody, J.: Prediction Risk and Architecture Selection for Neural Networks. In Cherkassky, V., et al. editors, From Statistics to Neural Networks: Theory and Pattern Recognition Applications, NATO ASI Series F. Springer-Verlag (1994)
Murata N., Yoshizawa S., Amari S.: Network Information Criterion— Determining the Number of Hidden Units for an Artificial Neural Network Model. IEEE Transactions on Neural Networks (1994) 5, 865–872
Pizarro J., et al.: Noise derived information criterion for model selection. Proceedings of European Symposium on Artificial Neural Networks, ESANN (2002) pp. 155–160
Sarle W.: Donojo-Jonhstone benchmarks: neural nets results, ftp://ftp.sas.com/pub/neural/dojo/dojo.html (1999)
Stone, M.. Cross-validation: a review. Math. Operations for Sch. Statist., ser. Statistic, (1978) 9, 1.
Vapnik, V. Statistical Learning Theory John Wiley, New York (1995)
Zapranis A., Refenes A.: Principles of Neural Model Identification, Selection and Adequacy: with applications to financial economics. (Perspectives in neural computing). Springer-Verlag London (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Guerrero, E., Pizarro, J., Yáñez, A., Galindo, P. (2003). A new penalty-based criterion for model selection in regularized nonlinear models. In: Mira, J., Álvarez, J.R. (eds) Computational Methods in Neural Modeling. IWANN 2003. Lecture Notes in Computer Science, vol 2686. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44868-3_48
Download citation
DOI: https://doi.org/10.1007/3-540-44868-3_48
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40210-7
Online ISBN: 978-3-540-44868-6
eBook Packages: Springer Book Archive