Minimum Risk Neural Networks and Weight Decay Technique
To enhance the generalization of neural network model, we proposed a novel neural network, Minimum Risk Neural Networks (MRNN), whose principle is the combination of minimizing the sum of squares of error and maximizing the classification margin, based on the principle of structural risk minimization. Therefore, the objective function of MRNN is the combination of the sum of squared error and the sum of squares of the slopes of the classification function. Besides, we derived a more sophisticated formula similar to the traditional weight decay technique from the MRNN, establishing a more rigorous theoretical basis for the technique. This study employed several real application examples to test the MRNN. The results led to the following conclusions. (1) As long as the penalty coefficient was in the appropriate range, MRNN performed better than pure MLP. (2) MRNN may perform better in difficult classification problems than MLP using weight decay technique.
Keywordsmulti-layer perceptrons weight decay support vector machine structural risk minimization
Unable to display preview. Download preview PDF.
- 2.Krogh, A., Hertz, J.A.: A Simple Weight Decay Can Improve Generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, San Mateo, CA, pp. 450–957 (1992)Google Scholar
- 3.Krogh, A., Hertz, J.A.: A Simple Weight Decay Can Improve Generalization. In: Advances in Neural Information Processing Systems, vol. 4, pp. 950–957 (1992)Google Scholar
- 4.Hinton, G.E., Camp, D.: Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights. In: Proceedings of the Sixth Annual Conference on Computational Learning Theory, pp. 5–13 (1993)Google Scholar
- 11.UCI Machine Learning Repository Content Summary (2008), http://archive.ics.uci.edu/ml/