Model Selection with Small Samples
Recently, a new model selection criterion called the subspace information criterion (SIC) was proposed. SIC gives an unbiased estimate of the generalization error with finite samples. In this paper, we theoretically and experimentally evaluate the effectiveness of SIC in comparison with existing model selection techniques. Theoretical evaluation includes the comparison of the generalization measure, approximation method, and restriction on model candidates and learning methods. The simulations show that SIC outperforms existing techniques especially when the number of training examples is small and the noise variance is large.
KeywordsModel Selection Bayesian Information Criterion Unbiased Estimate Model Candidate Generalization Error
Unable to display preview. Download preview PDF.
- Sugiyama, M. and Ogawa, H.: Subspace information criterion for model selection. Neural Computation, (2001), (to appear).Google Scholar
- Takeuchi, K.: On the selection of statistical models by AIC. Journal of the Society of Instrument and Contral Engineering, vol. 22, no. 5, pp. 445–453, (1983), (in Japanese).Google Scholar