Abstract
The performance of a learning algorithm is usually measured in terms of prediction error. It is important to choose an appropriate estimator of the prediction error. This paper analyzes the statistical properties of the K-fold cross-validation prediction error estimator. It investigates how to compare two algorithms statistically. It also analyzes the sensitivity to the changes in the training/test set. Our main contribution is to experimentally study the statistical property of repeated cross-validation to stabilize the prediction error estimation, and thus to reduce the variance of the prediction error estimator. Our simulation results provide an empirical evidence to this conclusion. The experimental study has been performed on PAL dataset for age estimation task.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Aam-library, http://groups.google.com/group/asmlibrary?pli=1
Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), http://www.csie.ntu.edu.tw/~cjlin/libsvm
Chen, C., Chang, Y., Ricanek, K., Wang, Y.: Face age estimation using model selection. In: CVPRW, pp. 93–99 (2010)
Chen, C., Yang, W., Wang, Y., Ricanek, K., Luu, K.: Facial feature fusion and model selection for age estimation. In: 9th International Conference on Automatic Face and Gesture Recognition (2011)
Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10, 1895–1923 (1998)
Efron, B., Hastie, T., Johnstone, I., Tibshirani, R.: Least angle regression. Annal of Statistics 32, 407–499 (2004)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2009)
He, X., Niyogi, P.: Locality preserving projections. In: Proceedings of Advances in Neural Information Processing Systems 16 (2003)
Minear, M., Park, D.C.: A lifespan database of adult facial stimuli. Behavior Research Methods, Instruments, & Computers 36, 630–633 (2004)
Refaeilzadeh, L.T.P., Liu, H.: On comaprison of feature selection algorithms. In: Proceedings of AAAI Workshop on Evaluation Methods for Machine Learning II, pp. 34–39 (2007)
Patterson, E., Sethuram, A., Albert, M., Ricanek, K.: Comparison of synthetic face aging to age progression by forensic sketch artist. In: IASTED International Conference on Visualization, Imaging, and Image Processing, Palma de Mallorca, Spain (2007)
Refaeilzadeh, P., Tang, L., Liu, H.: Cross-validation. In: Encyclopedia of Database Systems, pp. 532–538 (2009)
Ricanek, K., Wang, Y., Chen, C., Simmons, S.J.: Generalized multi-ethnic face age-estimation. In: BTAS (2009)
Rodriguez, J., Perez, A., Lozano, J.: Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(3), 569–575 (2010)
Salzberg, S.: On comparing classifiers: Pitfalls to avoid and a recommended approach. Data Mining and Knowledge Discovery 1, 317–327 (1997)
Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58(1), 267–288 (1996)
Vapnik, V.: Statistical learning theory. Wiley Interscience, New York (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, C., Wang, Y., Chang, Y., Ricanek, K. (2012). Sensitivity Analysis with Cross-Validation for Feature Selection and Manifold Learning. In: Wang, J., Yen, G.G., Polycarpou, M.M. (eds) Advances in Neural Networks – ISNN 2012. ISNN 2012. Lecture Notes in Computer Science, vol 7367. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31346-2_52
Download citation
DOI: https://doi.org/10.1007/978-3-642-31346-2_52
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31345-5
Online ISBN: 978-3-642-31346-2
eBook Packages: Computer ScienceComputer Science (R0)