Abstract
Selecting the optimal number of features in a classifier ensemble normally requires a validation set or cross-validation techniques. In this paper, feature ranking is combined with Recursive Feature Elimination (RFE), which is an effective technique for eliminating irrelevant features when the feature dimension is large. Stopping criteria are based on out-of-bootstrap (OOB) estimate and class separability, both computed on the training set thereby obviating the need for validation. Multi-class problems are solved using the Error-Correcting Output Coding (ECOC) method. Experimental investigation on natural benchmark data demonstrates the effectiveness of these stopping criteria.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Windeatt, T.: Vote Counting Measures for Ensemble Classifiers. Pattern Recognition 36(12), 2743–2756 (2003)
Skuruchina, M., Duin, R.P.W.: Combining feature subsets in feature selection. In: Oza, N.C., et al. (eds.) MCS 2005. LNCS, vol. 3541, pp. 165–174. Springer, Heidelberg (2005)
Fukunaga, K.: Statistical Pattern Recognition. Academic Press, London (1990)
Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Analysis and Machine Intelligence 20(8), 832–844 (1998)
Bylander, T.: Estimating generalisation error two-class datasets using out-of-bag estimate. Machine Learning 48, 287–297 (2002)
Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles. Machine Learning 51, 181–207 (2003)
Windeatt, T.: Accuracy/Diversity and ensemble classifier design. IEEE Trans. Neural Networks 17(5), 1194–1211 (2006)
Windeatt, T.: Diversity Measures for Multiple Classifier System Analysis and Design. Information Fusion 6(1), 21–36 (2004)
Windeatt, T., Ghaderi, R.: Coding and Decoding Strategies for Multi-class Learning Problems. Information Fusion 4(1), 11–21 (2003)
Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artificial Intelligence Journal (special issue on relevance) 97(1-2), 273–324 (1997)
Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, 1157–1182 (2003)
Yu, L., Liu, H.: Efficient feature selection via analysis of relevance and redundancy. Journal of Machine Learning Research 5, 1205–1224 (2004)
Oza, N., Tumer, K.: Input Decimation ensembles: decorrelation through dimensionality reduction. In: Kittler, J., Roli, F. (eds.) MCS 2001. LNCS, vol. 2096, pp. 238–247. Springer, Heidelberg (2001)
Bryll, R., Gutierrez-Osuna, R., Quek, F.: Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern Recognition 36, 1291–1302 (2003)
Guyon, I., et al.: Gene selection for cancer classification using support vector machines. Machine Learning 46(1-3), 389–422 (2002)
Hsu, C., Huang, H., Schuschel, D.: The ANNIGMA-wrapper approach to fast feature selection for neural nets. IEEE Trans. System, Man and Cybernetics-Part B: Cybernetics 32(2), 207–212 (2002)
Wang, W., Jones, P., Partridge, D.: Assessing the impact of input features in a feedforward neural network. Neural Computing and Applications 9, 101–112 (2000)
Montana, J.J., Palmer, A.: Numeric Sensitivity analysis applied to feedforward neural networks. Neural Computing and Applications 12, 119–125 (2003)
Prechelt, L.: Proben1: A set of neural network Benchmark Problems and Benchmarking Rules. Tech. Report 21/94, Univ. Karlsruhe, Germany (1994)
Merz, C.J., Murphy, P.M.: UCI repository of machine learning databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Windeatt, T., Prior, M. (2007). Stopping Criteria for Ensemble-Based Feature Selection. In: Haindl, M., Kittler, J., Roli, F. (eds) Multiple Classifier Systems. MCS 2007. Lecture Notes in Computer Science, vol 4472. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72523-7_28
Download citation
DOI: https://doi.org/10.1007/978-3-540-72523-7_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72481-0
Online ISBN: 978-3-540-72523-7
eBook Packages: Computer ScienceComputer Science (R0)