Feature Selection Stability Assessment Based on the Jensen-Shannon Divergence
Feature selection and ranking techniques play an important role in the analysis of high-dimensional data. In particular, their stability becomes crucial when the feature importance is later studied in order to better understand the underlying process. The fact that a small change in the dataset may affect the outcome of the feature selection/ranking algorithm has been long overlooked in the literature. We propose an information-theoretic approach, using the Jensen-Shannon divergence to assess this stability (or robustness). Unlike other measures, this new metric is suitable for different algorithm outcomes: full ranked lists, partial sublists (top-k lists) as well as the least studied partial ranked lists. This generalized metric attempts to measure the disagreement among a whole set of lists with the same size, following a probabilistic approach and being able to give more importance to the differences that appear at the top of the list. We illustrate and compare it with popular metrics like the Spearman rank correlation and the Kuncheva’s index on feature selection/ranking outcomes artificially generated and on an spectral fat dataset with different filter-based feature selectors.
KeywordsFeature selection feature ranking stability robustness Jensen-Shannon divergence
Unable to display preview. Download preview PDF.
- 3.Boulesteix, A.-L., Slawski, M.: Stability and aggregation of ranked gene lists 10(5), 556–568 (2009)Google Scholar
- 5.Dunne, K., Cunningham, P., Azuaje, F.: Solutions to instability problems with sequential wrapper-based approaches to feature selection. Trinity College Dublin Computer Science Technical Report, 2002–2028Google Scholar
- 8.He, Z., Yu, W.: Stable feature selection for biomarker discovery. Technical Report arXiv:1001.0887 (January 2010)Google Scholar
- 10.Kalousis, A., Prados, J., Hilario, M.: Stability of feature selection algorithms. In: Fifth IEEE International Conference on Data Mining, p. 8. IEEE, Los Alamitos (2005)Google Scholar
- 13.Kuncheva, L.I.: A stability index for feature selection. In: Proceedings of the 25th conference on Proceedings of the 25th IASTED International Multi-Conference: Artificial Intelligence and Applications, pp. 390–395. ACTA Press (2007)Google Scholar
- 15.Loscalzo, S., Yu, L., Ding, C.: Consensus group stable feature selection. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2009, pp. 567–576 (2009)Google Scholar
- 16.Lustgarten, J.L., Gopalakrishnan, V., Visweswaran, S.: Measuring Stability of Feature Selection in Biomedical Datasets. In: AMIA Annual Symposium Proceedings, vol. 2009, p. 406. American Medical Informatics Association (2009)Google Scholar
- 17.MATLAB. version 7.10.0 (R2010a). The MathWorks Inc., Natick, Massachusetts (2010)Google Scholar
- 18.Osorio, M.T., Zumalacrregui, J.M., Alaiz-Rodrguez, R., Guzman-Martnez, R., Engelsen, S.B., Mateo, J.: Differentiation of perirenal and omental fat quality of suckling lambs according to the rearing system from fourier transforms mid-infrared spectra using partial least squares and artificial neural networks. Meat Science 83(1), 140–147 (2009)CrossRefGoogle Scholar
- 21.Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (1999)Google Scholar