A Consistency-Constrained Feature Selection Algorithm with the Steepest Descent Method

  • Kilho Shin
  • Xian Ming Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5861)


This paper proposes a new consistency-based feature selection algorithm, which presents a new balance to the fundamental tradeoff between the quality of outputs of feature selection algorithms and their efficiency. Consistency represents the extent of corrective relevance of features to classification, and hence, consistency-based feature selection algorithms such as INTERACT, LCC and CCC can select relevant features more correctly by taking interaction among features into account. INTERACT and LCC are fast by employing the linear search strategy. By contrast, CCC is slow, since it is based on the complete search strategy, but can output feature subsets of higher quality. The algorithm that we propose in this paper, on the other hand, takes the steepest descent method as the search strategy. Consequently, it can find better solutions than INTERACT and LCC, and simultaneously restrains the increase in computational complexity within a reasonable level: it evaluates \((|{\mathcal F}| + |{\tilde {\mathcal F}}|)(|{\mathcal F}| - |{\tilde {\mathcal F}}| + 1)/2\) feature subsets to output \({\tilde {\mathcal F}}\). We prove effectiveness of the new algorithm through experiments.


Feature Selection Feature Subset Real Dataset Feature Selection Algorithm Steep Descent Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: Criteria of max-dependency, max-relevance and min-redundancy. IEEE Transaction on Pattern Analysis and Machine Intelligence 27(8) (August 2005)Google Scholar
  2. 2.
    Yu, L., Liu, H.: Feature selection for high-dimensional data: a fast correlation-based filter solution. In: International Conference of Machine Learning (2003)Google Scholar
  3. 3.
    Biesiada, J., Duch, W.: Feature selection for high-dimensional data – a Kolmogorov-Smirnov correlation-based filter. Advances in Soft Computing 30, 95–103 (2005)CrossRefGoogle Scholar
  4. 4.
    Biesiada, J., Duch, W.: Feature selection for high-dimensional data – a Pearson redundancy based filter. Advances in Soft Computing 45, 242–249 (2008)CrossRefGoogle Scholar
  5. 5.
    Dash, M., Liu, H.: Consistency-based search in feature selection. Artificial Intelligence 151, 155–176 (2003)zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Almuallim, H., Dietterich, T.G.: Learning boolean concepts in the presence of many irrelevant features. Artificial Intelligence 69(1 - 2) (1994)Google Scholar
  7. 7.
    Zhao, Z., Liu, H.: Searching for interacting features. In: Proceedings of International Joint Conference on Artificial Intelligence, pp. 1156–1161 (2007)Google Scholar
  8. 8.
    Shin, K., Xu, X.: Consistency-based feature selection. In: 13th International Conferecne on Knowledge-Based and Intelligent Information & Engineering Systems (2009),
  9. 9.
    Blake, C.S., Merz, C.J.: UCI repository of machine learning databases. Technical report, University of California, Irvine (1998)Google Scholar
  10. 10.
    IEEE World Congress on Computational Intelligence: Performance prediction challenge (2006),

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Kilho Shin
    • 1
  • Xian Ming Xu
    • 1
  1. 1.Carnegie Mellon CyLabJapan

Personalised recommendations