A Blocking Strategy for Ranking Features According to Probabilistic Relevance

  • Gianluca BontempiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10122)


The paper presents an algorithm to rank features in “small number of samples, large dimensionality” problems according to probabilistic feature relevance, a novel definition of feature relevance. Probabilistic feature relevance, intended as expected weak relevance, is introduced in order to address the problem of estimating conventional feature relevance in data settings where the number of samples is much smaller than the number of features. The resulting ranking algorithm relies on a blocking approach for estimation and consists in creating a large number of identical configurations to measure the conditional information of each feature in a paired manner. Its implementation can be made embarrassingly parallel in the case of very large n. A number of experiments on simulated and real data confirms the interest of the approach.


Proposal Distribution Ranking Algorithm Feature Selection Technique Probabilistic Relevance Conditional Mutual Information 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author acknowledges the support of the “BruFence: Scalable machine learning for automating defense system” project (RBC/14 PFS-ICT 5), funded by the Institute for the encouragement of Scientific Research and Innovation of Brussels (INNOVIRIS, Brussels Region, Belgium).


  1. 1.
    Bontempi, G.: A blocking strategy to improve gene selection for classification of gene expression data. IEEE/ACM Trans. Comput. Biol. Bioinf. 4(2), 293–300 (2007)CrossRefGoogle Scholar
  2. 2.
    Bontempi, G., Meyer, P.E.: Causal filter selection in microarray data. In: Proceeding of the ICML 2010 Conference (2010)Google Scholar
  3. 3.
    Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)CrossRefzbMATHGoogle Scholar
  4. 4.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (1990)zbMATHGoogle Scholar
  5. 5.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. J. Mach. Learn. Res. 3, 1157–1182 (2003)zbMATHGoogle Scholar
  6. 6.
    Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artif. Intell. 97(1–2), 273–324 (1997)CrossRefzbMATHGoogle Scholar
  7. 7.
    Liaw, A., Wiener, M.: Classification and regression by randomforest. R News 2(3), 18–22 (2002)Google Scholar
  8. 8.
    Meyer, P.E., Bontempi, G.: Information-theoretic gene selection in expression data. In: Biological Knowledge Discovery Handbook. IEEE Computer Society (2014)Google Scholar
  9. 9.
    Montgomery, D.C.: Design and Analysis of Experiments. Wiley, Hoboken (2001)Google Scholar
  10. 10.
    Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1226–1238 (2005)CrossRefGoogle Scholar
  11. 11.
    Ramey, J.A.: Datamicroarray: Collection of Data Sets for Classification (2013). R package version 0.2.2Google Scholar
  12. 12.
    Robert, C.P., Casella, G.: Monte Carlo Statistical Methods. Springer, New York (1999)CrossRefzbMATHGoogle Scholar
  13. 13.
    Tsamardinos, I., Aliferis, C.: Towards principled feature selection: relevancy. In: Proceedings of the 9th International Workshop on Artificial Intelligence and Statistics (2003)Google Scholar
  14. 14.
    Tsamardinos, I., Aliferis, C.F., Statnikov, A.: Algorithms for large scale Markov blanket discovery. In: Proceedings of the 16th International FLAIRS Conference (FLAIRS 2003) (2003)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Machine Learning Group, Computer Science Department, Interuniversity Institute of Bioinformatics in Brussels (IB)2ULB, Université Libre de BruxellesBruxellesBelgium

Personalised recommendations