Certainty upon Empirical Distributions

  • Joan Garriga
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7104)


We address the problem of assessing the information conveyed by a finite discrete probability distribution, within the context of knowledge discovery. Our approach is based on two main axiomatic intuitions: (i) the minimum information is given in the case of a uniform distribution, and (ii) knowledge is akin to a notion of richness, related to the dimension of the distribution. From this perspective, we define a statistic that has a clear interpretation in terms of a measure of certainty, and we build up a plausible hypothesis, which offers a comprehensible insight of knowledge, with a consistent algebraic structure. This includes a native value for the uncertainty related to unseen events. Our approach is then faced up with entropy based measures. Finally, by implementing our measure in a decision tree induction algorithm, we show an empirical validation of the behavior of our measure with respect to entropy. Our conclusion is that the contributions of our measure are significant, and should definitely lead to more robust models.


knowledge discovery measures of information entropy 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aczél, J., Forte, B., Ng, C.T.: Why Shannon and Hartley entropies are natural. Adv. Appl. Probab. 6, 131–146 (1974)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Aczél, J., Daróczy, Z.: On Measures of Information and Their Characterizations. Academic Press, New York (1975)zbMATHGoogle Scholar
  3. 3.
    Daróczy, Z.: Generalized information functions. Information and Control 16, 36–51 (1970)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Gini, C.W.: Variability and Mutability, contribution to the study of statistical distributions and relations. In: Studi Economico-Giuricici della R. Universita de Cagliari (1912)Google Scholar
  5. 5.
    Herfindahl, O.C.: Concentration in the U.S. Steel Industry. Unpublished doctoral dissertation. Columbia University (1950)Google Scholar
  6. 6.
    Kvalseth, T.O.: Entropy and correlation: some comments. IEEE transactions on Systems, Man and Cybernetics 17(3), 517–519 (1987)CrossRefGoogle Scholar
  7. 7.
    Lenca, P., Lallich, S., Vaillant, B.: Construction of an Off-Centered Entropy for the Supervised Learning of Imbalanced Classes: Some First Results. Communications in Statistics- Theory and Methods 39(3), 493–507 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Quinlan, J.R.: Induction of decision trees. Machine Learning 1(1), 81–106 (1986)Google Scholar
  9. 9.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  10. 10.
    Rényi, A.: On Measures of Entropy and Information. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 547–561. University of California Press (1961)Google Scholar
  11. 11.
    Shannon, C.E.: A Mathematical Theory of Communication. The Bell System Technical Journal 27, 379–423, 623-656 (1948)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Simpson, E.H.: Measurement of Diversity. Nature 163, 688 (1949)CrossRefzbMATHGoogle Scholar
  13. 13.
    Theil, H.: On the estimation of relationships involving qualitative variables. The American Journal of Sociology 76(1), 103–154 (1970)CrossRefGoogle Scholar
  14. 14.
    Frank, A., Asuncion, A.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine (2010), Google Scholar
  15. 15.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA Data Mining Software: An Update. SIGKDD Explorations 11(1) (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Joan Garriga
    • 1
  1. 1.Dptmnt. de Llenguatges i Sistemes InformàticsUniversitat Politècnica de CatalunyaSpain

Personalised recommendations