Entropy, Combinatorial Dimensions and Random Averages

  • Shahar Mendelson
  • Roman Vershynin
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2375)


In this article we introduce a new combinatorial parameter which generalizes the VC dimension and the fat-shattering dimension, and extends beyond the function-class setup. Using this parameter we establish entropy bounds for subsets of the n-dimensional unit cube, and in particular, we present new bounds on the empirical covering numbers and gaussian averages associated with classes of functions in terms of the fat-shattering dimension.


Convex Body Absolute Constant Combinatorial Dimension Entropy Bound Large Cube 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    N. Alon, S. Ben-David, N. Cesa-Bianchi, D. Hausser, Scale sensitive dimensions, uniform convergence and learnability, Journal of the ACM 44 (1997), 615–631.zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    M. Anthony, P. L. Bartlett, Neural Network Learning, Theoretical Foundations, Cambridge University Press, 1999.Google Scholar
  3. 3.
    P. Bartlett, P. Long, Prediction, learning, uniform convergence, and scale-sensitive dimensions, J. Comput. System Sci. 56 (1998), 174–190.zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    R. M. Dudley, Uniform central limit theorems, Cambridge University Press, 1999.Google Scholar
  5. 5.
    E. Giné, J. Zinn, Gaussian charachterization of uniform Donsker classes of functions, Annals of Probability, 19 (1991), 758–782.zbMATHMathSciNetCrossRefGoogle Scholar
  6. 6.
    D. Haussler, Sphere packing numbers for subsets of Boolean n-cube with bounded Vapnik-Chervonenkis dimension, Journal of Combinatorial Theory A 69 (1995), 217–232.MathSciNetGoogle Scholar
  7. 7.
    W. Hoeffding, Probability inequalities for sums of bounded random variables, J. Amer. Statist. Assoc. 58 (1963), 13–30.zbMATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    M. Ledoux and M. Talagrand, Probability in Banach spaces, Springer, 1991.Google Scholar
  9. 9.
    S. Mendelson, Rademacher averages and phase transitions in Glivenko-Cantelli classes, IEEE transactions on Information Thery, Jan 2002.Google Scholar
  10. 10.
    S. Mendelson, Improving the sample complexity using global data To appear, IEEE transactions on Information Theory.Google Scholar
  11. 11.
    S. Mendelson, Geometric parameters of Kernel Machines, These proceedings.Google Scholar
  12. 12.
    V. Milman, G. Schechtman, Asymptotic theory of finite dimensional normed spaces, Lecture Notes in Math., vol. 1200, Springer Verlag, 1986.Google Scholar
  13. 13.
    A. Pajor, Sous espaces l 1 n des espaces de Banach, Hermann, Paris, 1985.zbMATHGoogle Scholar
  14. 14.
    G. Pisier, The volume of convex bodies and Banach space geometry, Cambridge University Press, 1989.Google Scholar
  15. 15.
    M. Talagrand, Type, infratype, and Elton-Pajor Theorem, Inventiones Math. 107 (1992), 41–59.zbMATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    M. Talagrand, Sharper bounds for Gaussian and empirical processes, Annals of Probability, 22(1), 28–76, 1994.zbMATHMathSciNetCrossRefGoogle Scholar
  17. 17.
    V. Vapnik, A. Chervonenkis, Necessary and sufficient conditions for uniform convergence of means to mathematical expectations, Theory Prob. Applic. 26(3), 532–553, 1971.CrossRefMathSciNetGoogle Scholar
  18. 18.
    A. W. Van-der-Vaart, J. A. Wellner, Weak convergence and Empirical Processes, Springer-Verlag, 1996.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Shahar Mendelson
    • 1
  • Roman Vershynin
    • 2
  1. 1.RSISEThe Australian National UniversityCanberraAustralia
  2. 2.Department of Mathematical SciencesUniversity of AlbertaEdmontonCanada

Personalised recommendations