Lower Bounds on Expected Redundancy for Classes of Continuous Markov Sources

Conference paper


A nonasymptotic lower bound is derived for the per symbol expected redundancy based on n observations from a continuous dth order Markov source. The bound is minimax over a Lipschitz class of such sources. The constant in the lower bound is explicitly described in terms of d. By making d go to infinity with n at an appropriate rate, it is shown that no universal rate of expected redundancy exists for the class of Markov sources of all orders, and this provides an alternative and simpler derivation of a similar result by Shields. Similar results are obtained for the Kullback Leibler estimation error for the joint density of d-tuples based on n observations from a continuous (d — 1)st order Markov source.


Joint Density Predictive Density Finite Alphabet Prefix Code Lipschitz Class 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Assouad, P. (1983). Deux remarques sur l’estimation. Comptes Rendus de l’Academie des Sciences de Paris. 296, 1021–1024.MathSciNetMATHGoogle Scholar
  2. Birge, L. (1985). Non-asymptotic minimax risk for Hellinger balls. Probability and Mathematical Statistics 5, 21–29.MathSciNetMATHGoogle Scholar
  3. Clarke, B. S. and Barron, A. R. (1990). Information theoretic asymptotics of Bayes methods. IEEE Trans. Infor. Th. 36, 453–471.MathSciNetMATHCrossRefGoogle Scholar
  4. Devroye, L. (1983). On arbitrary slow rates of global convergence in density estimation. Z. Wahrsch. Verw. Gebiete 62, 475–483.MathSciNetMATHCrossRefGoogle Scholar
  5. Devroye, L. (1987). A course in density estimation. Progress in probability and statistics 14, Birkhauser.Google Scholar
  6. Donoho, D. L., Liu, R. C., and MacGibbon, B. (1990). Minimax risk over hyperrectangles, and implications. Ann. Statist. 18, 1416–1437.MathSciNetMATHCrossRefGoogle Scholar
  7. Doob, J. L. (1953). Stochastic Processes. Wiley, New York.MATHGoogle Scholar
  8. Rissanen, J. (1986). Stochastic complexity and modeling. Ann. Statist. 14, 1080–1100.MathSciNetMATHCrossRefGoogle Scholar
  9. Shields, P. C. (1991). Universal redundancy rates don’t exist. Preprint.Google Scholar
  10. Yu, B. and Speed, T. P. (1992). Data compression and histograms. Probab. Th. Rel. Fields 2, 195–229.MathSciNetCrossRefGoogle Scholar
  11. Yu, B. (1992). A note on the nonexistence of universal redundancy rates. (in preparation)Google Scholar

Copyright information

© Springer-Verlag New York, Inc. 1994

Authors and Affiliations

  • Bin Yu
    • 1
  1. 1.University of Wisconsin-MadisonUSA

Personalised recommendations