Abstract
Supplementing Vovk and V’yugin’s ‘if’ statement, we show that Bayesian compression provides the best enumerable compression for parameter-typical data if and only if the parameter is Martin-Löf random with respect to the prior. The result is derived for uniformly discretizable statistical models, introduced here. They feature the crucial property that given a discretized parameter, we can compute how much data is needed to learn its value with little uncertainty. Exponential families and certain nonparametric models are shown to be uniformly discretizable.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Vovk, V.G., V’yugin, V.V.: On the empirical validity of the Bayesian method. J. Roy. Statist. Soc. B 55, 253–266 (1993)
Vovk, V.G., V’yugin, V.V.: Prequential level of impossibility with some applications. J. Roy. Statist. Soc. B 56, 115–123 (1994)
Vitányi, P., Li, M.: Minimum description length induction, Bayesianism and Kolmogorov complexity. IEEE Trans. Inform. Theor. 46, 446–464 (2000)
Takahashi, H.: On a definition of random sequences with respect to conditional probability. Inform. Comput. 206, 1375–1382 (2008)
Gács, P.: On the symmetry of algorithmic information. Dokl. Akad. Nauk SSSR 15, 1477–1480 (1974)
Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, Heidelberg (1997)
van Lambalgen, M.: Random Sequences. PhD thesis, Universiteit van Amsterdam (1987)
Barron, A., Rissanen, J., Yu, B.: The minimum description length principle in coding and modeling. IEEE Trans. Inform. Theor. 44, 2743–2760 (1998)
Grünwald, P.D.: The Minimum Description Length Principle. MIT Press, Cambridge (2007)
Yu, B., Speed, T.P.: Data compression and histograms. Probab. Theor. Rel. Fields 92, 195–229 (1992)
Hopcroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages and Computation. Addison-Wesley, Reading (1979)
Elias, P.: Universal codeword sets and representations for the integers. IEEE Trans. Inform. Theor. 21, 194–203 (1975)
Barron, A.R.: Logically Smooth Density Estimation. PhD thesis, Stanford University (1985)
Dawid, A.: Statistical theory: The prequential approach. J. Roy. Statist. Soc. A 147, 278–292 (1984)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, Chichester (1991)
Li, L., Yu, B.: Iterated logarithmic expansions of the pathwise code lengths for exponential families. IEEE Trans. Inform. Theor. 46, 2683–2689 (2000)
Barndorff-Nielsen, O.E.: Information and Exponential Families. Wiley, Chichester (1978)
Jeffreys, H.: Theory of Probability, 3rd edn. Oxford University Press, Oxford (1961)
Dębowski, Ł.: On the vocabulary of grammar-based codes and the logical consistency of texts (2008) E-print, http://arxiv.org/abs/0810.3125
Csiszar, I., Shields, P.C.: The consistency of the BIC Markov order estimator. Ann. Statist. 28, 1601–1619 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Dębowski, Ł. (2009). Computable Bayesian Compression for Uniformly Discretizable Statistical Models. In: Gavaldà, R., Lugosi, G., Zeugmann, T., Zilles, S. (eds) Algorithmic Learning Theory. ALT 2009. Lecture Notes in Computer Science(), vol 5809. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04414-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-04414-4_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04413-7
Online ISBN: 978-3-642-04414-4
eBook Packages: Computer ScienceComputer Science (R0)