Abstract
This is an outline of a modeling principle based upon the search for the shortest code length of the data, defined to be the stochastic complexity. This principle is generally applicable to statistical problems, and when restricted to the special exponential family, arising in the maximum entropy formalism with a set of moment constraints, it provides a generalization which permits the set of the constraints or their number to be optimized as well.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
van Campenhout, J.M. and Cover, T.M. (1981), ‘Maximum Entropy and Conditional Probability’, IEEE Trans. Inf. Thy, IT-27, Nr. 4, 483–489.
Chaitin, G.J. (1975), ‘A Theory of Program Size Formally Identical to to Information Theory’, J. ACM, 22, 329–340.
Chaitin, G.J. (1987), Algorithmic Information Theory, Cambridge University Press, Cambridge, 175 pages.
Dawid, A.P. (1984), ‘Present Position and Potential Developments: Some Personal Views, Statistical Theory, The Prequential Approach’, J. Royal Stat. Soc. Series A, 147, Part 2,278–292 (with discussions).
Hoel, A.E. and Kennard, R.W. (1970), “Ridge regression: Biased estimation for nonorthogonal problems’, Technometrics, 12, 55–68.
James, W. and Stein, C. (1961), ‘Estimation with quadratic loss’, Proc. 4th Berkeley Symp. 1, 363–379.
Jaynes, E. T. (1982), Papers on Probability, Statistics, and Statistical Physics, a reprint collection, D. Reidel, Dordrecht-Holland.
Rao, C.R. (1975), ‘Simultaneous Estimation of Parameters in Different Linear Models and Applications to Biometric Problems’, Biometrics, 31, 545–554.
Rao, C.R. (1981), ‘Prediction of future observations in polynomial growth curve models’, Proc. Indian Stat. Inst. Golden Jubilee Int. Con! on Statistics: Applications and New Directions., 512–520.
Kolmogorov, A.N. (1965), ‘Three Approaches to the Quantitative Definition of Information’, Problems of Information Transmission 1, 4–7.
Rissanen, J. (1978), ‘Modeling by shortest data description’, Automatica, 14, pp. 465–471.
Rissanen, J. (1983), ‘A Universal Prior for Integers and Estimation by Minimum Description Length’, Annals of Statistics, 11, No.2, 416–431.
Rissanen, J. (1984), ‘Universal Coding, Information, Prediction, and Estimation’, IEEE Trans. Inf. Thy, IT-30, Nr. 4, 629–636.
Rissanen, J. (1986a), ‘Stochastic Complexity and Modeling’, Annals of Statistics, 14, No 3, 1080–1100.
Rissanen, J. (1986b), ‘A Predictive Least Squares Principle’, IMA J. of Math. Control and Information, 3, Nos 2–3, 211–222.
Rissanen, J. (1987a), ‘Stochastic Complexity’, Journal of the Royal Statistical Society, Series B, 49, No.3, 223–265 (with discussions).
Rissanen, J. (1987b), ‘Stochastic Complexity and the MDL Principle’, Econometric Reviews, 6, nr 1, 85–102.
Rissanen, J. (1987c), ‘Complexity and Information in Contingency Tables’, Proceedings of The Second International Tampere Conference in Statistics, June 1–4, Tampere, Finland.
Schwarz, G. (1978), ‘Estimating the Dimension of a Model’, Annals of Statistics, 6, 461–464.
Solomonoff, R.J. (1964), ‘A Formal Theory of Inductive Inference’. Part I, Information and Control 7, 1–22; Part II, Information and Control 7, 224–254.
Wallace, C.S. and Boulton, D.M. (1968), ‘An Information Measure for Classification’, The Computer Journal, 11, No.2, 185–194.
Wallace, C.S. and Freeman, P.R. (1987), ‘Estimation and Inference by Compact Coding’, Journal of the Royal Statistical Society, Series B, 49, No.3, 239–265 (with discussions).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1988 Kluwer Academic Publishers
About this chapter
Cite this chapter
Rissanen, J. (1988). Stochastic Complexity and the Maximum Entropy Principle. In: Erickson, G.J., Smith, C.R. (eds) Maximum-Entropy and Bayesian Methods in Science and Engineering. Fundamental Theories of Physics, vol 31-32. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-3049-0_7
Download citation
DOI: https://doi.org/10.1007/978-94-009-3049-0_7
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-7871-9
Online ISBN: 978-94-009-3049-0
eBook Packages: Springer Book Archive