Stochastic Complexity and Statistical Inference

  • Jorma Rissanen
Conference paper
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 83)


We regard the fundamental task in statistics to be to “understand” a given set of observations. By “understanding” we mean in broad terms the discovery of the various constraints and regu-larities that restrict the data. An attempt to “understand” the data presents us with a dilemma. As our main source of information we only have the observed data, to which we, in the final analysis, end up in fitting a model, rather like passing a smooth curve through a scatter of data points. At the same time we realize that too good a fit is not what we want; after all, we can always get a perfect fit by just adding enough parameters to the model. Instead, intuitively, we want a model which captures the vaguely defined underlying regular features in the data, which we hope will hold even in the future and hence will enable us to make reliable predictions.


Prediction Error Code Length Parent Distribution Binary Digit Traditional Statistic 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akaike, H. (1974) “A new look at the statistical model identification’’, IEEE Trans. AC-19, 716–723.Google Scholar
  2. 2.
    Dawid, A.P. (1984), ’’Present Position and Potential Developments: Some Personal Views, Statistical Theory, The Prequential Approach’’, J. Royal Stat. Soc. Series A, Vol. 147, Part 2, 278–292.Google Scholar
  3. 3.
    Geisser, S. and Eddy, W. (1979), “A Predictive Approach to Model Selection’’, J. American Stat. Ass., Vol. 74, Nr. 365, 153–160.CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Chaitin, G.J. (1975), ’’A Theory of Program Size Formally Identical to Information Theory’’, JACM, Vol. 22, No. 3, 329–340.CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Hjorth, U. (1982), “Model Selection and Forward Validation”, Scand. J. Stat. 9, 95–105. 416–431.Google Scholar
  6. 6.
    Kalman, R.E., (1983), Identifiability and Modeling in Econometrics’’, in Developments in Statistics, 4, P. Krishnaiah ed., 97–134, Academic Press.Google Scholar
  7. 7.
    Kolmogorov, A.N. (1965), “Three Approaches to the Quantitative Definition of Informa-tion’’, Problems of Information Transmission 1, 4–7.MathSciNetGoogle Scholar
  8. 8.
    Kullback, S. (1959), Information Theory and Statistics. John Wiley & Sons.Google Scholar
  9. 9.
    Quenouille, M.H. (1956), “Notes on Bias in Estimation’’, Biometrika, 43, 353–360.CrossRefzbMATHMathSciNetGoogle Scholar
  10. 10.
    Rao, C.R. (1981), “Prediction of future observations in polynomial growth curve models’’, Proc. Indian Stat. Inst. Golden Jubilee Int. Conf. on Statistics: Applications and New Di-rections., 512–520.Google Scholar
  11. 11.
    Rissanen, J. (1978), “Modeling by shortest data description’’, Automatica, Vol. 14, pp. 465–471.CrossRefzbMATHGoogle Scholar
  12. 12.
    Rissanen, J. (1983), “A Universal Prior for Integers and Estimation by Minimum De-scription Length’’, Ann. of Statistics, Vol. 11, No. 2Google Scholar
  13. 13.
    Rissanen, J. (1984), “Universal Coding, Information, Prediction, and Estimation”, IEEE Trans. Inf. Theory, Vol. IT-30, Nr. 4, 629–636.CrossRefMathSciNetGoogle Scholar
  14. 14.
    Rissanen, J. (1985), “Minimum Description Length Principle’’, Encyclopedia of Statistical Sciences, Vol. V, (S. Kotz & N. L. Johnson eds.), pp. 523–527. John Wiley and Sons, New York.Google Scholar
  15. 15.
    Rissanen, J. (1986), ’’Stochastic Complexity and Modeling’’, Ann. of Statistics, September (1986) (to appear).Google Scholar
  16. 16.
    Rissanen, J. (1986), ’’The Selection-of-Variables Problem’’, (to appear).Google Scholar
  17. 17.
    Solomonoff, R.J. (1964), “A Formal Theory of Inductive Inference”. Part I, Information and Control 7, 1–22; Part II, Information and Control 7, 224–254.Google Scholar
  18. 18.
    Stone, M. (1974), “Cross-Validatory Choice and Assessment of Statistical Predictions’’, J. Royal Stat. Soc., Ser. B, 36, 111–147.Google Scholar
  19. 19.
    Stone, M. (1977), “Asymptotics for and against Cross-Validation”, Biometrika 64, 29–35.CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1986

Authors and Affiliations

  • Jorma Rissanen
    • 1
  1. 1.San JoseUSA

Personalised recommendations