Skip to main content
SpringerLink
Log in

Kolmogorov Complexity in Perspective Part I: Information Theory and Randomness

  • Chapter
  • First Online:
Constructivity and Computability in Historical and Philosophical Perspective

Part of the book series: Logic, Epistemology, and the Unity of Science ((LEUS,volume 34))

Abstract

We survey diverse approaches to the notion of information: from Shannon entropy to Kolmogorov complexity. Two of the main applications of Kolmogorov complexity are presented: randomness and classification. The survey is divided in two parts in the same volume. Part I is dedicated to information theory and the mathematical formalization of randomness based on Kolmogorov complexity. This last application goes back to the 1960s and 1970s with the work of Martin-Löf, Schnorr, Chaitin, Levin, and has gained new impetus in the last years.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For an easy proof, identify such a triple with a binary word with six letters 0 for the six balls and two letters 1 to mark the partition in the three cells.

  2. 2.

    For other European languages with a lot of diacritic marks, one has to consider the 256 codes of Extended ASCII which have length 8. And for non European languages, one has to turn to the 65 536 codes of Unicode which have length 16.

  3. 3.

    Berry’s paradox is mentioned by Bertrand Russell (1908, p.222 or 150), who credited G.G. Berry, an Oxford librarian, for the suggestion.

  4. 4.

    Through the works of Alonzo Church (via lambda calculus), Alan Mathison Turing (via Turing machines) and Kurt Gödel and Jacques Herbrand (via Herbrand-Gödel systems of equations) and Stephen Cole Kleene (via the recursion and minimization operators).

  5. 5.

    In French, Lacombe (1960) used the expression semi-fonction semi-récursive.

  6. 6.

    A seed for computer virology, cf. Bonfante et al. (2006).

  7. 7.

    Delahaye’s books (Delahaye 1999, 2006) present a very attractive survey on Kolmogorov complexity.

  8. 8.

    Kolmogorov is one of the rare probabilists – up to now – not to believe that Kolmogorov’s axioms for probability theory do constitute the last word about formalizing randomness…

  9. 9.

    Li and Vitányi (1997), pp. 89–92, gives a detailed account of when who did what.

References

  • Becher, V., Figueira, S., Nies, A., & Picchi, S. (2005). Program size complexity for possibly infinite computations. Notre Dame Journal of Formal Logic, 46(1), 51–64.

    Article  Google Scholar 

  • Becher, V., Figueira, S., Grigorieff, S., & Miller, J. (2006). Random reals and halting probabilities. The Journal of Symbolic Logic, 71(4), 1411–1430.

    Article  Google Scholar 

  • Bienvenu, L., & Merkle, W. (2007). Reconciling data compression and Kolmogorov complexity. In ICALP 2007, Wroclaw (LNCS, Vol. 4596, pp. 643–654).

    Google Scholar 

  • Bienvenu, L., Merkle, W., & Shen, A. (2008). A simple proof of Miller-Yu theorem. Fundamenta Informaticae, 83(1–2), 21–24.

    Google Scholar 

  • Bonfante, G., Kaczmarek, M., & Marion, J.-Y. (2006). On abstract computer virology: From a recursion-theoretic perspective. Journal in Computer Virology, 1(3–4), 45–54.

    Article  Google Scholar 

  • Calude, C., & Jürgensen, H. (2005). Is complexity a source of incompleteness? Advances in Applied Mathematics, 35, 1–15.

    Article  Google Scholar 

  • Chaitin, G. (1966). On the length of programs for computing finite binary sequences. Journal of the ACM, 13, 547–569.

    Article  Google Scholar 

  • Chaitin, G. (1969). On the length of programs for computing finite binary sequences: Statistical considerations. Journal of the ACM, 16, 145–159.

    Article  Google Scholar 

  • Chaitin, G. (1971). Computational complexity & Gödel incompleteness theorem. ACM SIGACT News, 9, 11–12.

    Article  Google Scholar 

  • Chaitin, G. (1974). Information theoretic limitations of formal systems. Journal of the ACM, 21, 403–424.

    Article  Google Scholar 

  • Chaitin, G. (1975). A theory of program size formally identical to information theory. Journal of the ACM, 22, 329–340.

    Article  Google Scholar 

  • Delahaye, J. P. (1999). Information, complexité, hasard (2nd ed.). Paris: Hermès.

    Google Scholar 

  • Delahaye, J. P. (2006). Complexités: Aux limites des mathématiques et de l’informatique. Montreal: Belin-Pour la Science.

    Google Scholar 

  • Durand, B., & Zvonkin, A. (2004). Complexité de Kolmogorov. In E. Charpentier, A. Lesne, & N. Nikolski (Eds.), L’héritage de Kolmogorov en mathématiques (pp. 269–287). Berlin: Springer.

    Google Scholar 

  • Fallis, D. (1996). The source of Chaitin’s incorrectness. Philosophia Mathematica, 4, 261–269.

    Article  Google Scholar 

  • Feller, W. (1968). Introduction to probability theory and its applications (Vol. 1, 3rd ed.). New York: Wiley.

    Google Scholar 

  • Ferbus-Zanda, M., & Grigorieff, S. (2004). Is randomnes native to computer science? In G. Paun, G. Rozenberg, & A. Salomaa (Eds.), Current trends in theoretical computer science (pp. 141–179). Singapore: World Scientific.

    Chapter  Google Scholar 

  • Ferbus-Zanda, M., & Grigorieff, S. (2006). Kolmogorov complexity and set theoretical representations of integers. Mathematical Logic Quarterly, 52(4), 381–409.

    Article  Google Scholar 

  • Gács, P. (1974). On the symmetry of algorithmic information. Soviet Mathematics Doklady, 15, 1477–1480.

    Google Scholar 

  • Gács, P. (1980). Exact expressions for some randomness tests. Zeitschrift für Mathematische Logik und Grundlagen der Mathematik, 26, 385–394.

    Article  Google Scholar 

  • Gács, P. (1993). Lectures notes on descriptional complexity and randomness (pp. 1–67). Boston: Boston University. http://cs-pub.bu.edu/faculty/gacs/Home.html.

  • Huffman, D. A. (1952). A method for construction of minimum-redundancy codes. Proceedings IRE, 40, 1098–1101.

    Google Scholar 

  • Knuth, D. (1981). The art of computer programming. Volume 2: Semi-numerical algorithms (2nd ed.). Reading: Addison-Wesley.

    Google Scholar 

  • Kolmogorov, A. N. (1933). Grundbegriffe der Wahscheinlichkeitsrechnung. Berlin: Springer [Foundations of the theory of probability, Chelsea, 1956].

    Google Scholar 

  • Kolmogorov, A. N. (1963). On tables of random numbers. Sankhya, The Indian Journal of Statistics, Series A, 25, 369–376.

    Google Scholar 

  • Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1–7.

    Google Scholar 

  • Kolmogorov, A. N. (1983). Combinatorial foundation of information theory and the calculus of probability. Russian Mathematical Surveys, 38(4), 29–40.

    Article  Google Scholar 

  • Lacombe, D. (1960). La théorie des fonctions récursives et ses applications. Bulletin de la Société Mathématique de France, 88, 393–468.

    Google Scholar 

  • van Lambalgen, M. (1989). Algorithmic information theory. The Journal of Symbolic Logic, 54(4), 1389–1400.

    Article  Google Scholar 

  • Levin, L. (1973). On the notion of a random sequence. Soviet Mathematics Doklady, 14, 1413–1416.

    Google Scholar 

  • Levin, L. (1974). Laws of information conservation (non-growth) and aspects of the foundation of probability theory. Problems of Information Transmission, 10(3), 206–210.

    Google Scholar 

  • Li, M., & Vitányi, P. (1997). An introduction to Kolmogorov complexity and its applications (2nd ed.). New York: Springer.

    Book  Google Scholar 

  • Martin-Löf, P. (1966). The definition of random sequences. Information and Control, 9, 602–619.

    Article  Google Scholar 

  • Martin-Löf, P. (1971). Complexity of oscilations in infinite binary sequences. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 19, 225–230.

    Article  Google Scholar 

  • Miller, J., & Yu, L. (2008). On initial segment complexity and degrees of randomness. Transactions of the American Mathematical Society, 360, 3193–3210.

    Article  Google Scholar 

  • Von Mises, R. (1919). Grundlagen der wahrscheinlichkeitsrechnung. Mathematische Zeitschrift, 5, 52–99.

    Article  Google Scholar 

  • Von Mises, R. (1939). Probability, statistics and truth. New York: Macmillan. (Reprinted: Dover 1981)

    Google Scholar 

  • Moschovakis, J. R. (1993). An intuitionistic theory of lawlike, choice and lawless sequences. In J. Oikkonen & J. Väänänen (Eds.), Logic colloquium ’90, Helsinski (Lecture notes in logic, Vol. 2, pp. 191–209). Berlin: Springer.

    Google Scholar 

  • Moschovakis, J. R. (1994). More about relatively lawless sequences. The Journal of Symbolic Logic, 59(3), 813–829.

    Article  Google Scholar 

  • Moschovakis, J. R. (1996). A classical view of the intuitionistic continuum. Annals of Pure and Applied Logic, 81, 9–24.

    Article  Google Scholar 

  • Von Neumann, J. (1951). Various techniques used in connection with random digits. In A. S. Householder, G. E. Forsythe, & H. H. Germond (Eds.), Monte Carlo method (National bureau of standards applied mathematics series, Vol. 12, pp. 36–38). Washington, D.C.: U.S. Government Printing Office.

    Google Scholar 

  • Nies, A., Stephan, F., & Terwijn, S. A. (2005). Randomness, relativization and Turing degrees. The Journal of Symbolic Logic, 70(2), 515–535.

    Article  Google Scholar 

  • Paul, W. (1979). Kolmogorov’s complexity and lower bounds. In L. Budach (Ed.), Proceedings of 2nd international conference on fundamentals of computation theory, Berlin/Wendisch-Rietz (pp. 325–334). Berlin: Akademie.

    Google Scholar 

  • Raatikainen, P. (1998). On interpreting Chaitin’s incompleteness theorem. Journal of Philosophical Logic, 27(6), 569–586.

    Article  Google Scholar 

  • Russell, B. (1908). Mathematical logic as based on the theory of types. American Journal of Mathematics, 30, 222–262. (Reprinted in From Frege to Gödel: A source book in mathematical logic, 1879–1931, pp. 150–182, J. van Heijenoort, Ed., 1967)

    Google Scholar 

  • Schnorr, P. (1971a). A unified approach to the definition of random sequences. Mathematical Systems Theory, 5, 246–258.

    Article  Google Scholar 

  • Schnorr, P. (1971b). Zufälligkeit und Wahrscheinlichkeit (Lecture notes in mathematics, Vol. 218). Berlin/New York: Springer.

    Google Scholar 

  • Schnorr, P. (1973). A process complexity and effective random tests. Journal of Computer and System Sciences, 7, 376–388.

    Article  Google Scholar 

  • Shannon, C. E. (1948). The mathematical theory of communication. Bell System Technical Journal, 27, 379–423.

    Article  Google Scholar 

  • Soare, R. (1996). Computability and recursion. Bulletin of Symbolic Logic, 2, 284–321.

    Article  Google Scholar 

  • Solomonov, R. (1964a). A formal theory of inductive inference, part I. Information and Control, 7, 1–22.

    Article  Google Scholar 

  • Solomonov, R. (1964b). A formal theory of inductive inference, part II. Information and Control, 7, 224–254.

    Article  Google Scholar 

  • Zvonkin, A., & Levin, L. (1970). The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms. Russian Mathematical Surveys, 6, 83–124.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LIAFA, CNRS & Université Paris Diderot – Paris 7, Case 7014, 75205, Paris Cedex 13, France

    Marie Ferbus-Zanda & Serge Grigorieff

Authors
  1. Marie Ferbus-Zanda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Serge Grigorieff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Serge Grigorieff .

Editor information

Editors and Affiliations

  1. Institut d’histoire et de philosophie des sciences et des techniques (IHPST; CNRS-Paris1-ENS), Paris, France

    Jacques Dubucs

  2. Institut d’histoire et de philosophie des sciences et des techniques (IHPST; CNRS-Paris1-ENS), Paris, France

    Michel Bourdeau

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Ferbus-Zanda, M., Grigorieff, S. (2014). Kolmogorov Complexity in Perspective Part I: Information Theory and Randomness. In: Dubucs, J., Bourdeau, M. (eds) Constructivity and Computability in Historical and Philosophical Perspective. Logic, Epistemology, and the Unity of Science, vol 34. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-9217-2_3

Download citation

Publish with us

Policies and ethics

Search

Navigation