Abstract
We survey diverse approaches to the notion of information: from Shannon entropy to Kolmogorov complexity. Two of the main applications of Kolmogorov complexity are presented: randomness and classification. The survey is divided in two parts in the same volume.
Part II is dedicated to the relation between logic and information system, within the scope of Kolmogorov algorithmic information theory. We present a recent application of Kolmogorov complexity: classification using compression, an idea with provocative implementation by authors such as Bennett, Vitányi and Cilibrasi among others. This stresses how Kolmogorov complexity, besides being a foundation to randomness, is also related to classification. Another approach to classification is also considered: the so-called “Google classification”. It uses another original and attractive idea which is connected to the classification using compression and to Kolmogorov complexity from a conceptual point of view. We present and unify these different approaches to classification in terms of Bottom-Up versus Top-Down operational modes, of which we point the fundamental principles and the underlying duality. We look at the way these two dual modes are used in different approaches to information system, particularly the relational model for database introduced by Codd in the 1970s. These operational modes are also reinterpreted in the context of the comprehension schema of axiomatic set theory ZF. This leads us to develop how Kolmogorov’s complexity is linked to intensionality, abstraction, classification and information system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
A byte is a sequence of eight binary digits. It can also be seen as a number between 0 and 255.
- 4.
The notion of information content of an object is detailed in Part I. According to Kolmogorov, this is, by definition, the algorithmic complexity of that object.
- 5.
We come back in Sect. 4.4 below on information processing with statistics.
- 6.
A formal definition of compressors is given in Part I.
- 7.
The notion of black box is a scientific concept introduced by Norbert Wiener (1948). This concept is one of the fundamental principles of Cybernetics. It is issued from the multidisciplinary exchanges during the Macy conferences which were held in New-York, 1942–1953. Indeed, the emergence of cybernetics and information theory owes much to these conferences.
- 8.
Point 4, Sect. 4.3.2 relativizes the obtained results.
- 9.
Irving (1978).
- 10.
Véronis (2005).
- 11.
Kleene formally and completely characterizes the notion of recursive function (also called computable function), by adding the minimization schema (1936) to the composition and recursion schemas – these two last schemas characterize the primitive recursive functions which constitute a proper subclass of the class of computable functions: the Ackermann function (1928) is computable but not primitive recursive. From a programming point of view, the minimization schema corresponds to the while loop (while F(x) do P where F(x) is a Boolean valued property and P is a program) (cf. the book by Shoenfield (2001)).
- 12.
This is the case for tail-recursion definitions. In some programming languages such as LISP such tail-recursion programs are generally executed (when the programs executor is well written) in an iterative way. Tail-recursion programs represent a limit case between iterative programs and recursive programs.
- 13.
Depending on how much abstraction is wanted (or how much refinement is wanted), a text will be represented by a binary word (the blank spaces separating words being also encoded as special characters) or by a sequence of binary strings (each word in the text being represented by a string in the sequence). In this paper, we mostly consider encodings of texts with binary words (in particular, for the examples) and not sequences of binary words, and we consider sets of such texts.
- 14.
It is one way of seeing things! The one reflected by the Anglo-Saxon terminology “top-down”. What is essential is that texts are apprehended from the outside, in opposition to apprehension from the inside.
- 15.
With a purely syntactic automatic translator, such as the one in Google, one can get results like the following one: “Alonzo Church” translated as “Église d’Alonzo” (i.e. church in Alonzo)!
- 16.
Let us mention that a new concept emerged: that of thesaurus which is somehow an abstract semantics related to classification. A thesaurus is a particular type of documentary language (yet a new concept) which, for a given domain, lists along a graph words and their different relations: synonymy, metaphor, hierarchy, analogy, comparison, etc. Thus, a thesaurus is a kind of normalized and classified vocabulary for a particular domain. Otherwise said, the words a thesaurus contains, constitute an hierarchical dictionary of keywords for the considered domain. One can possibly add definitions of words or consider the classification of words (according to the future usage of the thesaurus) to be sufficient. It is a remarkable tool. First used for disciplines around documentation and for large databanks, it is now used almost everywhere. To build a thesaurus, one follows a bottom-up or a top-down mode or mixes both modes, exactly like in the case of keywords. More details on the notion of thesaurus in the section devoted to databases (cf. Sect. 4.4.2).
- 17.
The abstract notion of isomorphism in mathematics is a form of duality. Some dualities are not reduced to isomorphisms. Typically, Boolean algebras with the complement operation (in addition to additive and multiplicative operations) contain an internal duality and are the basis of deep dualities such as Stone duality which links the Boolean algebras family and some topological spaces. The complement operation confronts us to many problems and deep results…
- 18.
Since Gottlob Frege invention, at the end of the nineteenth century, of the mathematical logic and the formalization of the mathematical language that results from it, mathematicians have de facto to deal with two distinct categories of mathematical symbols: the function symbols and relational symbols (or predicate symbols) in complement of symbols representing objects. To each of these two large classes of symbols respectively correspond algorithms and information systems.
- 19.
The information systems in which we highlight a type of programming that we named relational programming in a research report: Ferbus-Zanda (1986). We present in this paper the link between functional programming and relational programming.
- 20.
Some exhaustive descriptions of algorithms about trading and taxes date from Babylonia (2000 BC to 200 AC). Information systems really emerged with mecanography (end of nineteenth century) and the development of computer science. However, there are far earlier examples of what we could now call information systems since they show a neat organization and presentation of data on a particular subject: for instance, the Roman census.
- 21.
Observe that these semantics correspond respectively to Arend Heyting’s semantics and Alfred Tarski’s semantics.
- 22.
Multics was the first important operating system to store files as nodes in a tree (in fact a graph). Created in 1965, it has been progressively replaced since 1980 by Unix. Derived from Multics, it includes a new feature: multiple users management. Now, all operating systems are based on Unix. Multics was a turning point in the problem of data storage: until now, one speaks of hierarchical model and net model. But, in fact, these “models” have been recognized as models only after Codd introduced the relational model! Finally, observe that the graph structure of the Web also comes from the organization of files with Multics.
- 23.
This combinatory logic has much to do with the programming language Cobol created in 1959.
- 24.
The dedication in his last book (Codd 1990) is as follows: To fellow pilots and aircrew in the Royal Air Force during War II and the dons at Oxford. These people were the source of my determination to fight for what I believe was right during the 10 or more years in which government, industry, and commerce were strongly opposed to the relational approach to database management.
- 25.
Oracle is now a company worthing billions dollars.
- 26.
UML (Unified Modelling Language) is a formal language, which is used as a method for modeling in many topics, in particular, in computer science with databases and Object-Oriented Conception (OOC) – in fact, this is the source of UML.
- 27.
Ferbus-Zanda (In preparation-a).
- 28.
One should rather say that keywords – used with web browsers – constitute very elementary database queries (of course, database queries are much older than the Web which emerged only in the 1990s).
- 29.
Lines are usually presented as tuples but, conceptually, this is not correct: in Codd’s relational model there is no order between the lines nor between the columns. Codd insisted on that point. In fact, conceptually and in practice, this is quite important: queries should be expressed as conditions (i.e. formulas) in the relational algebra, using names of attributes and of tables. For example, it means that queries cannot ask for the first or twentieth line (or column).
- 30.
In fact, any such “complete” dictionary is necessarily circular: a word a is defined using the word b which is itself defined with other words themselves defined in the dictionary. It requires some knowledge external to the dictionary to really grasp the “meaning” of words. Note that this incompleteness is more or less “hidden”. On the other hand, in a synonym dictionary, the structure essentially relies on circular definitions. This is less apparent with paper dictionaries: for a given word, there will be only references to its synonyms. However, with digital dictionaries, this circularity is really striking: links “carry” the reader to the diverse synonyms and can be implemented with pointers.
- 31.
Codd was strongly opposed to any addition from the object approach to the relational model. Indeed, the so-called “First Normal Form” (due to Codd) formally forbids the possibility of an attribute structured as a list, a tree or a graph (which is exactly what OOC would do). When he elaborated his model, this was a reasonable choice: the object approach is quite destructuralization while Codd’s approach was a structuralization one. Let us mention that Codd also opposed Chen’s Entity/Relationship model (nobody’s perfect)!
- 32.
Ibid. Note 27.
- 33.
An implementation of Codd’s relational model for databases is a DBMS (DataBase Management System). Any DBMS includes an interpreter of the language SQL (such an interpreter is, in fact, an implementation of Codd’s relational algebra, the fundamental calculus in this theoretical model).
- 34.
More formally: \(\forall x\ \exists y\ \ \forall z\ \ (z \in y\longleftrightarrow (z \in x \wedge \mathcal{P}(z)))\).
- 35.
One can also constraint in different ways this property \(\mathcal{P}\). In particular, to avoid circularities such as the one met when \(\mathcal{P}\) contains some universal quantification on sets, hence quantifies on the set it is supposed to define (this was called impredicativity by Henri Poincaré).
- 36.
Russel’s paradox insures that the following extension of the comprehension schema is contradictory: \(\exists y\ \ \ y =\{\ z\mbox{; }\mathcal{P}(z)\ \}\), i.e. \(\exists y\ \ \forall z\ \ (z \in y\longleftrightarrow \mathcal{P}(z))\). Indeed, consider the property \(\mathcal{P}\) such that \(\mathcal{P}(u)\) if and only if \(u\notin u\), then we get y ∈ y if and only if \(y\notin y\).
- 37.
- 38.
Recall that once an information has been put on the Web, it is almost impossible to remove it…
- 39.
\(K_{\varphi }(y) =\min \{ \vert p\vert:\varphi (p) = y\}\) where \(K_{\varphi }: \mathcal{O}\rightarrow \mathbb{N}\) where \(\varphi:\{ 0, 1\}^{{\ast}}\rightarrow \mathcal{O}\) is a partial function (intuitively \(\varphi\) executes program p as a LISP interpreter does) and \(\mathcal{O}\) is a set endowed with a computability structure. We take the convention that \(\min \varnothing = +\infty \) (cf. Part I).
- 40.
Wiener’s book (1948), raised many controversies (and Wiener exchanged a lot with von Neumann about it).
- 41.
A subject going back to Charles Sanders Pierce (1839–1914).
- 42.
Recall that the very original idea on which Vitanyi based the classification using compression is to compute an approximate value of this complexity via usual compression algorithms.
- 43.
As in the two forthcoming (technical) papers: Ferbus-Zanda, M. & Grigorieff, S. Kolmogorov complexity and higher order set theoretical representations of integers and Ferbus-Zanda, M. & Grigorieff, S. Infinite computations, Kolmogorov complexity and base dependency.
- 44.
We study the duality of functional and relational in Ferbus-Zanda (In preparation-c). The relation between ASMs and Kolmogorov complexity and the reconsideration of these theories with a relational point of view are developed in a forthcoming paper: Ferbus-Zanda (In preparation-b).
- 45.
This is what we started in Ibid. Note 44.
- 46.
Personal communication while he was visiting our university in Paris.
- 47.
Howard (1980).
- 48.
Joachim Lambeck also published in the 1970s, about this correspondence concerning the combinatorics of the cartesian closed categories and the intuitionist propositional logic. Note that Nicolaas Debruijn (Authomath system) and Per Martin-Löf had also a decisive influence upon the original Curry-Howard isomorphism. Martin-Löf saw the typed lambda calculus, which he was developing, as a (real) programming language (cf. Martin-Löf 1979). Similarly, Thierry Coquand elaborated the theory of Construction on which is based the Coq system, initially developed by Gérard Huet at the INRIA (France) in the 1980s. (See also Note 50.)
- 49.
The notion of cut in the Sequent Calculus and the Natural Deduction is a fundamental notion in proof theory. It was introduced by Gerhard Gentzen in the 1930s – and these two logical calculus too. In some cases one can see a cut as a form of abstraction where a multiplicity of particular cases are replaced by a general case. In the sequent calculus, a cut is defined by means of the cut rule, which is a generalization of the Modus Ponens. The fundamental result of Gentzen is the Hauptsatz, which states that every proof in the sequent calculus can be transformed in a proof of the same conclusion without using this cut rule.
- 50.
In fact, Church’s original λ-calculus can be extended with constants and new reduction rules in order to extend to classical logic with the notion of continuation, Thimothy Griffin, 1990 – and possibly classical logic plus axioms such as the axiom of dependent choice – the original Curry-Howard correspondence between intuitionist logic and usual typed λ-calculus. This is the core of Jean-Louis Krivine’s work who introduced some of those fundamental constants which have a deep computer science significance (cf. Krivine 2003).
- 51.
A redex in a λ-term t is a subterm of t on which a one-step reduction can be readily applied, for instance, with β-reduction, this is a subterm of the form ((λx.u)v) and it reduces to u[v∕x], which is the term u in which every occurrence of x is replaced by v (some variable capture problems have to be adequately avoided).
- 52.
Let us mention the remarkable collection of unpublished papers and notes (Dijkstra 1982).
References
Bennett, C. (1988). Logical depth and physical complexity. In R. Herken (Ed.), In the universal turing machine: A half-century survey (pp. 227–257). Oxford University Press: New York.
Bennett, C., Gács, P., Li, M., Vitányi, P., & Zurek, W. (1998). Information distance. IEEE Transactions on Information Theory, 44(4), 1407–1423.
Chaitin, G. (1966). On the length of programs for computing finite binary sequences. Journal of the ACM, 13, 547–569.
Chaitin, G. (1969). On the length of programs for computing finite binary sequences: Statistical considerations. Journal of the ACM, 16, 145–159.
Chaitin, G. (1975). A theory of program size formally identical to information theory. Journal of the ACM, 22, 329–340.
Chen, P. S. (1976). The entity-relationship model: Toward a unified view of data. ACM Transactions on Database Systems, 1(1), 9–36.
Cilibrasi, R. (2003). Clustering by compression. IEEE Transactions on Information Theory, 51(4), 1523–1545.
Cilibrasi, R., & Vitányi, P. (2005). Google teaches computers the meaning of words. ERCIM News, 61.
Cilibrasi, R., & Vitányi, P. (2007). The Google similarity distance. IEEE Transactions on Knowledge and Data Engineering, 19(3), 370–383.
Codd, E. W. (1970). A relational model of data for large shared databanks. Communications of the ACM, 13(6), 377–387.
Codd, E. W. (1990). The relational model for database management. Version 2. Reading: Addison-Wesley.
Danchin, A. (1998). The delphic boat: What genomes tell us. Paris: Odile Jacob [Cambridge/ London: Harvard University Press, 2003.]
Delahaye, J. P. (1999). Information, complexité, hasard (2nd ed.). Hermès.
Delahaye, J. P. (2004). Classer musiques, langues, images, textes et génomes. Pour La Science, 316, 98–103.
Delahaye, J. P. (2006). Complexités: Aux limites des mathématiques et de l’informatique. Belin-Pour la Science.
Dershowitz, N., & Gurevich, Y. (2008). A natural axiomatization of computability and proof of Church’s thesis. The Bulletin of Symbolic Logic, 14(3), 299–350.
Dijkstra, E. W. (1972). The humble programmer. In ACM Turing Lecture. Available on the Web from http://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD340.html
Dijkstra, E. W. (1982). Selected writings on computing: A personal perspective. New York: Springer.
Durand, B., & Zvonkin, A. (2004–2007). Kolmogorov complexity, In E. Charpentier, A. Lesne, & N. Nikolski (Eds.), Kolmogorov’s heritage in mathematics (pp. 269–287). Berlin: Springer. (pp. 281–300, 2007).
Evangelista, A., & Kjos-Hanssen, B. (2006). Google distance between words. Frontiers in Undergraduate Research, University of Connecticut.
Feller, W. (1968). Introduction to probability theory and its applications (3rd ed., Vol. 1). New York: Wiley.
Ferbus-Zanda, M. (1986). La méthode de résolution et le langage Prolog [The resolution method and the language Prolog], Rapport LITP, No-8676.
Ferbus-Zanda, M. (In preparation-a). Logic and information system: Relational and conceptual databases.
Ferbus-Zanda, M. (In preparation-b). Kolmogorov complexity and abstract states machines: The relational point of view.
Ferbus-Zanda, M. (In preparation-c). Duality: Logic, computer science and Boolean algebras.
Ferbus-Zanda, M. (In preparation-d). Logic and information system: Cybernetics, cognition theory and psychoanalysis.
Ferbus-Zanda, M., & Grigorieff, S. (2004). Is randomness native to computer science? In G. Paun, G. Rozenberg, & A. Salomaa (Eds.), Current trends in theoretical computer science (pp. 141–179). River Edge: World Scientific.
Ferbus-Zanda, M., & Grigorieff, S. (2006). Kolmogorov complexity and set theoretical representations of integers. Mathematical Logic Quarterly, 52(4), 381–409.
Ferbus-Zanda, M., & Grigorieff, S. (2010). ASMs and operational algorithmic completeness of lambda calculus. In N. Dershowitz & W. Reisig (Eds.), Fields of logic and computation (Lecture notes in computer science, Vol. 6300, pp. 301–327). Berlin/Heidelberg: Springer.
Howard, W. (1980). The formulas-as-types notion of construction. In J. P. Seldin & J. R. Hindley (Eds.), Essays on combinatory logic, lambda calculus and formalism (pp. 479–490). London: Academic.
Irving, J. (1978). The world according to garp. Modern Library. Ballantine.
Kolmogorov, A. N. (1956). Grundbegriffe der Wahscheinlichkeitsrechnung [Foundations of the theory of probability]. New York: Chelsea Publishing. (Springer, 1933).
Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1–7.
Kolmogorov, A. N. (1983). Combinatorial foundation of information theory and the calculus of probability. Russian Mathematical Surveys, 38(4), 29–40.
Krivine, J. L. (2003). Dependent choice, ‘quote’ and the clock. Theoretical Computer Science, 308, 259–276. See also http://www.pps.univ-paris-diderot.fr/~krivine/
Li, M., & Vitányi, P. (1997). An introduction to Kolmogorov complexity and its applications (2nd ed.). New York: Springer.
Li, M., Chen, X., Li, X., Mav, B., & Vitányi, P. (2003). The similarity metrics. In 14th ACM-SIAM 1357 symposium on discrete algorithms, Baltimore. Philadelphia: SIAM
Martin-Löf, P. (1979, August 22–29). Constructive mathematics and computer programming. Paper read at the 6-th International Congress for Logic, Methodology and Philosophy of Science, Hannover.
Shannon, C. E. (1948). The mathematical theory of communication. Bell System Technical Journal, 27, 379–423.
Shoenfield, J. (2001). Recursion theory, (Lectures notes in logic 1, New ed.). Natick: A.K. Peters
Solomonoff, R. (1964). A formal theory of inductive inference, Part I & II. Information and control, 7, 1–22; 224–254.
Véronis, J. (2005). Web: Google perd la boole. (Web: Googlean logic) Blog. January 19, 2005 from http://aixtal.blogspot.com/2005/01/web-google-perd-la-boole.html, http://aixtal.blogspot.com/2005/02/web-le-mystre-des-pages-manquantes-de.html, http://aixtal.blogspot.com/2005/03/google-5-milliards-de-sont-partis-en.html.
Wiener, N. (1948). Cybernetics or control and communication in the animal and the machine (2nd ed.). Paris/Hermann: The Technology Press. (Cambridge: MITs, 1965).
Acknowledgements
For Francine Ptakhine, who gave me liberty of thinking and writing.
Thanks to Serge Grigorieff and Chloé Ferbus for listening, fruitful communication and for the careful proofreading and thanks to Maurice Nivat who welcomed me at the LITP in 1983.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Ferbus-Zanda, M. (2014). Kolmogorov Complexity in Perspective Part II: Classification, Information Processing and Duality. In: Dubucs, J., Bourdeau, M. (eds) Constructivity and Computability in Historical and Philosophical Perspective. Logic, Epistemology, and the Unity of Science, vol 34. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-9217-2_4
Download citation
DOI: https://doi.org/10.1007/978-94-017-9217-2_4
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-017-9216-5
Online ISBN: 978-94-017-9217-2
eBook Packages: Humanities, Social Sciences and LawPhilosophy and Religion (R0)