Universal Types and Simulation of Individual Sequences

  • Gadiel Seroussi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2976)


We define the universal type class of an individual sequence x \(_{\rm 1}^{n}\), in analogy to the classical notion used in the method of types of information theory. Two sequences of the same length are said to be of the same universal (LZ) type if and only if they yield the same set of phrases in the incremental parsing of Ziv and Lempel (1978). We show that the empirical probability distributions of any finite order k of two sequences of the same universal type converge, in the variational sense, as the sequence length increases. Consequently, the logarithms of the probabilities assigned by any k-th order probability assignment to two sequences of the same universal type converge, for any k. We estimate the size of a universal type class, and show that its behavior parallels that of the conventional counterpart, with the LZ78 code length playing the role of the empirical entropy. We present efficient procedures for enumerating the sequences in a universal type class, and for drawing a sequence from the class with uniform probability. As an application, we consider the problem of universal simulation of individual sequences. A sequence drawn with uniform probability from the universal type class of x \(_{\rm 1}^{n}\) is a good simulation of x \(_{\rm 1}^{n}\) in a well defined mathematical sense.


Type Class Uniform Probability Parsing Tree Individual Sequence Bridge Node 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Csiszár, I., Körner, J.: Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic, New York (1981)zbMATHGoogle Scholar
  2. 2.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Inc., New York (1991)zbMATHCrossRefGoogle Scholar
  3. 3.
    Csiszár, I.: The method of types. IEEE Trans. Inform. Theory IT-44, 2505–2523 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Merhav, N., Weinberger, M.J.: On universal simulation of information sources using training data. Technical Report HPL-2002-263, Hewlett-Packard Laboratories (2002); To appear in IEEE Transactions on Information TheoryGoogle Scholar
  5. 5.
    Ziv, J., Lempel, A.: Compression of individual sequences via variable-rate coding. IEEE Trans. Inform. Theory IT-24, 530–536 (1978)zbMATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Rissanen, J.: A universal data compression system. IEEE Trans. Inform. Theory IT-29, 656–664 (1983)zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Seroussi, G.: Universal types and simulation of individual sequences (in preparation, 2003)Google Scholar
  8. 8.
    Knuth, D.E.: The Art of Computer Programming. Seminumerical Algorithms, 3rd edn., vol. 2. Addison-Wesley, Reading (1998)Google Scholar
  9. 9.
    Freiling, C., Jungreis, D., Theberge, F., Zeger, K.: Almost all complete binary prefix codes have a self-synchronizing string. IEEE Trans. Inform. Theory IT-49, 2219–2225 (2003)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Cover, T.M.: Enumerative source encoding. IEEE Transactions on Information Theory IT-19, 73–77 (1973)zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Gadiel Seroussi
    • 1
  1. 1.Hewlett-Packard LaboratoriesPalo AltoUSA

Personalised recommendations