Lossless Coding

Abstract

The goal here is to encode a sequence of symbols in such a way that it is possible to decode it perfectly (lossless coding), and to decode it sequentially (prefix coding). One may then relate codes and probabilities: this is the essence of the Kraft-McMillan inequalities. If one aims at minimizing the codeword’s length, Shannon’s entropy gives an intrinsic limit, when the word to be encoded is regarded as a random variable. When the distribution of this random variable is known, then the optimal compression rate can be achieved (Shannon’s coding and Huffman’s coding). Moreover, as codeword lengths are identified with probability distributions, for any probability distribution, one may design a prefix code which encodes sequentially. This will be referred to as “coding according to this distribution”. Arithmetic coding, based on a probability distribution which is not necessarily the one of the source, will be particularly detailed. In this way, the algorithmic aspect of coding and the modeling of the source distribution are separated. Here the word “source” is used as a synonym for a random process. We finally point out some essential tools needed to quantify information, in particular the entropy rate of a process. This rate appears as an intrinsic lower bound for the asymptotic compression rate, for almost every source trajectory, as soon as it is ergodic and stationary. This also shows that is is crucial to encode words in blocks. Arithmetic coding has the advantage of encoding in blocks and “online”. If arithmetic coding is devised with the source distribution, then it asymptotically achieves the optimal compression rate. In the following chapters, we will be interested in the question of adapting the code to an unknown source distribution, which corresponds to a fundamentally statistical question.