Optimal Codes

Abstract

We saw in Chapter 1 how to encode information so that decoding is unique or instantaneous. In either case the basic requirement, given by Kraft’s or McMillan’s inequality, is that we should use sufficiently long code-words. This raises the question of efficiency: if the code-words are too long, then storage may be difficult and transmission may be slow. We therefore need to strike a balance between using words which are long enough to allow effective decoding, and short enough for economy. Prom this point of view, the best codes available are those called optimal codes, the instantaneous codes with least average word- length. We will prove that they exist, and we will examine Huffman’s algorithm for constructing them. For simplicity, we will concentrate mainly on the binary case (r = 2), though we will briefly outline how these ideas extend to non-binary codes.