Classical Information Theory

Abstract

We begin with the definition of information gained by knowing that an event A has occurred:

$$\iota (A) = -\log_2 {{\mathbf P}}(A).$$

((1))

(A dual point of view is also useful (although more evasive), where $\iota (A)$ is the amount of information needed to specify event A.) Here and below ${{\mathbf P}}$ stands for the underlying probability distribution. So the rarer an event A, the more information we gain if we know it has occurred. (More broadly, the rarer an event A, the more impact it will have. For example, the unlikely event that occurred in 1938 when fishermen caught a coelacanth – a prehistoric fish believed to be extinct – required a significant change to beliefs about evolution and biology. On the other hand, the likely event of catching a herring or a tuna would hardly imply any change in theories.)