Encoding in the presence of penalties. First variational problem

Abstract

The amount of information that can be recorded or transmitted is defined by a logarithm of the number of various recording realizations or transmission realizations, respectively. However, calculation of this number is not always a simple task. It can be complicated because of the presence of some constraints imposed on feasible realizations. In many cases, instead of direct calculation of the number of realizations, it is reasonable to compute the maximum value of recording entropy via maximization over distributions compatible with conditions imposed on the expected value of some random cost. This maximum value of entropy is called the capacity of a channel without noise. This variational problem is the first from the set of variational problems playing an important role in information theory.