Advertisement

Encoding of discrete information in the absence of noise and penalties

  • Roman V. Belavkin
  • Panos M. Pardalos
  • Jose C. Principe
Chapter

Abstract

The definition of the amount of information, given in Chapter  1, is justified when we deal with a transformation of information from one kind into another, i.e. when considering encoding of information. It is essential that the law of conservation of information amount holds under such a transformation. It is very useful to draw an analogy with the law of conservation of energy. The latter is the main argument for introducing the notion of energy. Of course, the law of conservation of information is more complex than the law of conservation of energy in two respects. The law of conservation of energy establishes an exact equality of energies, when one type of energy is transformed into another. However, in transforming information we have a more complex relation, namely ‘not greater’ (\(\leqslant \)), i.e. the amount of information cannot increase. The equality sign corresponds to optimal encoding. Thus, when formulating the law of conservation of information, we have to point out that there possibly exists such an encoding, for which the equality of the amounts of information occurs.

References

  1. 9.
    Fano, R.M.: Transmission of Information: A Statistical Theory of Communications, 1st edn. MIT Press, Cambridge (1961)Google Scholar
  2. 10.
    Fano, R.M.: Transmission of Information: A Statistical Theory of Communications (Translation to Russian). Mir, Moscow (1965)Google Scholar
  3. 11.
    Feinstein, A.: Foundations of Information Theory. McGraw-Hill, New York (1958)CrossRefGoogle Scholar
  4. 12.
    Feinstein, A.: Foundations of Information Theory (Translation to Russian). Inostrannaya Literatura, Moscow (1960)Google Scholar
  5. 23.
    Huffman, D.A.: A method for the construction of minimum redundancy codes. Proc. IRE 40(9), 1098–1101 (1952)CrossRefGoogle Scholar
  6. 28.
    Kraft, L.G.: A device for quantizing, grouping, and coding amplitude-modulated pulses. Master’s Thesis, Massachusetts Institute of Technology, Dept. of Electrical Engineering (1949)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Roman V. Belavkin
    • 1
  • Panos M. Pardalos
    • 2
  • Jose C. Principe
    • 3
  1. 1.Faculty of Science and TechnologyMiddlesex UniversityLondonUK
  2. 2.Industrial and Systems EngineeringUniversity of FloridaGainesvilleUSA
  3. 3.Electrical & Computer EngineeringUniversity of FloridaGainesvilleUSA

Personalised recommendations