Rational Neural Models Based on Information Theory

Abstract

Rationality necessitates that the neuron must base its decisions on probability measures established on the interrogative proposition q which essentially defines the neuron as an observer and which is established during learning. This requirement is induced by the fact that probability is the only measure of degree of plausible belief which meets all logical consistency requirements. Similarly, the establishment of a probability measure by a rational neuron must entail the usage of a maximized entropy criterion since this is the only logically consistent means of doing so as based on observable information. A maximum entropy (ME) formulation can be shown to provide the basic functional form of the model neuron including synaptic weights and a sigmoidal transfer characteristic. However, this formulation requires the specification of linear constraints which are unavailable. Alternatively, a maximum mutual information (MMI) formulation is shown to be fully constrained in this regard and can make exclusive use of locally available information. Solutions take the form of the Hopfield neuron model with a requirement for Hebbian learning.