Parallel Algorithms for Maximum Entropy Calculation

Abstract

Maximum entropy extensions of partial statistical information lead to Gibbs distributions. In applications, the utility of these extensions depends upon our ability to perform various operations on the Gibbs distributions, such as random sampling, identification of the mode, and calculation of expectations. In many applications these operations are computationally intractable by conventional techniques, but it is possible to perform these in parallel. The architecture for a completely parallel machine dedicated to computing functionals of Gibbs distributions is suggested by the connection between Gibbs distributions and statistical mechanics. The Gibbs distribution defines a collection of local physical rules that dictate the programming of the machine’s processors. The machine’s dynamics can be described by a Markov process that is demonstrably ergodic with marginal distribution equal to the specified Gibbs distribution. These two properties are easily exploited to perform the desired operations.