Gibbs Sampling as a Natural Statistical Analog of Constraints Techniques: Prediction in Science under General Probabilistic Uncertainty

Abstract

One of the main objectives of science is to predict future events, i.e., more precisely, the results of future measurements and observations. If we take into account the probabilistic uncertainty related to the inaccuracy of the measurement results, to the inaccuracy of the model, and to the inaccuracy of the prior information, then the most adequate approach is to generate a posterior distribution by using Bayes’ theorem. For the simplest posterior distributions, we can deduce explicit analytical formulas for the resulting statistical characteristics (mean, standard deviation, etc.) of the predict future measurement result. However, in general, such formulas are not possible, so we have to use a Monte-Carlo simulation of the corresponding joint distribution of the future measurement results and model parameters.

The main computational challenge here is that there is no general algorithm for simulating an arbitrary multi-variate distribution; such algorithms are known only for single-variate distributions and – in some cases – for the case of several variables. Thus, we need to reduce the general simulation problem to such simplified cases. We show that this problem can be solved by using the general constraints approach, and that this idea clarifies Gibbs sampling – one of the most widely used techniques for such simulation. This interpretation of Gibbs sampling enables us to analyze Gibbs sampling – in particular, to obtain a (somewhat counterintuitive) result that while a straightforward parallelization is possible for deterministic constraint propagation, parallelization does not work even in the simplest two-variable probabilistic case.