Minimum-Entropy Data Clustering Using Reversible Jump Markov Chain Monte Carlo

Abstract

Many problems in data analysis, especially in signal and image processing, require the unsupervised partitioning of data into a set of ‘self-similar’ classes or clusters. An ideal partitioning unambiguously assigns each datum to a single class and one thinks of the data as being generated by a number of data generators, one for each class. Many algorithms have been proposed for such analysis and for the estimation of the optimal number of partitions. The majority of popular and computationally feasible techniques rely on assuming that classes are hyper-ellipsoidal in shape. In the case of Gaussian mixture modelling [15,6] this is explicit; in the case of dendogram linkage methods (which typically rely on the L ₂ norm) it is implicit [9]. For some data sets this leads to an over-partitioning. Alternative methods, based on valley seeking [6] or maxima-tracking in scale-space [16,18,13] for example, have the advantage that they are free from such assumptions. They can be, however, sensitive to noise and computationally intensive in high-dimensional spaces. In this paper we re-consider the issue of data partitioning from an information-theoretic viewpoint and show that minimisation of partition entropy may be used to evaluate the most probable set of data generators. Rather than formulate the problem as one of a traditional model-order estimation to infer the most probable number of classes we employ a reversible jump mechanism in a Markov-chain Monte Carlo (MCMC) sampler which explores the space of different model sizes.