Hierarchical Linkage Clustering with Distributions of Distances for Large-Scale Record Linkage

Abstract

Distance-based clustering techniques such as hierarchical clustering use a single estimate of distance for each pair of observations; their results then rely on the accuracy of this estimate. However, in many applications, datasets include measurement error or are too large for traditional models, meaning a single estimate of distance between two observations may be subject to error or computationally prohibitive to calculate. For example, in many of today’s large-scale record linkage problems, datasets are prohibitively large, making distance estimates computationally infeasible. By using a distribution of distance estimates instead (e.g. from an ensemble of classifiers trained on subsets of recordpairs), these issues may be resolved. We present a large-scale record linkage framework that incorporates classifier ensembles and “distribution linkage” clustering to identify clusters of records corresponding to unique entities. We examine the performance of several different distributional summary measures in hierarchical clustering. We motivate and illustrate this approach with an application of record linkage to the United States Patent and Trademark Office database.

An Erratum for this chapter can be found at http://dx.doi.org/10.1007/978-3-319-11257-2_28