Neighbourhood Blocking for Record Linkage

Abstract

This paper describes Neighbourhood Blocking – a novel method for the indexing step in the record linkage process. Record Linkage is the task of identifying database records referring to the same entity without the aid of definitive key fields. It has applications in data integration, fraud detection and other areas. This involves comparing pairs of records. If done indiscriminately, the size of this task is quadratic in dataset size. Therefore, various indexing methods are typically used to reduce the number of record pairs subjected to detailed comparison. Neighbourhood Blocking generalizes two existing indexing methods – Standard Blocking and Sorted Neighbourhood Indexing. It also allows meaningful treatment of missing values and a limited number of blocking field mismatches. Comparison of the Cartesian product of the blocks is avoided through the use of recursion. Numerical experiments and tests on benchmark datasets are reported in which Neighbourhood Blocking is compared to Standard Blocking and Sorted Neighbourhood Indexing. Under the conditions tested, Neighbourhood Blocking is found to frequently produce superior index quality, often at the expense of increased runtime. Scale testing indicates that index production speeds for Neighbourhood Blocking and Standard Blocking are similar when the database size is sufficiently large.