Advertisement

Neighbourhood Blocking for Record Linkage

  • Daniel EliasEmail author
  • Josiah Poon
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11393)

Abstract

This paper describes Neighbourhood Blocking – a novel method for the indexing step in the record linkage process. Record Linkage is the task of identifying database records referring to the same entity without the aid of definitive key fields. It has applications in data integration, fraud detection and other areas. This involves comparing pairs of records. If done indiscriminately, the size of this task is quadratic in dataset size. Therefore, various indexing methods are typically used to reduce the number of record pairs subjected to detailed comparison. Neighbourhood Blocking generalizes two existing indexing methods – Standard Blocking and Sorted Neighbourhood Indexing. It also allows meaningful treatment of missing values and a limited number of blocking field mismatches. Comparison of the Cartesian product of the blocks is avoided through the use of recursion. Numerical experiments and tests on benchmark datasets are reported in which Neighbourhood Blocking is compared to Standard Blocking and Sorted Neighbourhood Indexing. Under the conditions tested, Neighbourhood Blocking is found to frequently produce superior index quality, often at the expense of increased runtime. Scale testing indicates that index production speeds for Neighbourhood Blocking and Standard Blocking are similar when the database size is sufficiently large.

References

  1. 1.
    Khairul Nizam, B., Shahrul Azman, M.N.: Efficient identity matching using static pruning q-gram indexing approach. Decis. Support Syst. 73, 97–108 (2015)CrossRefGoogle Scholar
  2. 2.
    Christen, P.: Data Matching. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-31164-2CrossRefGoogle Scholar
  3. 3.
    Dunn, H.L.: Record linkage. Am. J. Public Health Nation’s Health 39, 1412–1416 (1946)CrossRefGoogle Scholar
  4. 4.
    Fellegi, I.P., Sunter, A.B.: A theory for record linkage. J. Am. Stat. Assoc. 64, 1183–1210 (1969)CrossRefGoogle Scholar
  5. 5.
    Hernandez, M.A., Stolfo, S.J.: The merge/purge problem for large databases. In: Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, San Jose, California, USA. ACM (1995)Google Scholar
  6. 6.
    Kopcke, H., Thor, A., Rahm, E.: Learning-based approaches for matching web data entities. IEEE Internet Comput. 14(4), 23–31 (2010).  https://doi.org/10.1109/MIC.2010.58CrossRefGoogle Scholar
  7. 7.
    Papenbrock, T., Heise, A., Naumann, F.: Progressive duplicate detection. IEEE Trans. Knowl. Data Eng. 27(5), 1316–1329 (2015).  https://doi.org/10.1109/TKDE.2014.2359666CrossRefGoogle Scholar
  8. 8.
    Poon, S.K., et al.: An ensemble approach for record matching in data linkage. Stud. Health Technol. Inform. 227, 113–119 (2016)Google Scholar
  9. 9.
    Ramadan, B., Peter Christen, H.L., Gayler, R.W.: Dynamic sorted neighborhood indexing for real-time entity resolution. J. Data Inf. Qual. 6(4), 15 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Commonwealth BankSydneyAustralia
  2. 2.University of SydneySydneyAustralia

Personalised recommendations