PC-Filter: A Robust Filtering Technique for Duplicate Record Detection in Large Databases

  • Ji Zhang
  • Tok Wang Ling
  • Robert M. Bruckner
  • Han Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3180)


In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of O(N 4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the ”Key Selection” problem, the ”Scope Specification” problem and the ”Low Recall” problem that the existing methods suffer from.


Edit Distance Longe Common Subsequence Longe Common Subsequence Recall Level Record Similarity 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ananthakrishna, R., Chaudhuri, S., Ganti, V.: Eliminating Fuzzy Duplicates in Data Warehouses. In: Proceedings of VLDB 2002, Hong Kong, China, pp. 586–597 (2002)Google Scholar
  2. 2.
    Chaudhuri, S., Ganjam, K., Ganti, V., Motwani, R.: Robust and Efficient Fuzzy Match for Online Data Cleaning. In: Proceedings of ACM SIGMOD 2003, San Diego, USA, pp. 313–324 (2003)Google Scholar
  3. 3.
    Gravano, L., Ipeirotis, P.G., Koudas, N., Srivastava, D.: Text Joins for Data Cleansing and Integration in an RDBMS. In: Proceedings ICDE 2003, pp. 729–731 (2003)Google Scholar
  4. 4.
    Hernandez, M.: A Generation of Band Joins and the Merge/Purge Problem. Technical Report CUCS-005-1995, Columbia University (February 1996)Google Scholar
  5. 5.
    Hernandez, M.A., Stolfo, S.J.: The Merge/Purge Problem for Large Documents. In: Proceedings of the 1995 ACM-SIGMOD, pp. 127–138 (1995)Google Scholar
  6. 6.
    Low, W.L., Lee, M.L., Ling, T.W.: A Knowledge-Based Framework for Duplicates Elimination. Information Systems: Special Issue on Data Extraction, Cleaning and Reconciliation 26(8), Elsevier Science (2001)Google Scholar
  7. 7.
    Monge, A.E., Elkan, C.P.: An Efficient Domain-independent Algorithm for detecting Approximately Duplicate Document Records. In: Proceedings of SIGMOD Workshop on Research issues and Data Mining and Knowledge Discovery (1997)Google Scholar
  8. 8.
    Monge, A.E., Elkan, C.P.: The Field Matching Problem: Algorithms and Application. In: Proceedings of SIGKDD 1996, pp. 267–270 (1996)Google Scholar
  9. 9.
    Li, Z., Sung, S.Y., Sun, P., Ling, T.W.: A New Efficient Data Cleansing Method. In: Hameurlain, A., Cicchetti, R., Traunmüller, R. (eds.) DEXA 2002. LNCS, vol. 2453, p. 484. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  10. 10.
    Sung, S.Y., Li, Z., Peng, S.: A Fast Filtering Scheme for Large Document Cleansing. In: Proceedings of CIKM 2002, pp. 76–83 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Ji Zhang
    • 1
  • Tok Wang Ling
    • 2
  • Robert M. Bruckner
    • 3
  • Han Liu
    • 1
  1. 1.Department of Computer ScienceUniversity of TorontoTorontoCanada
  2. 2.School of ComputingNational University of SingaporeSingapore
  3. 3.Microsoft ResearchRedmondUSA

Personalised recommendations