PC-Filter: A Robust Filtering Technique for Duplicate Record Detection in Large Databases
In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of O(N 4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the ”Key Selection” problem, the ”Scope Specification” problem and the ”Low Recall” problem that the existing methods suffer from.
KeywordsEdit Distance Longe Common Subsequence Longe Common Subsequence Recall Level Record Similarity
Unable to display preview. Download preview PDF.
- 1.Ananthakrishna, R., Chaudhuri, S., Ganti, V.: Eliminating Fuzzy Duplicates in Data Warehouses. In: Proceedings of VLDB 2002, Hong Kong, China, pp. 586–597 (2002)Google Scholar
- 2.Chaudhuri, S., Ganjam, K., Ganti, V., Motwani, R.: Robust and Efficient Fuzzy Match for Online Data Cleaning. In: Proceedings of ACM SIGMOD 2003, San Diego, USA, pp. 313–324 (2003)Google Scholar
- 3.Gravano, L., Ipeirotis, P.G., Koudas, N., Srivastava, D.: Text Joins for Data Cleansing and Integration in an RDBMS. In: Proceedings ICDE 2003, pp. 729–731 (2003)Google Scholar
- 4.Hernandez, M.: A Generation of Band Joins and the Merge/Purge Problem. Technical Report CUCS-005-1995, Columbia University (February 1996)Google Scholar
- 5.Hernandez, M.A., Stolfo, S.J.: The Merge/Purge Problem for Large Documents. In: Proceedings of the 1995 ACM-SIGMOD, pp. 127–138 (1995)Google Scholar
- 6.Low, W.L., Lee, M.L., Ling, T.W.: A Knowledge-Based Framework for Duplicates Elimination. Information Systems: Special Issue on Data Extraction, Cleaning and Reconciliation 26(8), Elsevier Science (2001)Google Scholar
- 7.Monge, A.E., Elkan, C.P.: An Efficient Domain-independent Algorithm for detecting Approximately Duplicate Document Records. In: Proceedings of SIGMOD Workshop on Research issues and Data Mining and Knowledge Discovery (1997)Google Scholar
- 8.Monge, A.E., Elkan, C.P.: The Field Matching Problem: Algorithms and Application. In: Proceedings of SIGKDD 1996, pp. 267–270 (1996)Google Scholar
- 10.Sung, S.Y., Li, Z., Peng, S.: A Fast Filtering Scheme for Large Document Cleansing. In: Proceedings of CIKM 2002, pp. 76–83 (2002)Google Scholar