Abstract
Set similarity join is a database operation used to find out all similar pairs of sets from two collections over sets. Due to the high versatility, it has been applied in many applications in various domains, including text processing, image processing, etc. One of its drawbacks is the high computational costs, especially when the size of the given collections is large. In this paper, we propose a scheme for efficient set similarity join on Intel Xeon Phi, one of the latest many core processor. In order to make best use of high computational power of Intel Xeon Phi, we employ following approaches: (1) we compress each record by b-bit MinHash to fit them in MCDRAM which is a high bandwidth on-chip memory; and (2) we apply some optimizations such as utilization of 512-bit SIMD instructions and loop unrolling. Experimental results show that our proposed method outperforms CPU implementation.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
References
Broder, A.Z., Charikar, M., Frieze, A.M., Mitzenmacher, M.: Min-wise independent permutations. J. Comput. Syst. Sci. 60(3), 630–659 (2000)
Chaudhuri, S., Ganti, V., Kaushik, R.: A primitive operator for similarity joins in data cleaning. In: ICDE (2006)
Cruz, M.S.H., Kozawa, Y., Amagasa, T., Kitagawa, H.: Accelerating set similarity joins using GPUs. Trans. Large-Scale Data- Knowl.-Cent. Syst. 28, 1–22 (2016)
Jha, S., He, B., Lu, M., Cheng, X., Huynh, H.P.: Improving main memory hash joins on intel xeon phi processors: an experimental approach. Proc. VLDB 8(6), 642–653 (2015)
Jiang, Y., Deng, D., Wang, J., Li, G., Feng, J.: Efficient parallel partition-based algorithms for similarity search and join with edit distance constraints. In: EDBT/ICDT Workshop (2013)
Li, P., König, A.C.: b-Bit minwise hashing. In: WWW (2010)
Li, P., Owen, A.B., Zhang, C.-H.: One permutation hashing. In: NIPS (2012)
Lieberman, M.D., Sankaranarayanan, J., Samet, H.: A fast similarity join algorithm using graphics processing units. In: ICDE, pp. 1111–1120 (2008)
Lu, M., Liang, Y., Huynh, H.P., Ong, Z., He, B., Goh, R.S.M.: MrPhi: an optimized mapreduce framework on intel xeon phi coprocessors. IEEE TPDS 26(11), 3066–3078 (2015)
Metwally, A., Faloutsos, C.: V-SMART-Join: a scalable mapreduce framework for all-pair similarity joins of multisets and vectors. PVLDB 5(8), 704–715 (2012)
Rong, C., Lin, C., Silva, Y.N., Wang, J., Lu, W., Du, X.: Fast and scalable distributed set similarity joins for big data analytics. In: ICDE (2017)
Xiao, C., Wang, W., Lin, X., Yu, J.X., Wang, G.: Efficient similarity joins for near-duplicate detection. ACM TODS 36(3), 151–1541 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Sugano, K., Amagasa, T., Kitagawa, H. (2018). Approximate Set Similarity Join Using Many-Core Processors. In: Hartmann, S., Ma, H., Hameurlain, A., Pernul, G., Wagner, R. (eds) Database and Expert Systems Applications. DEXA 2018. Lecture Notes in Computer Science(), vol 11030. Springer, Cham. https://doi.org/10.1007/978-3-319-98812-2_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-98812-2_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98811-5
Online ISBN: 978-3-319-98812-2
eBook Packages: Computer ScienceComputer Science (R0)