Abstract
Due to the so-called “curse of dimensionality” causing poor performance when querying in the high-dimensional space, the high-dimensional approximate kNN (AkNN) query has been extensively explored to trade accuracy for efficiency. In this paper, we propose a Local Intrinsic Dimension-based Hashing (LIDH) method for the high-dimensional AkNN query which locates a definite searching range by Local Intrinsic Dimensionality for filtering data points. Specifically, we propose a filter-refinement model for the AkNN query to avoid the virtual rehashing with fewer index space. Experimental evaluations demonstrate that our method can provide higher I/O and CPU efficiency while retaining satisfactory query accuracies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gan, J., Feng, J., Fang, Q., Ng, W.: Locality-sensitive hashing scheme based on dynamic collision counting. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, Scottsdale, AZ, USA, 20–24 May 2012, pp. 541–552 (2012)
Huang, Q., Feng, J., Zhang, Y., Fang, Q., Ng, W.: Query-aware locality-sensitive hashing for approximate nearest neighbor search. PVLDB 9(1), 1–12 (2015)
Karger, D.R., Ruhl, M.: Finding nearest neighbors in growth-restricted metrics. In: Proceedings on 34th Annual ACM Symposium on Theory of Computing, 19–21 May 2002, Montréal, Québec, Canada, pp. 741–750 (2002)
Houle, M.E., Kashima, H., Nett, M.: Generalized expansion dimension. In: 12th IEEE International Conference on Data Mining Workshops, ICDM Workshops, Brussels, Belgium, 10 December 2012, pp. 587–594 (2012)
Casanova, G., Englmeier, E., Houle, M.E., Kröger, P., Nett, M., Schubert, E., Zimek, A.: Dimensional testing for reverse k-nearest neighbor search. PVLDB 10(7), 769–780 (2017)
Houle, M.E.: Dimensionality, discriminability, density and distance distributions. In: 13th IEEE International Conference on Data Mining Workshops, ICDM Workshops, TX, USA, 7–10 December 2013, pp. 468–473 (2013)
Amsaleg, L., Chelly, O., Furon, T., Girard, S., Houle, M.E., Kawarabayashi, K., Nett, M.: Estimating local intrinsic dimensionality. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 August 2015, pp. 29–38 (2015)
Houle, M.E., Ma, X., Nett, M., Oria, V.: Dimensional testing for multi-step similarity search. In: 12th IEEE International Conference on Data Mining, ICDM 2012, Brussels, Belgium, 10–13 December 2012, pp. 299–308 (2012)
Houle, M.E., Ma, X., Oria, V.: Effective and efficient algorithms for flexible aggregate similarity search in high dimensional spaces. IEEE Trans. Knowl. Data Eng. 27(12), 3258–3273 (2015)
Houle, M.E., Ma, X., Oria, V., Sun, J.: Efficient similarity search within user-specified projective subspaces. Inf. Syst. 59, 2–14 (2016)
Hoeffding, W.: Probability inequalities for sums of bounded random variables. Publ. Am. Stat. Assoc. 58(301), 13–30 (1963)
Acknowledgements
This work is supported by the National Natural Science Foundation of China (61472071 and 61433008), the Fundamental Research Funds for the Central Universities (N171605001) and the Natural Science Foundation of Liaoning Province (2015020018).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Song, Y., Gu, Y., Yu, G. (2018). LIDH: An Efficient Filtering Method for Approximate k Nearest Neighbor Queries Based on Local Intrinsic Dimension. In: Cai, Y., Ishikawa, Y., Xu, J. (eds) Web and Big Data. APWeb-WAIM 2018. Lecture Notes in Computer Science(), vol 10987. Springer, Cham. https://doi.org/10.1007/978-3-319-96890-2_22
Download citation
DOI: https://doi.org/10.1007/978-3-319-96890-2_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96889-6
Online ISBN: 978-3-319-96890-2
eBook Packages: Computer ScienceComputer Science (R0)