Improving Document Similarity Calculation Using Cosine-Similarity Graphs
Data mining information using various indices and determining candidates that can be judged as having the same tendency based on similarity between documents is common. The accuracy of similarity largely depends on a sufficient amount of data and requires advanced analysis using natural language processing. In this paper, we present an approach for filtering based on the cosine similarity graph and clustering between candidates. For filtering candidates, we focus on the inflection point of the graphs when plotting by sorting similarities in descending order. By clustering among higher similarities, we aim at filtering candidates that cannot be removed without analyzing advanced natural language processing. The proposed method was applied to movie reviews and sightseeing location reviews written in Japanese. Although this study is a work in progress, it shows that candidates can be recommended without having to manually apply natural language processing, such as preparing stopwords for each category.
KeywordsDocument similarity Cosine similarity Content-based filtering
- 2.Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on International Conference on Machine Learning, ICML 2014, vol. 32, pp. II–1188–II–1196 (2014)Google Scholar
- 3.MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, pp. 281–297 (1967)Google Scholar
- 4.Töscher, A., Jahrer, M., Legenstein, R.: Improved neighborhood-based algorithms for large-scale recommender systems. In: Proceedings of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition, NETFLIX 2008, pp. 4:1–4:6 (2008)Google Scholar