Abstract
To improve the retrieval efficiency and performance of the large scale information retrieval systems, analyzed existing algorithm for Web search and duplicated Web pages detection. However, it has some drawback in terms of precision and efficiency because of its generality and no specialty. In this paper, with crawler and duplicated pages analysis, addressed two issues of the topic-oriented Web crawler and near-replicas detection. One is how to make the definition of the topic; the other is how to eliminate duplicate pages. It aimed to visit only topic-oriented pages, and got a great scale of hyperlinks which link to the topic-oriented pages. The crawl and Web pages detection method is a novel one, which was based on the semi-structured features of the website and content information. The results of experiment show that it is better than that of the existing algorithms proposed in the literature.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Yu, H.J., Liu, Y.Q., Zhang, M., Ru, L.Y., Ma, S.P.: Research in Search Engine User Behavior Based on Log Analysis. Journal of Chinese Information Processing 21(1), 109–114 (2007) (in Chinese with English abstract)
Bra, P.D., Houdben, G., Kornatzky, Y.: Information Retrieval in Distributed Hypertexts. In: Proc of the 4th RIAO Conference, New York, pp. 481– 491 (1994)
Gant, G., Srinivasan, P.: Topic-Driven Crawlers: Machine Learningissues. In: Proc of ACM TOIT (2004)
Angkawattanawit, N., Rungsawang, A.: Learnable Topic- Specific Web Crawler. Massive Information & Knowledge Engineering 28(2), 97–114 (2005)
Menczer, F., Pant, G., Srinivasan, P.: Topic Web Crawlers: Evaluating Adaptive Algorithm. ACM Transactions on Internet Technology 4(4), 378–419 (2004)
Ye, S.Z., Ji, R.W., Ma, W.Y.: A Systematic Study on Parameter Correlations in Large-Scale Duplicate Document Detection. Knowledge and Information Systems 14, 217–232 (2007)
Hosseini, S., Cox, J.R.: Optimal Solution of Off-line And On-Line Generalized Caching. Technical Report, WUCS-96-20, washington University at St. Louis (1996)
Voorhees, E.M.: The Philosophy of Information Retrieval Evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 355–370. Springer, Heidelberg (2002)
Scholer, F., Williams, H.E., Yiannis, J., Zobel, J.: Compression of Inverted Indexes for Fast Query Evaluation. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 222–229 (2002)
Cockburn, A., Jones, S.: Which way now? Analysing and easing inadequacies in WWW navigation. International Journal of Human-Computer Studies 45, 105–129
Silverstein, C., Henzinger, M., Marais, H.: Analysis of A Very Large Web Search Engine Query Log. SIGIR Forum 33(1), 6–12 (1998)
Andrei, B.: A Taxonomy of Web Search. SIGIR Forum 36(2) (2002)
Voorhees, E.M., Lori, P., Buckland, E.: The Eleventh Text Retrieval Conference (TREC 2002), vol. 11. National Institute of Standards and Technology, NIST (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhang, YH., Zhang, F. (2012). Research on New Algorithm of Topic-Oriented Crawler and Duplicated Web Pages Detection. In: Huang, DS., Ma, J., Jo, KH., Gromiha, M.M. (eds) Intelligent Computing Theories and Applications. ICIC 2012. Lecture Notes in Computer Science(), vol 7390. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31576-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-31576-3_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31575-6
Online ISBN: 978-3-642-31576-3
eBook Packages: Computer ScienceComputer Science (R0)