Abstract
Focused crawlers aim to automatically discover online content resources relevant to a domain of interest by automatically navigating through the Web link structure and selecting which hyperlinks to follow based on an estimation of their relevance to the topic of interest; to this end, classifier-guided approaches are typically employed for identifying hyperlinks having the higher likelihood of leading to relevant content. However, the training data used for building these classifiers might not be entirely representative of the domain of interest, or the domain of interest might change over time. To meet these challenges, this work proposes a novel adaptive focused crawling framework that allows the classifiers that underlie the hyperlink selection policy to be adapted based on the evidence they encounter during their crawls. Our framework uses two different approaches to retrain its models: (i) Interactive Adaptation, where a user manually evaluates the discovered resources, and (ii) Automatic Adaptation, where the framework uses the already trained classifiers to assess the relevance of newly discovered resources. The evaluation experiments in the domain of Islamic extremism indicate the effectiveness of online learning in focused crawling.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The actual URLs are not provided so as to avoid the inclusion of potentially sensitive information, but can be made available upon request.
- 2.
References
Chen, H.: Dark Web: Exploring and Data Mining the Dark Side of the Web, vol. 30. Springer, Heidelberg (2011). https://doi.org/10.1007/978-1-4614-1557-2
Chen, H., Chung, W., Qin, J., Reid, E., Sageman, M., Weimann, G.: Uncovering the dark web: a case study of Jihad on the web. J. Assoc. Inf. Sci. Technol. 59(8), 1347–1359 (2008)
Davison, B.D.: Topical locality in the web. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 272–279. ACM (2000)
De Bra, P.M., Post, R.: Information retrieval in the world-wide web: making client-based searching feasible. Comput. Netw. ISDN Syst. 27(2), 183–192 (1994)
Fu, T., Abbasi, A., Chen, H.: A focused crawler for dark web forums. J. Assoc. Inf. Sci. Technol. 61(6), 1213–1231 (2010)
Hassan, T., Cruz, C., Bertaux, A.: Ontology-based approach for unsupervised and adaptive focused crawling. In: Proceedings of The International Workshop on Semantic Big Data, p. 2. ACM (2017)
Iliou, C., Kalpakis, G., Tsikrika, T., Vrochidis, S., Kompatsiaris, I.: Hybrid focused crawling for homemade explosives discovery on surface and dark web. In: 2016 11th International Conference on Availability, Reliability and Security (ARES), pp. 229–234. IEEE (2016)
Khan, A., Baharudin, B., Lee, L.H., Khan, K.: A review of machine learning algorithms for text-documents classification. J. Adv. Inf. Technol. 1(1), 4–20 (2010)
Masud, M.M., et al.: Addressing concept-evolution in concept-drifting data streams. In: 2010 IEEE 10th International Conference on Data Mining (ICDM), pp. 929–934. IEEE (2010)
Olston, C., Najork, M.: Web crawling. Found. Trends Inf. Retrieval 4(3), 175–246 (2010)
Pant, G., Srinivasan, P.: Learning to crawl: comparing classification schemes. ACM Trans. Inf. Syst. (TOIS) 23(4), 430–462 (2005)
Pant, G., Srinivasan, P.: Link contexts in classifier-guided topical crawlers. IEEE Trans. Knowl. Data Eng. 18(1), 107–122 (2006)
Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. (CSUR) 34(1), 1–47 (2002)
Torkestani, J.A.: An adaptive focused web crawling algorithm based on learning automata. Appl. Intell. 37(4), 586–601 (2012)
Acknowledgements
This work was supported by the TENSOR project (H2020-700024), funded by the European Commission.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Iliou, C., Tsikrika, T., Kalpakis, G., Vrochidis, S., Kompatsiaris, I. (2018). Adaptive Focused Crawling Using Online Learning. In: Bodrunova, S. (eds) Internet Science. INSCI 2018. Lecture Notes in Computer Science(), vol 11193. Springer, Cham. https://doi.org/10.1007/978-3-030-01437-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-01437-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01436-0
Online ISBN: 978-3-030-01437-7
eBook Packages: Computer ScienceComputer Science (R0)