Abstract
A constantly growing amount of high-quality information is stored in pages coming from the Hidden Web. Such pages are accessible only through a query interface that a Hidden-Web site provides and may span a variety of topics.
In order to provide centralized access to the Hidden Web, previous works have focused on query generation techniques that aim at downloading all content of a given Hidden Web site with the minimum cost. In certain settings however, we are interested in downloading only a specific part of such a site. For example, in a news database, a user may be interested in retrieving only sports articles but no politics. In this case, we need to make the best use of our resources in downloading only the portion of the Hidden Web site that we are interested in.
In this paper, we study how we can build a topically-focused Hidden Web crawler that can autonomously extract topic-specific pages from the Hidden Web by searching only the subset that is related to the corresponding category. To this end, we present query generation techniques that take into account the topic that we are interested in. We propose a number of different crawling policies and we experimentally evaluate them with data from two popular sites.
Partially supported by PIRG06-GA-2009-256603.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
The Open Directory Project, http://www.dmoz.org
Stack Exchange, http://stackexchange.com/
Álvarez, M., Raposo, J., Pan, A., Cacheda, F., Bellas, F., Carneiro, V.: Deepbot: a focused crawler for accessing hidden web content. In: Proc. of Int. Workshop on Data Enginering Issues in E-commerce and Services, DEECS 2007, NY, USA (2007)
Angkawattanawit, N., Rungsawang, A.: Learnable crawling: An efficient approach to topic-specific web resource discovery (2002)
Barbosa, L., Freire, J.: Siphoning hidden-web data through keyword-based interfaces. In: Proceedings of SBBD, Brazil, (2004)
Barbosa, L., Freire, J.: An adaptive crawler for locating hidden-web entry points. In: Proceedings of the WWW Conference, NY, USA (2007)
Baykan, E., Henzinger, M., Marian, L., Weber, I.: Purely url-based topic classification. In: Proceedings of the WWW Conference, Madrid, Spain (2009)
Bergholz, A., Chidlovskii, B.: Crawling for domain-specific hidden web resources. In: Proceedings of WISE, DC, USA (2003)
Chakrabarti, S., van den Berg, M., Dom, B.: Focused crawling: a new approach to topic-specific web resource discovery. In: Proceedings of the WWW Conference, NY, USA (1999)
Diligenti, M., Coetzee, F., Lawrence, S., Giles, C.L., Gori, M.: Focused crawling using context graphs. In: Proceedings of VLDB, CA, USA (2000)
Ipeirotis, P.G., Gravano, L.: Distributed search over the hidden web: hierarchical database sampling and selection. In: Proceedings of VLDB, Hong Kong (2002)
Ipeirotis, P.G., Gravano, L., Sahami, M.: Probe, count, and classify: categorizing hidden web databases. SIGMOD Rec. 30, 67–78 (2001)
Liu, W., Xiao, J., Yang, J.: A sample-guided approach to incremental structured web database crawling. In: Proceedings of ICIA, Harbin, China (2010)
Madhavan, J., Ko, D., Kot, L., Ganapathy, V., Rasmussen, A., Halevy, A.: Google’s deep web crawl. In: Proceedings of VLDB, Auckland, New Zealand (2008)
Menczer, F., Pant, G., Srinivasan, P.: Topic-driven crawlers: Machine learning issues. ACM TOIT (submitted, 2002)
Noh, S., Choi, Y., Seo, H., Choi, K., Jung, G.: An Intelligent Topic-Specific Crawler Using Degree of Relevance. In: Yang, Z.R., Yin, H., Everson, R.M. (eds.) IDEAL 2004. LNCS, vol. 3177, pp. 491–498. Springer, Heidelberg (2004)
Ntoulas, A., Zerfos, P., Cho, J.: Downloading textual hidden web content through keyword queries. In: Proceedings of JCDL, NY, USA (2005)
Raghavan, S., Garcia-Molina, H.: Crawling the hidden web. In: Proceedings of VLDB, San Francisco, CA, USA (2001)
Seshadri, S., Cooper, B.F.: Routing queries through a peer-to-peer infobeacons network using information retrieval techniques. IEEE TPDS 18, 1754–1765 (2007)
Wang, Y., Lu, J., Chen, J.: Crawling deep web using a new set covering algorithm. In: Proceedings of the ADMA Conference, Berlin, Heidelberg (2009)
Wu, P., Wen, J.-R., Liu, H., Ma, W.-Y.: Query selection techniques for efficient crawling of structured web sources. In: Proceedings of the ICDE, Washington, DC, USA (2006)
Wu, W., Yu, C., Doan, A., Meng, W.: An interactive clustering-based approach to integrating source query interfaces on the deep web. In: Proceedings of SIGMOD, NY, USA (2004)
Yang, Y., Bansal, N., Dakka, W., Ipeirotis, P., Koudas, N., Papadias, D.: Query by document. In: Proceedings of WSDM, NY, USA (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Liakos, P., Ntoulas, A. (2012). Topic-Sensitive Hidden-Web Crawling. In: Wang, X.S., Cruz, I., Delis, A., Huang, G. (eds) Web Information Systems Engineering - WISE 2012. WISE 2012. Lecture Notes in Computer Science, vol 7651. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-35063-4_39
Download citation
DOI: https://doi.org/10.1007/978-3-642-35063-4_39
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-35062-7
Online ISBN: 978-3-642-35063-4
eBook Packages: Computer ScienceComputer Science (R0)