Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Recommended Reading
Cho J, Garcia-Molina H. Effective page refresh policies for web crawlers. ACM Trans Database Syst. 2003;28(4):390–426.
Coffman Jr EG, Liu Z, Weber RR. Optimal robot scheduling for web search engines. J Sched. 1998;1(1):15–29.
Dikaiakos MD, Stassopoulou A, Papageorgiou L. An investigation of web crawler behavior: characterization and metrics. Comput Commun. 2005;28(8):880–97.
Edwards J, McCurley KS, Tomlin J. An adaptive model for optimizing performance of an incremental web crawler. In: Proceedings of the 10th International World Wide Web Conference; 2001. p. 106–13.
Fielding R, Gettys J, Mogul J, Frystyk H, Mastinter L, Leach P, Berners-Lee T. Hypertext transfer protocol – HTTP/1.1, RFC 2616 http://www.w3.org/Protocols/rfc2616/rfc2616.html
Podlipnig S, Böszörmenyi L. A survey of web cache replacement strategies. ACM Comput Surv. 2003;35(4):374–98.
Sitemap protocol specification. http://www.sitemaps.org/protocol.php
Wang J. A survey of web caching schemes for the internet. ACM SIGCOMM Comput Commun Rev. 1999;29(5):36–46.
Yuan X, MacGregor MH, Harms J. An efficient scheme to remove crawler traffic from the internet. In: Proceedings of the 11th International Conference on Computer Communications and Networks; 2002. p. 90–5.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2018 Springer Science+Business Media, LLC, part of Springer Nature
About this entry
Cite this entry
McCurley, K.S. (2018). Incremental Crawling. In: Liu, L., Özsu, M.T. (eds) Encyclopedia of Database Systems. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8265-9_196
Download citation
DOI: https://doi.org/10.1007/978-1-4614-8265-9_196
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-8266-6
Online ISBN: 978-1-4614-8265-9
eBook Packages: Computer ScienceReference Module Computer Science and Engineering