Abstract
An essential component of information mining and pattern discovery on the Web is the Web Crawling Agent (WCA). General-purpose Web Crawling Agents, which were briefly described in Chapter 1, are intended to be used for building generic portals. The diverse and voluminous nature of Web documents presents formidable challenges to the design of high performance WCAs. They require both powerful processors and a tremendous amount of storage, and yet even then can only cover restricted portions of the Web. Nonetheless, despite their fundamental importance in providing Web services, the design of WCAs is not well-documented in the literature. This chapter describes the conceptual design and implementation of Web crawling agents.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer Science+Business Media New York
About this chapter
Cite this chapter
Chang, G., Healey, M.J., McHugh, J.A.M., Wang, J.T.L. (2001). Web Crawling Agents. In: Mining the World Wide Web. The Information Retrieval Series, vol 10. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-1639-2_8
Download citation
DOI: https://doi.org/10.1007/978-1-4615-1639-2_8
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-5654-7
Online ISBN: 978-1-4615-1639-2
eBook Packages: Springer Book Archive