Scalable Web Search by Adaptive Online Agents: An InfoSpiders Case Study

Abstract

The trend of the recent years in distributed information environments is a good example of the life-like complexity that we expect to observe in most aspects of information and computational science. The explosion of the Web and electronic mail, multiplying the number of information providers and consumers many times over and bringing the Internet inside the average home, has created formidable new opportunities and challenges in almost every area of computer and information science.

In an effort to address such problems, researchers in artificial intelligence and information retrieval have already been successful in developing agent-based techniques to automate many tedious tasks and facilitate the management of the growing amounts of information flooding users. But the work has just begun. There is still much need for tools to assist users in ways that scale with the growth of the Web, and adapt to both the personal preferences of the user and the changes in user and environmental conditions.

This chapter discusses an agent-based approach to building scalable information searching algorithms. For systems designed to let users locate relevant information in highly distributed and decentralized databases, such as the Web, we argue that scalability is one of the main limitations of th current state of the art. Given such an ambitious goal, it probably comes as no surprise that the solution proposed here draws on many ideas and issues discussed in other parts of this book: cooperation in multi-agent systems, information chain economy in rational agents, and spawning and security in mobile agents.