Neural Agent for Text Database Discovery

Abstract

As the number and diversity of text databases on the Internet increases rapidly, users are faced with finding the text databases that are relevant to the user query. Identifying the relevant text databases out of many candidates for a given query is called the text database discovery problem. In this paper, we propose a novel approach, a neural approach, to the text database discovery problem. First, we present a neural agent that learns about underlying text databases from the user’s relevance feedback. For a given query, the neural agent, which is sufficiently trained on the basis of the neural net learning mechanism, discovers the text databases associated with the relevant documents and retrieves those documents effectively. In order to scale our approach with the large number of text databases, we also propose the hierarchical organization of neural agents which reduces the total training cost at the acceptable level. Finally, we evaluate the performance of our approach by comparing it to those of the conventional well-known approaches.