Information System Evaluation
Interest in the evaluation techniques for Information Retrieval Systems has significantly increased with the commercial use of information retrieval technologies in the everyday life of the millions of users of the Internet. Until 1993 the evaluations were done primarily by academicians using a few small, well known corpora of test documents or even smaller test databases created within academia. The evaluations focused primarily on the effectiveness of search algorithms. The creation of the annual Text Retrieval Evaluation Conference (TREC) sponsored by the Defense Advanced Research Projects Agency (DARPA) and the National Institute of Standards and Technology (NIST) changed the standard process of evaluating information systems. Conferences have been held every year, starting from 1992, usually in the Fall months. The conference provides a standard database consisting of gigabytes of test data, search statements and the expected results from the searches to academic researchers and commercial companies for testing of their systems. This has placed a standard baseline into comparisons of algorithms. Although there is now a standard database, there is still debate on the accuracy and utility of the results from use of the test corpus. Section 10.2 introduces the measures that are available for evaluating information systems. The techniques are compared stressing their utility from an academic as well as a commercial perspective. Section 10.3 gives examples of results from major comparisons of information systems and algorithms.
KeywordsQuery Expansion Relevant Item Search Statement Information Retrieval System Test Database
Unable to display preview. Download preview PDF.