Multimedia Retrieval Evaluation
Multimedia Retrieval Evaluation is the activity of measuring the effectiveness of one or more multimedia search techniques. A common way of evaluating multimedia retrieval systems is by comparing them to each other in community wide benchmarks. In such benchmarks participants are invited to submit their retrieval results for a given set of topics, the relevance of the submitted items is checked, and effectiveness measures for each of the submissions are reported. Multimedia retrieval evaluation measures the effectiveness of multimedia retrieval systems or techniques by looking at how well the information need as described by a topic is satisfied by the results retrieved by the system or technique. Efficiency of the techniques is typically not taken into account, but may be studied separately.
Until the mid-1990s, no commonly used evaluation methodology existed for multimedia retrieval. An important reason for this is that the field has merely been a...
- 2.Draper SW, Dunlop MD, Ruthven I, van Rijsbergen CJ, editors. In: Proceedings of the MIRA 99: Evaluating Interactive Information Retrieval. eWiC, Electronic Workshops in Computing; 1999.Google Scholar
- 3.Gunther NJ, Beretta G. A benchmark for image retrieval using distributed systems over the internet: BIRDS-I. Technical report HPL-2000-162, HP laboratories. 2000.Google Scholar
- 4.Hull D. Using statistical testing in the evaluation of retrieval experiments. In: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 1993. p. 329–38.Google Scholar
- 6.Leung CHC, Ho-Shing IPH. Benchmarking for content-based visual information search. In: Advances in Visual Information Systems, 4th International Conference; 2000. p. 442–56.Google Scholar
- 9.Smeaton AF, Over P, Costello CJ, de Vries AP, Doermann D, Hauptmann A, Rorvig ME, Smith JR, Wu L. The TREC-2001 video track: information retrieval on digital video information. In: Proceedings of the 6th European Conference on Research and Advanced Technology for Digital Libraries; 2002. p. 266–75.zbMATHCrossRefGoogle Scholar
- 10.Smith JR. Image retrieval evaluation. In: Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries; 1998. p. 112–3.Google Scholar
- 11.Voorhees EM, Harman DK. Overview of the eighth text retrieval conference (TREC-8). In: Proceedings of the 8th Text Retrieval Conference; 2000.Google Scholar
- 12.Voorhees EM, Harman DK. TREC: experiment and evaluation in information retrieval (digital libraries and electronic publishing). Cambridge: MIT; 2005. ISBN 0262220733.Google Scholar
- 13.Westerveld T. Trecvid as a re-usable test-collection for video retrieval. In: Proceedings of the Multimedia Information Retrieval Workshop; 2005.Google Scholar
- 14.Zobel J. How reliable are the results of large-scale information retrieval experiments? In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 1998. p. 307–14.Google Scholar