Evaluation Metrics for Structured Text Retrieval
Evaluation of XML retrieval effectiveness; Performance metrics
Anevaluation metric is used to evaluate the effectiveness of information retrieval systems and to justify theoretical and/or pragmatic developments of these systems. It consists of a set of measures that follow a common underlying evaluation methodology.
There are many metrics that can be used to evaluate the effectiveness of structured text retrieval systems. These metrics are based on different evaluation assumptions, incorporate different hypotheses of the expected user behavior, and implement their own evaluation methodologies to handle the level of overlap among the units of retrieval.
Over the past 5 years, the initiative for the evaluation of XML retrieval (INEX) has investigated various aspects of structured text retrieval, by particularly focusing on XML retrieval. Major advances, both in terms of approaches to XML retrieval and evaluation of XML retrieval, have been made...
- 3.Kazai G. Choosing an ideal recall-base for the evaluation of the focused task: sensitivity analysis of the XCG evaluation measures. In: Proceedings of the Comparative Evaluation of XML Information Retrieval Systems: Fifth Workshop of the Initiative for the Evaluation of XML Retrieval; 2007. p. 35–44.Google Scholar
- 4.Kazai G, Lalmas M. Notes on what to measure in INEX. In: Proceedings of the INEX 2005 Workshop on Element Retrieval Methodology; 2005. p. 22–38.Google Scholar
- 6.Kazai G, Lalmas M, de Vries AP. The overlap problem in content-oriented XML retrieval evaluation. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2004. p. 72–9.Google Scholar
- 7.Pehcevski J. Evaluation of effective XML information retrieval. RMIT University, Melbourne, Australia, Ph.D. thesis, 2006.Google Scholar
- 8.Pehcevski J, Thom JA. HiXEval: highlighting XML retrieval evaluation. In: Advances in XML Information Retrieval and Evaluation: Proceedings of the Fourth Workshop of the Initiative for the Evaluation of XML Retrieval; 2006. p. 43–57.Google Scholar
- 9.Piwowarski B, Dupret G. Evaluation in (XML) information retrieval: expected precision-recall with user modelling (EPRUM). In: Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2006. p. 260–7.Google Scholar
- 10.Piwowarski B, Gallinari P. Expected ratio of relevant units: a measure for structured information retrieval. In: Proceedings of the 2nd International Workshop of the Initiative for the Evaluation of XML Retrieval; 2003. p. 158–66.Google Scholar
- 13.de Vries A, Kazai G, Lalmas M. Tolerance to irrelevance: a user-effort evaluation of retrieval systems without predefined retrieval unit. In: Proceedings of the 7th International Conference on Computer-Assisted Information Retrieval; 2004. p. 463–73.Google Scholar
- 14.Woodley A. and Geva S. XCG overlap at INEX 2004. In: Proceedings of the 4th International Workshop of the Initiative for the Evaluation of XML Retrieval; 2005. p. 25–39.Google Scholar