Abstract
We consider Information Retrieval evaluation, especially at Trec with the trec_eval program. It appears that systems obtain scores regarding not only the relevance of retrieved documents, but also according to document names in case of ties (i.e., when they are retrieved with the same score). We consider this tie-breaking strategy as an uncontrolled parameter influencing measure scores, and argue the case for fairer tie-breaking strategies. A study of 22 Trec editions reveals significant differences between the Conventional unfair Trec’s strategy and the fairer strategies we propose. This experimental result advocates using these fairer strategies when conducting evaluations.
This work was partly realized as part of the Quaero Programme, funded by OSEO, French State agency for innovation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Robertson, S.: On the history of evaluation in IR. J. Inf. Sci. 34(4), 439–456 (2008)
Harman, D.K. (ed.): TREC-1: Proceedings of the First Text REtrieval Conference, Gaithersburg, MD, USA, NIST (February 1993)
Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)
NIST: README file for trec_eval 8.1, http://trec.nist.gov/trec_eval
Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H., Hidaka, S.: Overview of IR Tasks at the First NTCIR Workshop. In: Proceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, NACSIS, pp. 11–44 (1999)
Peters, C., Braschler, M.: European Research Letter – Cross-Language System Evaluation: the CLEF Campaigns. J. Am. Soc. Inf. Sci. Technol. 52(12), 1067–1072 (2001)
Clough, P., Sanderson, M.: The CLEF 2003 Cross Language Image Retrieval Track. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (eds.) CLEF 2003. LNCS, vol. 3237, pp. 581–593. Springer, Heidelberg (2004)
Voorhees, E.M.: Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference, pp. 315–323. ACM, New York (1998)
Zobel, J.: How Reliable are the Results of large-scale Information Retrieval Experiments? In: SIGIR 1998: Proceedings of the 21st Annual International ACM SIGIR Conference, pp. 307–314. ACM Press, New York (1998)
Buckley, C., Voorhees, E.M.: Retrieval System Evaluation. In: [3], ch. 3, pp. 53–75
Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (July 2008)
Voorhees, E.M.: Overview of the TREC 2004 Robust Track. In: Voorhees, E.M., Buckland, L.P. (eds.) TREC 2004: Proceedings of the 13th Text REtrieval Conference, Gaithersburg, MD, USA, NIST (2004)
Sanderson, M., Zobel, J.: Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference, pp. 162–169. ACM, New York (2005)
Hull, D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In: SIGIR 1993: Proceedings of the 16th annual international ACM SIGIR conference, pp. 329–338. ACM Press, New York (1993)
Raghavan, V., Bollmann, P., Jung, G.S.: A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst. 7(3), 205–229 (1989)
McSherry, F., Najork, M.: Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 414–421. Springer, Heidelberg (2008)
Di Nunzio, G.M., Ferro, N.: DIRECT: A System for Evaluating Information Access Components of Digital Libraries. In: Rauber, A., Christodoulakis, S., Tjoa, A.M. (eds.) ECDL 2005. LNCS, vol. 3652, pp. 483–484. Springer, Heidelberg (2005)
Joachims, T., Li, H., Liu, T.Y., Zhai, C.: Learning to Rank for Information Retrieval (LR4IR 2007). SIGIR Forum 41(2), 58–62 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cabanac, G., Hubert, G., Boughanem, M., Chrisment, C. (2010). Tie-Breaking Bias: Effect of an Uncontrolled Parameter on Information Retrieval Evaluation. In: Agosti, M., Ferro, N., Peters, C., de Rijke, M., Smeaton, A. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2010. Lecture Notes in Computer Science, vol 6360. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15998-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-15998-5_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15997-8
Online ISBN: 978-3-642-15998-5
eBook Packages: Computer ScienceComputer Science (R0)