Abstract
We investigate which retrieval measures successfully discern robustness gains in the monolingual (Bulgarian, French, Hungarian, Portuguese and English) information retrieval tasks of the Ad-Hoc Track of the Cross-Language Evaluation Forum (CLEF) 2006. In all 5 of our experiments with blind feedback (a technique known to impair robustness across topics), the mean scores of the Average Precision, Geometric MAP and Precision@10 measures increased (and most of these increases were statistically significant), implying that these measures are not suitable as robust retrieval measures. In contrast, we found that measures based on just the first relevant item, such as a Generalized Success@10 measure, successfully discerned some robustness gains, particularly the robustness advantage of expanding Title queries by using the Description field instead of blind feedback.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Chen, H., Karger, D.R.: Less is More: Probabilistic Models for Retrieving Fewer Relevant Documents. In: SIGIR 2006, pp. 429–436 (2006)
Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2006: Ad Hoc Track Overview. LNCS, vol. 4730, Springer, Heidelberg (to appear)
Robertson, S.: On GMAP – and other transformations. In: CIKM 2006, pp. 78–83 (2006)
Savoy, J.: CLEF and Multilingual information retrieval resource page http://www.unine.ch/info/clef/
Tomlinson, S.: CJK Experiments with Hummingbird SearchServerTM at NTCIR-5. In: Proceedings of NTCIR-5 (2005)
Tomlinson, S.: Early Precision Measures: Implications from the Downside of Blind Feedback. In: SIGIR 2006, pp. 705–706 (2006)
Tomlinson, S.: Enterprise, QA, Robust and Terabyte Experiments with Hummingbird SearchServerTM at TREC 2005. In: Proceedings of TREC 2005 (2006)
Tomlinson, S.: European Ad Hoc Retrieval Experiments with Hummingbird SearchServerTM at CLEF 2005. In: Working Notes for the CLEF 2005 Workshop (2005)
Voorhees, E.M.: Overview of the TREC 2003 Robust Retrieval Track. In: Proceedings of TREC 2003 (2004)
Voorhees, E.M.: Overview of the TREC 2004 Robust Retrieval Track. In: Proceedings of TREC 2004 (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tomlinson, S. (2007). Comparing the Robustness of Expansion Techniques and Retrieval Measures. In: Peters, C., et al. Evaluation of Multilingual and Multi-modal Information Retrieval. CLEF 2006. Lecture Notes in Computer Science, vol 4730. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74999-8_18
Download citation
DOI: https://doi.org/10.1007/978-3-540-74999-8_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74998-1
Online ISBN: 978-3-540-74999-8
eBook Packages: Computer ScienceComputer Science (R0)