Abstract
Magnitude estimation is a psychophysical scaling technique whereby numbers are assigned to stimuli to reflect the ratios of their perceived intensity. We report on a crowdsourcing experiment aimed at understanding if magnitude estimation can be used to gather reliable relevance judgements for documents, as is commonly required for test collection-based evaluation of information retrieval systems. Results on a small dataset show that: (i) magnitude estimation can produce relevance rankings that are consistent with more classical ordinal judgements; (ii) both an upper-bounded and an unbounded scale can be used effectively, though with some differences; (iii) the presentation order of the documents being judged has a limited effect, if any; and (iv) only a small number repeat judgements are required to obtain reliable magnitude estimation scores.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Eisenberg, M.: Measuring relevance judgements. Information Processing and Management 24, 373–389 (1988)
Gescheider, G.: Psychophysics: The Fundamentals. Lawrence Erlbaum Associates, 3rd edn. (1997)
McGee, M.: Usability magnitude estimation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47(4), 691–695 (2003)
Moskowitz, H.R.: Magnitude estimation: notes on what, how, when, and why to use it. Journal of Food Quality 1(3), 195–227 (1977)
Sormunen, E.: Liberal relevance criteria of TREC: Counting on negligible documents? In: 25th SIGIR, pp. 324–330. ACM, New York (2002)
Spink, A., Greisdorf, H.: Regions and levels: Measuring and mapping users’ relevance judgments. JASIST 52(2), 161–173 (2001)
Stevens, S.S.: A metric for the social consensus. Science 151(3710), 530–541 (1966)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Maddalena, E., Mizzaro, S., Scholer, F., Turpin, A. (2015). Judging Relevance Using Magnitude Estimation. In: Hanbury, A., Kazai, G., Rauber, A., Fuhr, N. (eds) Advances in Information Retrieval. ECIR 2015. Lecture Notes in Computer Science, vol 9022. Springer, Cham. https://doi.org/10.1007/978-3-319-16354-3_23
Download citation
DOI: https://doi.org/10.1007/978-3-319-16354-3_23
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16353-6
Online ISBN: 978-3-319-16354-3
eBook Packages: Computer ScienceComputer Science (R0)