Encyclopedia of Database Systems

2018 Edition
| Editors: Ling Liu, M. Tamer Özsu

Bpref

  • Nick Craswell
Reference work entry
DOI: https://doi.org/10.1007/978-1-4614-8265-9_489

Definition

Bpref is a preference-based information retrieval measure that considers whether relevant documents are ranked above irrelevant ones. It is designed to be robust to missing relevance judgments, such that it gives the same experimental outcome with incomplete judgments that Mean Average Precision would with complete judgments.

Key Points

In a test collection where all relevant documents have been identified, experiments using bpref and MAP should give the same outcome, for example both systems should agree that system A is better than system B. However, if the relevance judgments are incomplete, for example where only half the pool has been judged, MAP becomes unstable and may incorrectly show that system B is better than system A. The bpref measure was developed to maintain the correct ordering of systems (A better than B) even with incomplete judgments.

Given a ranked list of search results and a set of R known relevant documents and N known irrelevant documents, bpref...
This is a preview of subscription content, log in to check access.

Recommended Reading

  1. 1.
    Buckley C, Voorhees EM. Retrieval evaluation with incomplete information. In: Proceedings of 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval; 2004. p. 25–32.Google Scholar
  2. 2.
    Yilmaz E, Aslam JA. Estimating average precision with incomplete and imperfect judgments. In: Proceedings of International Conference on Information and Knowledge Management; 2006. p. 102–11.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Microsoft Research CambridgeCambridgeUK

Section editors and affiliations

  • Weiyi Meng
    • 1
  1. 1.Dept. of Computer ScienceState University of New York at BinghamtonBinghamtonUSA