Abstract
Inferring the distributions of relevant and nonrelevant documents over a ranked list of scored documents returned by a retrieval system has a broad range of applications including information filtering, recall-oriented retrieval, metasearch, and distributed IR. Typically, the distribution of documents over scores is modeled by a mixture of two distributions, one for the relevant and one for the nonrelevant documents, and expectation maximization (EM) is run to estimate the mixture parameters. A large volume of work has focused on selecting the appropriate form of the two distributions in the mixture. In this work we consider the form of the distributions as a given and we focus on the inference algorithm. We extend the EM algorithm (a) by simultaneously considering the ranked lists of documents returned by multiple retrieval systems, and (b) by encoding in the algorithm the constraint that the same document retrieved by multiple systems should have the same, global, probability of relevance. We test the new inference algorithm using TREC data and we demonstrate that it outperforms the regular EM algorithm. It is better calibrated in inferring the probability of document’s relevance, and it is more effective when applied on the task of metasearch.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Arampatzis, A., Kamps, J.: A signal-to-noise approach to score normalization. In: Proceeding of the 18th CIKM 2009, pp. 797–806. ACM, New York (2009)
Arampatzis, A., Kamps, J., Robertson, S.: Where to stop reading a ranked list?: threshold optimization using truncated score distributions. In: Proceedings 32nd ACM SIGIR, pp. 524–531. ACM, New York (2009)
Arampatzis, A., Robertson, S.: Modeling score distributions in information retrieval. Information Retrieval 14, 26–46 (2011)
Arampatzis, A., van Hameren, A.: The score-distributional threshold optimization for adaptive binary classification tasks. In: Proceedings of the 24th ACM SIGIR, pp. 285–293. ACM, New York (2001)
Aslam, J.A., Yilmaz, E.: Inferring document relevance from incomplete information. In: Proceedings of the 6th CIKM, pp. 633–642. ACM Press (November 2007)
Aslam, J.A., Yilmaz, E., Pavlu, V.: The maximum entropy method for analyzing retrieval measures. In: Proceedings of the 28th ACM SIGIR, pp. 27–34. ACM Press (August 2005)
Baumgarten, C.: A probabilistic solution to the selection and fusion problem in distributed information retrieval. In: Proceedings of the 22nd ACM, pp. 246–253. ACM, New York (1999)
Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Heidelberg (2006)
Bookstein, A.: When the most “pertinent” document should not be retrieved—an analysis of the swets model. Information Processing & Management 13(6), 377–383 (1977)
Dai, K., Kanoulas, E., Pavlu, V., Aslam, J.A.: Variational bayes for modeling score distributions. Information Retrieval 14(1), 47–67 (2011)
Kanoulas, E., Pavlu, V., Dai, K., Aslam, J.A.: Modeling the Score Distributions of Relevant and Non-relevant Documents. In: Azzopardi, L., Kazai, G., Robertson, S., Rüger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 152–163. Springer, Heidelberg (2009)
Kanoulas, E., Dai, K., Pavlu, V., Aslam, J.A.: Score distribution models: assumptions, intuition, and robustness to score manipulation. In: SIGIR 2010: Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 242–249. ACM, New York (2010), http://portal.acm.org/citation.cfm?doid=1835449.1835491
Manmatha, R., Rath, T., Feng, F.: Modeling score distributions for combining the outputs of search engines. In: Proceedings of the 24th SIGIR, pp. 267–275. ACM, New York (2001)
Robertson, S.: On Score Distributions and Relevance. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECIR 2007. LNCS, vol. 4425, pp. 40–51. Springer, Heidelberg (2007)
Swets, J.A.: Information retrieval systems. Science 141(3577), 245–250 (1963)
Swets, J.A.: Effectiveness of information retrieval methods. American Documentation 20, 72–89 (1969)
Zhang, Y., Callan, J.: Maximum likelihood estimation for filtering thresholds. In: Proceedings of the 24th ACM SIGIR, pp. 294–302. ACM, New York (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Dai, K., Pavlu, V., Kanoulas, E., Aslam, J.A. (2012). Extended Expectation Maximization for Inferring Score Distributions. In: Baeza-Yates, R., et al. Advances in Information Retrieval. ECIR 2012. Lecture Notes in Computer Science, vol 7224. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28997-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-642-28997-2_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-28996-5
Online ISBN: 978-3-642-28997-2
eBook Packages: Computer ScienceComputer Science (R0)