Abstract
System combination has been shown to improve overall performance on many rank-based retrieval tasks, often by combining results from multiple systems into a single ranked list. In contrast, set-based retrieval tasks call for a technique to combine results in ways that require decisions on whether each document is in or out of the result set. This paper presents a set-generating unsupervised system combination framework that draws inspiration from evaluation techniques in sparse data settings. It argues for the existence of a duality between evaluation and system combination, and then capitalizes on this duality to perform unsupervised system combination. To do this, the framework relies on the consensus of the systems to estimate latent “goodness” for each system. An implementation of this framework using data programming is compared to other unsupervised system combination approaches to demonstrate its effectiveness on CLEF and MATERIAL collections.
This work has been supported in part by IARPA/AFRL contract FA8650-17-C-9117.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For the MATERIAL Somali and Swahili collections, \(\zeta =40\). For the CLEF French collection, \(\zeta =240\) to account for the fact that \(\frac{N_{\text {total}}}{N_{\text {relevant}}}\) for CLEF French is 6 times more than that of MATERIAL Somali or Swahili. See Table 1.
- 2.
For MATERIAL Swahili and Somali, development and evaluation collections are provided by IARPA. For CLEF French, we selected query sets 2000–2003 with document sets ATS 94, Le Monde 94 as the development collection, and query sets 2004–2006 with document sets ATS 95, Le Monde 95 as the evaluation collection.
References
Belkin, N., et al.: Combining the evidence of multiple query representations for information retrieval. IP&M 31(3), 431–448 (1995)
de Borda, J.C.: Mémoire sur les élections au scrutin (1781)
Cormack, G., et al.: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: SIGIR, vol. 9, pp. 758–759 (2009)
Darwish, K., Oard, D.: Probabilistic structured query methods. In: SIGIR (2003)
Dempster, A., et al.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc. 39(1), 1–22 (1977)
Fiscus, J.G., Ajot, J., Garofolo, J.S., Doddingtion, G.: Results of the 2006 spoken term detection evaluation. In: SIGIR, vol. 7, pp. 51–57 (2007)
Haddow, B., et al.: The University of Edinburgh’s submissions to the WMT18 news translation task. In: WMT, Belgium, Brussels, pp. 403–413, October 2018
Harman, D.: Overview of the third text retrieval conference. In: TREC (1995)
Karakos, D., et al.: Score normalization and system combination for improved keyword spotting. In: ASRU, pp. 210–215 (2013)
Klementiev, A., Roth, D., Small, K.: Unsupervised rank aggregation with distance-based models. In: ICML, pp. 472–479 (2008)
Koehn, P., et al.: Moses: open source toolkit for statistical machine translation. In: ACL, pp. 177–180 (2007)
Lee, J., et al.: Analyses of multiple evidence combination. In: SIGIR (1997)
NIST: The Official Original Derivation of AQWV (2017). https://www.nist.gov/sites/default/files/documents/2017/10/26/aqwv_derivation.pdf
Niu, X., et al.: Bi-directional differentiable input reconstruction for low-resource neural machine translation. CoRR abs/1811.01116 (2018)
Di Nunzio, G.M., Ferro, N., Mandl, T., Peters, C.: CLEF 2006: ad hoc track overview. In: Peters, C., et al. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 21–34. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74999-8_3
Nuray, R., Can, F.: Automatic ranking of information retrieval systems using data fusion. IP&M 42(3), 595–614 (2006)
Ratner, A., et al.: Data programming: creating large training sets, quickly. In: NIPS, pp. 3567–3575 (2016)
Shaw, J., Fox, E.: Combination of multiple searches. In: TREC (1994)
Soboroff, I., et al.: Ranking retrieval systems without relevance judgments. In: SIGIR, pp. 66–73 (2001)
Wang, Y., Metze, F.: An in-depth comparison of keyword specific thresholding and sum-to-one score normalization. In: ISCA (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Shing, HC., Barrow, J., Galuščáková, P., Oard, D.W., Resnik, P. (2019). Unsupervised System Combination for Set-Based Retrieval with Expectation Maximization. In: Crestani, F., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2019. Lecture Notes in Computer Science(), vol 11696. Springer, Cham. https://doi.org/10.1007/978-3-030-28577-7_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-28577-7_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28576-0
Online ISBN: 978-3-030-28577-7
eBook Packages: Computer ScienceComputer Science (R0)