Abstract
Human computation techniques, such as crowdsourcing and games, have demonstrated their ability to accomplish portions of information retrieval (IR) tasks that machine-based techniques find challenging. Query refinement is one such IR task that may benefit from human involvement. We conduct an experiment that evaluates the contributions of participants from Amazon Mechanical Turk (N = 40). Each of our crowd participants is randomly assigned to use one of two query interfaces: a traditional web-based interface or a game-based interface. We ask each participant to manually construct queries to respond to a set of OHSUMED information needs and we calculate their resulting recall and precision. Those using a web interface are provided feedback on their initial queries and asked to use this information to reformulate their original queries. Game interface users are provided with instant scoring and asked to refine their queries based on their scores. In our experiment, crowdsourcing-based methods in general provide a significant improvement over machine algorithmic methods, and among crowdsourcing methods, games provide a better mean average precision (MAP) for query reformulations as compared to a non-game interface.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Pseudo relevance feedback, also known as blind relevance feedback, automates the manual part of relevance feedback through local document analysis. The pseudo relevance feedback method is to perform normal retrieval to find an initial set of most relevant documents, assume that the top “k” ranked documents are relevant, and then perform relevance feedback techniques as before under this assumption. Evidence suggests that this method tends to work better than global document analysis (Xu and Croft 1996).
References
Ageev M, Guo Q, Lagun D, Agichtein E (2011) Find it if you can: a game for modeling different types of web search success using interaction data. In: Proceedings of SIGIR’11. ACM, New York, pp 345–354
Allan J, Papka R, Lavrenko V (1998) On-line new event detection and tracking. In: Proceedings of SIGIR’98. ACM, New York, pp 37–45
Alonso O, Lease M (2011) Crowdsourcing for information retrieval: principles, methods, and applications. In: Proceedings of SIGIR’11. ACM, New York, pp 1299–1300
Alonso O, Mizzaro S (2012) Using crowdsourcing for TREC relevance assessment. Inf Process Manage 48(6):1053–1066
Anick P (2003) Using terminological feedback for web search refinement: a log-based study. In: Proceedings of SIGIR’03. ACM, New York, pp 88–95
Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of SIGIR’04. ACM, New York, pp 25–32
Dasdan A, Drome C, Kolay S, Alpern M, Han A, Chi T, Hoover J, Davtchev I, Verma S (2009) Thumbs-Up: a game for playing to rank search results. In: Proceedings of the ACM SIGKDD workshop on human computation, Paris. ACM, New York, pp 36–37
Efthimiadis EN (2000) Interactive query expansion: a user-based evaluation in a relevance feedback environment. J Am Soc Inf Sci 51(11):989–1003
Harris CG (2012) An evaluation of search strategies for user-generated video content. In: Proceedings of the WWW Workshop on Crowdsourcing Web Search (Lyon, France), pp 48–53
Harris CG, Srinivasan P (2012) Applying human computation mechanisms to information retrieval. Proc Am Soc Inf Sci Technol 49(1):1–10
Harris CG, Srinivasan P (2013) Comparing crowd-based, game-based, and machine-based approaches in initial query and query refinement tasks. In: Advances in information retrieval. Springer, Berlin Heidelberg, pp 495–506
Hersh W, Buckley C, Leone T, Hickam D (1994) OHSUMED: an interactive retrieval evaluation and new large test collection for research. In: Proceedings of SIGIR’94. Springer, London, pp 192–201
Jones KS (1972) A statistical interpretation of term specificity and its application in retrieval. J Doc 28(1):11–21
Law E, Ahn L von, Mitchell T (2009) Search war: a game for improving web search. In: Proceedings of the ACM SIGKDD workshop on human computation, Paris. ACM, New York, pp 31–31
Lease M, Yilmaz E (2012) Crowdsourcing for information retrieval. ACM, New York, SIGIR Forum 45:2 (January 2012), pp 66–75
McKibbon KA, Haynes RB, Walker Dilks CJ, Ramsden MF, Ryan NC, Baker L, Flemming T, Fitzgerald D (1990) How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Comput Biomed Res 23(6):583–593
Milne D, Nichols DM, Witten IH (2008) A competitive environment for exploratory query expansion. In: Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries (JCDL’08). ACM, New York, pp 197–200
Robertson SE, Walker S, Jones S, Hancock-Beaulieu MM, Gatford M (1995) Okapi at TREC-3. NIST Special Publication SP-1995. Gaithersburg, Maryland, USA, pp 109–121
Ruthven I (2003) Re-examining the potential effectiveness of interactive query expansion. In: Proceedings of SIGIR’03. ACM, New York, pp 213–220
Spink A, Jansen BJ, Wolfram D, Saracevic T (2002) From e-sex to e-commerce: web search changes. Computer 35(3):107–109
Strohman T, Metzler D, Turtle H, Croft WB (2005) Indri: a language model-based search engine for complex queries. In: Proceedings of the international conference on intelligence analysis, McLean, VA. Poster, 2–6 May 2005
Xu J, Croft WB (1996) Query expansion using local and global document analysis. In: Proceedings of SIGIR’96. ACM, New York, pp 4–11
Yan T, Kumar V, Ganesan D (2010) CrowdSearch: exploiting crowds for accurate real-time image search on mobile phones. In: Proceedings of MobiSys’10. ACM, New York, pp 77–90
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Harris, C.G., Srinivasan, P. (2013). Human Computation for Information Retrieval. In: Michelucci, P. (eds) Handbook of Human Computation. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8806-4_18
Download citation
DOI: https://doi.org/10.1007/978-1-4614-8806-4_18
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-8805-7
Online ISBN: 978-1-4614-8806-4
eBook Packages: Computer ScienceComputer Science (R0)