Abstract
The notion of having a “living lab” to undertaken evaluations has been proposed by a number of proponents within the field of Information Retrieval (IR). However, what such a living lab might look like and how it might be setup has not been discussed in detail. Living labs have a number of appealing points such as realistic evaluation contexts where tasks are directly linked to user experience and the closer integration of research/academia and development/industry facilitating more efficient knowledge transfer. However, operationalizing a living lab opens up a number of concerns regarding security, privacy, etc. as well as challenges regarding the design, development and maintenance of the infrastructure required to support such evaluations. Here, we aim to further the discussion on living labs for IR evaluation and propose one possible architecture to create such an evaluation environment. To focus discussion, we put forward a proposal for a living lab on product search tasks within the context of an online shop.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Allan, J.: Hard track overview in trec 2003: High accuracy retrieval from documents. In: Text REtrieval Conference (2003)
Azzopardi, L., Järvelin, K., Kamps, J., Smucker, M.D.: SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum 44, 35–47 (2011)
Belkin, N.J.: Some(what) grand challenges for information retrieval. SIGIR Forum 42(1), 47–54 (2008)
Broder, A., Ciaramita, M., Fontoura, M., Gabrilovich, E., Josifovski, V., Metzler, D., Murdock, V., Plachouras, V.: To swing or not to swing: learning when (not) to advertise. In: Proc. of the 17th ACM Conf. on Information and Knowledge Management, CIKM 2008, pp. 1003–1012 (2008)
Dumais, S.T., Belkin, N.J.: The trec interactive tracks: Putting the user into search. In: Text REtrieval Conference (2005)
Jarvenpaa, S.L., Todd, P.A.: Consumer reactions to electronic shopping on the world wide web. Int. J. Electron. Commerce 1, 59–88 (1996)
Kamps, J., Geva, S., Peters, C., Sakai, T., Trotman, A., Voorhees, E.: Report on the SIGIR 2009 workshop on the future of IR evaluation. SIGIR Forum 43, 13–23 (2009)
Kelly, D.: Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval 3, 1–224 (2009)
Kelly, D., Dumais, S., Pedersen, J.O.: Evaluation challenges and directions for info. seeking support systems. Computer 42(3), 60–66 (2009)
Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Discov. 18, 140–181 (2009)
Pirolli, P.: Powers of 10: Modeling complex information-seeking systems at multiple scales. Computer 42, 33–40 (2009)
Sanderson, M.: Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval (FnTIR) 4(4), 247–375 (2010)
Sparck-Jones, K.: Information Retrieval Experiment. Butterworth & Co. (1981)
Voorhees, E.M., Harman, D.K.: TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)
White, R.W., Muresan, G., Marchionini, G.: SIGIR 2006 Workshop on Evaluating Exploratory Search Systems. SIGIR Forum 40, 52–60 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Azzopardi, L., Balog, K. (2011). Towards a Living Lab for Information Retrieval Research and Development. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds) Multilingual and Multimodal Information Access Evaluation. CLEF 2011. Lecture Notes in Computer Science, vol 6941. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23708-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-23708-9_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-23707-2
Online ISBN: 978-3-642-23708-9
eBook Packages: Computer ScienceComputer Science (R0)