Abstract
This study addresses the lack of an adequate test collection that can be used to evaluate search systems that exploit annotations to increase the retrieval effectiveness of an information search tool. In particular, a new approach is proposed that enables the automatic creation of multiple test collections without human effort. This approach takes advantage of the human relevance assessments contained in an already existing test collection and it introduces content-level annotations in that collection.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Ioannidis, Y., Maier, D., Abiteboul, S., Buneman, P., Davidson, S., Fox, E.A., Halevy, A., Knoblock, C., Rabitti, F., Schek, H.J., Weikum, G.: Digital library information-technology infrastructures. International Journal on Digital Libraries 5, 266–274 (2005)
Agosti, M., Ferro, N.: Annotations as Context for Searching Documents. In: Crestani, F., Ruthven, I. (eds.) CoLIS 2005. LNCS, vol. 3507, pp. 155–170. Springer, Heidelberg (2005)
Frommholz, I.: Annotation-based document retrieval with probabilistic logics. In: Fuhr, N., Kovacs, L., Meghini, C. (eds.) ECDL 2007. Proc. 11th European Conference on Research and Advanced Technology for Digital Libraries. LNCS, vol. 4675, pp. 321–332. Springer, Heidelberg (2007)
Voorhees, E.M., Harman, D.K. (eds.): TREC: Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)
Coppotelli, T.: Creazione di una collezione sperimentale per la valutazione di sistemi di reperimento dell’informazione che utilizzino le annotazioni (in Italian). Master’s thesis, Department of Information Engineering, University of Padua (2006)
Agosti, M., Coppotelli, T., Ferro, N., Pretto, L.: Exploiting relevance assessments for the creation of an experimental test collection to evaluate systems that use annotations. In: DELOS Conference, Pisa, Italy, pp. 195–202 (2007)
Agosti, M., Coppotelli, T., Ferro, N., Pretto, L.: Annotations and digital libraries: Designing adequate test-beds. In: Goh, D.H., Cao, T., Sølvberg, I., Rasmussen, E. (eds.) ICADL 2007. Proc. 10th International Conference on Asian Digital Libraries. LNCS, Springer, Heidelberg (in print, 2007)
Ferro, N.: Digital annotations: a formal model and its applications. In: Agosti, M. (ed.) Information Access through Search Engines and Digital Libraries, pp. 113–146. Springer, Heidelberg (in print, 2008)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Agosti, M., Coppotelli, T., Ferro, N., Pretto, L. (2007). An Approach for the Construction of an Experimental Test Collection to Evaluate Search Systems that Exploit Annotations. In: Thanos, C., Borri, F., Candela, L. (eds) Digital Libraries: Research and Development. DELOS 2007. Lecture Notes in Computer Science, vol 4877. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-77088-6_16
Download citation
DOI: https://doi.org/10.1007/978-3-540-77088-6_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-77087-9
Online ISBN: 978-3-540-77088-6
eBook Packages: Computer ScienceComputer Science (R0)