Abstract
Semantic enrichment of metadata is an important and difficult problem for digital heritage efforts such as Europeana. This paper gives motivations and presents the work of a recently completed Task Force that addressed the topic of evaluation of semantic enrichment. We especially report on the design and the results of a comparative evaluation experiment, where we have assessed the enrichments of seven tools (or configurations thereof) on a sample benchmark dataset from Europeana.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
In fact a possible outcome of our work could be to contribute datasets and gold standards to GERBIL so as to make it a more suitable platform for future evaluations in our domain.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
See Appendix A of [12] for the complete distribution of enrichments per property.
References
Bunescu, R., Paşca, M.: Using encyclopedic knowledge for named entity disambiguation. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 9–16 (2006)
Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic enrichments with controlled vocabularies in Europeana: challenges and consequences. In: Ioannides, M., Magnenat-Thalmann, N., Fink, E., Žarnić, R., Yen, A.-Y., Quak, E. (eds.) EuroMed 2014. LNCS, vol. 8740, pp. 238–247. Springer, Heidelberg (2014)
Agirre, E., Barrena, A., Lopez de Lacalle, O., Soroa A., Fernando S., Stevenson, M.: Matching cultural heritage items to Wikipedia. In: Proceedings of LREC 2012, Istanbul, Turkey (2012)
Isaac, A., Manguinhas, H., Stiller, J., Charles, V.: Report on Enrichment and Evaluation. The Hague, Netherlands (2015). http://pro.europeana.eu/taskforce/evaluation-and-enrichments
Usbeck, R., Röder, M., Ngonga Ngomo, A., et al.: GERBIL: general entity annotator benchmarking framework. In: Proceedings of 24th WWW Conference. ACM (2015). http://doi.acm.org/10.1145/2736277.2741626
Griffiths, J., Basset, S., Goodale, P., et al.: Evaluation of the second PATHS prototype. Technical report, Paths Project (2014)
Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the importance of a semantic and multilingual enrichment strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012)
Monti, J., Monteleone, M., Buono, M., et al: Cross-lingual information retrieval and semantic interoperability for cultural heritage repositories. In: RANLP 2013. Hissar, Bulgaria (2013). http://aclweb.org/anthology/R/R13/R13-1063.pdf
Petras, V., et al.: Cultural heritage in CLEF (CHiC) 2013. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 192–211. Springer, Heidelberg (2013)
Dobreva, M., Chowdhury, S.: A user-centric evaluation of the Europeana digital library. In: Chowdhury, G., Koo, C., Hunter, J. (eds.) ICADL 2010. LNCS, vol. 6102, pp. 148–157. Springer, Heidelberg (2010)
Tordai, A., van Ossenburg, J., Schreiber, G., et al.: Let’s agree to disagree: on the evaluation of vocabulary alignment. In: Proceedings of 6th K-CAP. ACM (2011)
Isaac, A., Manguinhas, H., Charles, V., Stiller, J., et al.: Comparative evaluation of semantic enrichments. Technical report (2015). Report available at http://pro.europeana.eu/taskforce/evaluation-and-enrichments. Data archive available at: https://www.assembla.com/spaces/europeana-r-d/documents?folder=58725383
Freire, N., Borbinha, J., Calado, P., Martins, B.: A metadata geoparsing system for place name recognition and resolution in metadata records. In: ACM/IEEE Joint Conference on Digital Libraries (2011). http://dx.doi.org/10.1145/1998076.1998140
Charles, V., Freire, N., Antoine, I.: Links, languages and semantics: linked data approaches in the European Library and Europeana. In: Linked Data in Libraries: Let’s make it happen! IFLA 2014, Satellite Meeting on Linked Data in Libraries (2014)
Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley Series in Probability and Mathematical Statistics, pp. 212–236. John Wiley & Sons Ltd (1981). (Chap. 13)
Randolph, J.: Free-marginal multirater Kappa (multirater κ free): an alternative to fleiss’ fixed-marginal multirater Kappa. Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, 14–15 October 2005. http://eric.ed.gov/?id=ED490661
Manning, C., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008). http://nlp.stanford.edu/IR-book/pdf/08eval.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Manguinhas, H. et al. (2016). Exploring Comparative Evaluation of Semantic Enrichment Tools for Cultural Heritage Metadata. In: Fuhr, N., Kovács, L., Risse, T., Nejdl, W. (eds) Research and Advanced Technology for Digital Libraries. TPDL 2016. Lecture Notes in Computer Science(), vol 9819. Springer, Cham. https://doi.org/10.1007/978-3-319-43997-6_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-43997-6_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-43996-9
Online ISBN: 978-3-319-43997-6
eBook Packages: Computer ScienceComputer Science (R0)