A Framework for the Evaluation of Automatic Metadata Enrichments

  • Juliane Stiller
  • Marlies Olensky
  • Vivien Petras
Part of the Communications in Computer and Information Science book series (CCIS, volume 478)


Automatic enrichment of collections connects data to vocabularies, which supports the contextualization of content and adds searchable text to metadata. The paper introduces a framework of four dimensions (frequency, coverage, relevance and error rate) that measure both the suitability of the enrichment for the object and the enrichments’ contribution to search success. To verify the framework, it is applied to the evaluation of automatic enrichments in the digital library Europeana. The analysis of 100 result sets and their corresponding queries (1,121 documents total) shows the framework is a valuable tool for guiding enrichments and determining the value of enrichment efforts.


Evaluation Europeana Digital Libraries Cultural Collections Framework Semantic Enrichment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Newman, D., Hagedorn, K., Chemudugunta, C., Smyth, P.: Subject Metadata Enrichment Using Statistical Topic Models. In: Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 366–375. ACM, New York (2007)Google Scholar
  2. 2.
    Agirre, E., Barrena, A., Fernandez, K., Miranda, E., Otegi, A., Soroa, A.: PATHSenrich: A Web Service Prototype for Automatic Cultural Heritage Item Enrichment. In: Aalberg, T., Papatheodorou, C., Dobreva, M., Tsakonas, G., Farrugia, C.J. (eds.) TPDL 2013. LNCS, vol. 8092, pp. 462–465. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  3. 3.
    Olensky, M., Stiller, J., Dröge, E.: Poisonous India or the Importance of a Semantic and Multilingual Enrichment Strategy. In: Dodero, J.M., Palomo-Duarte, M., Karampiperis, P. (eds.) MTSR 2012. CCIS, vol. 343, pp. 252–263. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  4. 4.
    Schreiber, G., Amin, A., Aroyo, L., van Assem, M., de Boer, V., Hardman, L., et al.: Semantic annotation and search of cultural-heritage collections: The MultimediaN E-Culture demonstrator. Web Semantics: Science, Services and Agents on the World Wide Web 6(4), 243–249 (2008)CrossRefGoogle Scholar
  5. 5.
    Jones, C.B., Purves, R., Ruas, A., Sanderson, M., Sester, M., van Kreveld, M., Weibel, R.: Spatial Information Retrieval and Geographical Ontologies an Overview of the SPIRIT Project. In: SIGIR 2002, Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 387–388. ACM, New York (2002)Google Scholar
  6. 6.
    Fernie, K., Griffiths, J., Davies, R.: D 8.1 Final Report. PATHS - Personalised Access to Cultural Heritage Spaces (2014),
  7. 7.
    Agirre, E., Barrena, A., Lopez de Lacalla, O., Soroa, A., Stevenson, M., Fernando, S.: Matching Cultural Heritage items to Wikipedia. In: Proceedings of the 8th International Conference on Language Resources and Evaluation, Istanbul, Turkey, pp. 1729–1735 (2012)Google Scholar
  8. 8.
    Shreeves, S.L., Knutson, E.M., Stvilia, B., Palmer, C.L., Twidale, M.B., Cole, T.W.: Is ‘Quality’ Metadata ‘Shareable’ Metadata? The Implications of Local Metadata Practices for Federated Collections. In: Thompson, H.A. (ed.) Proceedings of the Twelfth National Conference of the Association of College and Research Libraries, April 7-10, pp. 223–237. Association of College and Research Libraries, Minneapolis (2005)Google Scholar
  9. 9.
    Van Hage, W.R., Isaac, A., Aleksovski, Z.: Sample Evaluation of Ontology-Matching Systems. In: EON, vol. 2007, pp. 41–50 (November 2007)Google Scholar
  10. 10.
    Stiller, J., Petras, V., Gäde, M., Isaac, A.: Automatic Enrichments with Controlled Vocabularies in Europeana: Challenges and Consequences (to be published)Google Scholar
  11. 11.
    Stiller, J., Olensky, M., Isaac, A., Petras, V.: D7.8. Final Report on Innovative Multilingual Information Access. Europeana Version 2 (2014),

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Juliane Stiller
    • 1
  • Marlies Olensky
    • 1
  • Vivien Petras
    • 1
  1. 1.Berlin School of Library and Information ScienceHumboldt-Universität zu BerlinBerlinGermany

Personalised recommendations