THESEUS Meets ImageCLEF: Combining Evaluation Strategies for a New Visual Concept Detection Task 2009

  • Stefanie Nowak
  • Peter Dunker
  • Ronny Paduschek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5706)


Automatic methods for archiving, indexing and retrieving multimedia content become more and more important through the steadily increasing amount of digital data in the web and at home. THESEUS, a German research program, focuses on developing sophisticated algorithms and evaluation strategies for the automated processing of digital data. In this paper we present how evaluation is performed in THESEUS and introduce a generic framework for the evaluation of various video and image analysis algorithms. Besides, evaluation campaigns like the Cross Evaluation Language Forum (CLEF) and subprojects like ImageCLEF deal with the evaluation of such algorithms and provide an objective comparison of their performance. We relate the THESEUS tasks to the work done in ImageCLEF and propose a new task for ImageCLEF 2009.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Müller, H., Geissbuhler, A., Marchand-Maillet, S., Clough, P.: Benchmarking image retrieval applications. In: Proc. of the 10th Intern. Conf. Distributed Multimedia Systems, Workshop on Visual Information Systems, San Francisco (2004)Google Scholar
  2. 2.
    Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: MIR 2006: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321–330. ACM Press, New York (2006)Google Scholar
  3. 3.
    Müller, H., Marchand-Maillet, S., Pun, T.: The Truth about Corel-Evaluation in Image Retrieval. In: Lew, M., Sebe, N., Eakins, J.P. (eds.) CIVR 2002. LNCS, vol. 2383, pp. 38–49. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  4. 4.
    Datta, R., Joshi, D., Li, J., Wang, J., Surveys, A.: Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM Computing Surveys 40(2) (2008)Google Scholar
  5. 5.
    Huiskes, M.J., Lew, M.S.: The MIR Flickr Retrieval Evaluation. In: MIR 2008: Proceedings of the 2008 ACM International Conference on Multimedia Information Retrieval. ACM, New York (2008)Google Scholar
  6. 6.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In: Proc. 8th Int. Conf. Computer Vision, vol. 2, pp. 416–423 (2001)Google Scholar
  7. 7.
    Huang, Q., Dom, B.: Quantitative methods of evaluating image segmentation. In: IEEE International Conference on Image Processing, vol. 3, pp. 53–56 (1995)Google Scholar
  8. 8.
    Deselaers, T., Hanbury, A.: The Visual Concept Detection Task in ImageCLEF 2008. In: Peters, C., et al. (eds.) CLEF 2008. LNCS, vol. 5706, pp. 531–538. Springer, Heidelberg (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Stefanie Nowak
    • 1
  • Peter Dunker
    • 1
  • Ronny Paduschek
    • 1
  1. 1.Fraunhofer Institute for Digital Media Technology IDMTIlmenauGermany

Personalised recommendations