Overview of the ImageCLEFphoto 2008 Photographic Retrieval Task

  • Thomas Arni
  • Paul Clough
  • Mark Sanderson
  • Michael Grubinger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5706)


ImageCLEFphoto 2008 is an ad-hoc photo retrieval task and part of the ImageCLEF evaluation campaign. This task provides both the resources and the framework necessary to perform comparative laboratory-style evaluation of visual information retrieval systems. In 2008, the evaluation task concentrated on promoting diversity within the top 20 results from a multilingual image collection. This new challenge attracted a record number of submissions: a total of 24 participating groups submitting 1,042 system runs. Some of the findings include that the choice of annotation language is almost negligible and the best runs are by combining concept and content-based retrieval methods.


Performance Evaluation IAPR TC-12 Benchmark Image Retrieval Diversity Clustering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Voorhees, E.M., Harman, D.: Overview of the Seventh Text REtrieval Conference (TREC–7). In: The Seventh Text Retrieval Conference, Gaithersburg, MD, USA, November 1998, pp. 1–23 (1998)Google Scholar
  2. 2.
    Tian, S.K., Gao, Y., Huang, T.: Diversifying the image retrieval results. In: Proceedings of the 14th Annual ACM international Conference on Multimedia, MULTIMEDIA 2006, Santa Barbara, CA, USA, October 23-27, pp. 707–710. ACM, New York (2006)Google Scholar
  3. 3.
    Chen, H., Karger, D.R.: Less is more: probabilistic models for retrieving fewer relevant documents. In: Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information Retrieval, SIGIR 2006, Seattle, Washington, USA, August 6-11, pp. 429–436. ACM, New York (2006)Google Scholar
  4. 4.
    Sanderson, M., Tang, J., Arni, T., Clough, P.: What else is there? Search Diversity Examined. In: Boughanem, M., et al. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 562–569. Springer, Heidelberg (2009)Google Scholar
  5. 5.
    Clough, P., Grubinger, M., Deselaers, T., Hanbury, A., Müller, H.: Overview of the ImageCLEF 2006 photographic retrieval and object annotation tasks. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 579–594. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Grubinger, M., Clough, P., Hanbury, A., Müller, H.: Overview of the ImageCLEFPhoto 2007 photographic retrieval task. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 433–444. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  7. 7.
    Grubinger, M., Clough, P., Müller, H., Deselears, T.: The IAPR–TC12 Benchmark: A New Evaluation Resource for Visual Information Systems. In: International Workshop OntoImage 2006 Language Resources for Content-Based Image Retrieval, held in conjunction with LREC 2006, Genoa, Italy, May 22nd, pp. 13–23 (2006)Google Scholar
  8. 8.
    Grubinger, M., Clough, P.: On the Creation of Query Topics for ImageCLEFPhoto. In: Proceedings of the third MUSCLE / ImageCLEF workshop on image and video retrieval evaluation, Budapest, Hungary, September 19-21 (2007)Google Scholar
  9. 9.
    Arni, T., Tang, J., Sanderson, M., Clough, P.: Creating a test collection to evaluate diversity in image retrieval. In: Proceedings of the Workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments, held at SIGIR 2008 (2008)Google Scholar
  10. 10.
    Zhai, C., Cohen, W.W., Lafferty, J.: Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In: Proceedings of ACM SIGIR 2003, pp. 10–17 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Thomas Arni
    • 1
  • Paul Clough
    • 1
  • Mark Sanderson
    • 1
  • Michael Grubinger
    • 2
  1. 1.Department of Information StudiesUniversity of SheffieldUK
  2. 2.Victoria UniversityMelbourneAustralia

Personalised recommendations