Abstract
This chapter presents an academic and research perspective on the impact and importance of ImageCLEF and similar evaluation workshops in multimedia information retrieval (MIR). Three main themes are examined: the position of ImageCLEF compared with other evaluation conferences; general views on the usefulness of evaluation conferences and possible alternatives, and the impact and real–world meaning of evaluation metrics used within ImageCLEF. We examine the value of ImageCLEF, and related evaluation conferences, for the multimedia IR researcher as providing not only a forum for assessing and comparing outcomes but also serving to promote research aims, provide practical guidance (e.g. standard data sets) and inspire research directions.
Keywords
- Information Retrieval
- Information Retrieval System
- Video Retrieval
- Evaluation Campaign
- Music Information Retrieval
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Borlund P (2003) The concept of relevance in IR. Journal of the American Society for information Science and Technology 54(10):913–925
Bozzon A, Brambilla M, Fraternali P, Nucci F, Debald S, Moore E, Neidl W, Plu M, Aichroth P, Pihlajamaa O, Laurier C, Zagorac S, Backfried G, Weinland D, Croce V (2009) PHAROS: an audiovisual search platform. In: ACM International conference on research and development in information retrieval. ACM press, p 841
Brookes B (1981) Information technology and the science of information. Information retrieval research. London: Butterworths 1–8
Buckley C, Voorhees E (2000) Evaluating evaluation measure stability. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 40. ACM press
Clarke C, Kolla M, Cormack G, Vechtomova O, Ashkan A, Büttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 659–666
Cleverdon C, Mills J, Keen E (1966) Factors determining the performance of indexing systems,(Volume 1: Design). Cranfield: College of Aeronautics
Cooniss L, Ashford A, Graham M (2000) Information seeking behaviour in image retrieval. VISOR 1 final report. Technical report, Library and Information Commission Research Report, British Library
Cooniss L, Davis J, Graham M (2003) A user–oriented evaluation framework for the development of electronic image retrieval systems in the workplace: VISOR 2 final report. Technical report, Library and Information Commission Research Report, British Library
Cosijn E, Ingwersen P (2000) Dimensions of relevance. Information Processing & Management 36(4):533–550
Downie JS (2008) The music information retrieval evaluation exchange (2005–2007): a window into music information retrieval research. Acoustical Science and Technology 29(4):247–255
Ellis D (1996) The dilemma of measurement in information retrieval research. Journal of the American Society for Information Science 47(1):23–36
Harman D (2005) The importance of focused evaluations: A Case Study of TREC and DUC. Charting a New Course: Natural Language Processing and Information Retrieval 16:175–194
Heesch D, Pickering M, Rüger S, Yavlinsky A (2003) Video retrieval using search and browsing with key frames. In: TREC Video Retrieval Evaluation
Heesch D, Howarth P, Magalhães J, May A, Pickering M, Yavlinsky A, Rüger S (2004) Video retrieval using search and browsing. In: TREC Video Retrieval Evaluation
Hersh W, Turpin A, Price S, Chan B, Kramer D, Sacherek L, Olson D (2000) Do batch and user evaluations give the same results? In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, p 24. ACM press
Howarth P, Yavlinsky A, Heesch D, Rüger S (2005) Medical image retrieval using texture, locality and colour. In: Proceedings of the Cross Language Evaluation Forum 2004. Lecture Notes in Computer Science (LNCS), vol 3491. Springer, pp 740–749
Järvelin K (2009) Explaining user performance in information retrieval: Challenges to IR evaluation. Advances in Information Retrieval Theory 289–296
Järvelin K, Kekäläinen J (2002) Cumulated gain–based evaluation of ir techniques. ACM Transactions on Information Systems 20(4):422–446
Jesus R, Magalhães J, Yavlinsky A, Rüger S (2005) Imperial college at trecvid. In: TREC Video Retrieval Evaluation, Gaithersburg, MD
Llorente A, Zagorac S, Little S, Hu R, Kumar A, Shaik S, Ma X, Rüger S (2008) Semantic video annotation using background knowledge and similarity–based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)
Llorente A, Little S, Rüger S (2009) MMIS at ImageCLEF 2009: Non–parametric density estimation algorithms. In: Working notes of CLEF 2009, Corfu, Greece
Magalhães J, Overell S, Yavlinsky A, Rüger S (2006) Imperial college at TRECVID. In: TREC Video Retrieval Evaluation, Gaithersburg, MD
Mandl T, Gey F, Di Nunzio G, Ferro N, Larson R, Sanderson M, Santos D, Womser-Hacker C, Xie X (2008) GeoCLEF 2007: the CLEF 2007 cross–language geographic information retrieval track overview. In: Advances in Multilingual and Multimodal Information Retrieval. Lecture Notes in Computer Science (LNCS), vol 5152. Springer, pp 745–772
Moffat A, Zobel J (2008) Rank–biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems 27(1):1–27
Nowak S, Lukashevich H, Dunker P, Rüger S (2010) Performance measures for multilabel classification — a case study in the area of image classification. In: ACM SIGMM International Conference on Multimedia Information Retrieval, Philadelphia, Pennsylvania
Over P, Smeaton A (eds) (2007) TVS 2007: proceedings of the international workshop on TRECVid video summarization. ACM press
Overell S, Magalhães J, Rüger S (2006) Place disambiguation with co–occurrence models. In: Working notes CLEF 2006, Alicante, Spain
Overell S, Llorente A, Liu HM, Hu R, Rae A, Zhu J, Song D, Rüger S (2008) MMIS at ImageCLEF 2008: Experiments combining different evidence sources. In: Working notes of CLEF 2008, Aarhus, Denmark
Pickering M, Rüger S (2002) Multi–timescale video shot–change detection. In: Text Retrieval Conf, NIST (Trec, Gaithersburg, MD, Nov 2001), NIST Special Publication 500–250, pp 275–278
van Rijsbergen CJ (1989) Towards an information logic. In: Proceedings of the 12th annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 77–86
Rüger S (2010) Multimedia information retrieval. Lecture notes in the series Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan–Claypool
Sakai T (2007) On the reliability of information retrieval metrics based on graded relevance. Information Processing & Management 43(2):531–548
Salton G (1992) The state of retrieval system evaluation. Information Processing and Management 28(4):441–449. Special Issue: Evaluation Issues in Information Retrieval
Saracevic T (1995) Evaluation of evaluation in information retrieval. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval, pp 138–146. ACM press
Saracevic T (1997) Users lost: Reflections on the past, future, and limits of information science. ACM SIGIR Forum 31(2):16–27
Smeaton A, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: ACM International Workshop on Multimedia Information Retrieval. ACM press
Smeaton A, Over P, Kraaij W (2009) High–level feature detection from video in TRECVid: a 5–year retrospective of achievements. In: Divakaran A (ed) Multimedia Content Analysis: Theory and Applications. Springer, pp 151–174
Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content–based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1349–1380
Snoek CGM, Worring M, de Rooij O, van de Sande KEA, Yan R, Hauptmann AG (2008) VideOlympics: Real–time evaluation of multimedia retrieval systems. IEEE MultiMedia 15(1):86–91
Soboroff I, Nicholas C, Cahan P (2001) Ranking retrieval systems without relevance judgments. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 66–73. ACM press
Turpin A, Hersh W (2001) Why batch and user evaluations do not give the same results. In: Proceedings of the 24th annual international ACM SIGIR conference on research and development in information retrieval, pp 225–231. ACM press
Voorhees EM (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management 36(5):697–716
Voorhees EM (2002) The philosophy of information retrieval evaluation. In: Revised Papers from the Second Workshop of the Cross–Language Evaluation Forum on Evaluation of Cross–Language Information Retrieval Systems — CLEF 2001. Lecture Notes in Computer Science (LNCS). Springer, London, UK, pp 355–370
Westerveld T, van Zwol R (2006) The INEX 2006 multimedia track. In: Comparative Evaluation of XML Information Retrieval Systems, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 4518. Springer, pp 331–344
Zagorac S, Llorente A, Little S, Liu HM, Rüger S (2009) Automated content based video retrieval. In: TREC Video Retrieval Evaluation (TRECVid, Gaithersburg, MD)
Zobel J (1998) How reliable are the results of large–scale information retrieval experiments? In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM press, pp 307–314
van Zwol R, Kazai G, Lalmas M (2005) INEX 2005 multimedia track. In: Advances in XML Information Retrieval and Evaluation, International Workshop of the Initiative for the Evaluation of XML Retrieval. Lecture Notes in Computer Science (LNCS), vol 3977. Springer, pp 497–510
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Little, S., Llorente, A., Rüger, S. (2010). An Overview of Evaluation Campaigns in Multimedia Retrieval. In: Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds) ImageCLEF. The Information Retrieval Series, vol 32. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15181-1_27
Download citation
DOI: https://doi.org/10.1007/978-3-642-15181-1_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15180-4
Online ISBN: 978-3-642-15181-1
eBook Packages: Computer ScienceComputer Science (R0)