Abstract
This paper examines results from the last two years of the TRECVID video retrieval evaluations. While there is encouraging evidence about progress in video retrieval, there are several major disappointments confirming that the field of video retrieval is still in its infancy. Many publications blithely attribute improvements in retrieval tasks to the different techniques without paying much attention to the statistical reliability of the comparisons. We conduct an analysis of the official TRECVID evaluation results, using both retrieval experiment error rates and ANOVA measures, and demonstrate that the difference between many systems is not statistically significant. We conclude the paper with the lessons learned from both results with and without statistically significant difference.
This work was supported in part by the Advanced Research and Development Activity (ARDA) under contract number H98230-04-C-0406 and NBCHC040037.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ianeva, T., Boldareva, L., Westerveld, T., Cornacchia, R., Hiemstras, D., de Vries, A.P.: Probabilistic approaches to video retrieval. [6]
Chua, T.S., Neo, S.Y., Li, K.Y., Wang, G., Shi, R., Zhao, M., Xu, H.: TRECVID 2004 search and feature extraction task by NUS PRIS. [6] (2004)
Amir, A., Argillander, J.O., Berg, M., Chang, S.F., Hsu, W., Iyengar, G., Kender, J.R., Lin, C.Y., Naphade, M., Natsev1, A.P., Smith, J.R., Tesic, J., Wu, G., Yan, R., Zhang, D.: IBM research TRECVID-2004 video retrieval system. [6] (2004)
Yan, R., Yang, J., Hauptmann, A.G.: Learning query-class dependent weights in automatic video retrieval. In: Proceedings of the Twelfth ACM International Conference on Multimedia, pp. 548–555 (2004)
Hauptmann, A., Chen, M.Y., Christel, M., Huang, C., Lin, W.H., Ng, T., Papernick, N., Velivelli, A., Yang, J., Yan, R., Yang, H., Wactlar, H.D.: Confounded expectations: Informedia at TRECVID 2004. [6] (2004)
Proceedings of the TREC Video Retrieval Evaluation 2004. In: Proceedings of the TREC Video Retrieval Evaluation 2004 (2004)
NIST: Guidelines for the TRECVID 2004 evaluation. Webpage (2004), http://www-nlpir.nist.gov/projects/tv2004/tv2004.html
Voorhees, E.M., Buckley, C.: The effect of topic set size on retrieval experiment error. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 316–323. ACM Press, New York (2002)
Lin, W.H., Hauptmann, A.: Revisiting the effect of topic set size on retrieval error. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (2005)
Myers, J.L.: Fundamentals of Experimental Design. Allyn and Bacon, Boston (1972)
Braschler, M.: CLEF 2001 - Overview of Results. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 9–26. Springer, Heidelberg (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hauptmann, A., Lin, WH. (2005). Assessing Effectiveness in Video Retrieval. In: Leow, WK., Lew, M.S., Chua, TS., Ma, WY., Chaisorn, L., Bakker, E.M. (eds) Image and Video Retrieval. CIVR 2005. Lecture Notes in Computer Science, vol 3568. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11526346_25
Download citation
DOI: https://doi.org/10.1007/11526346_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-27858-0
Online ISBN: 978-3-540-31678-7
eBook Packages: Computer ScienceComputer Science (R0)