Abstract
Many different communities have conducted research on the efficacy of relevance feedback in multimedia information systems. Unlike text IR, performance evaluation of multimedia IR systems tends to conform to the accepted standards of the community within which the work is conducted. This leads to idiosyncratic performance evaluations and hampers the ability to compare different techniques fairly. In this paper we discuss some of the shortcomings of existing multimedia IR system performance evaluations. We propose a common framework in which to discuss the differing techniques proposed for relevance feedback and we develop a strategy for fairly comparing the relative performance of the techniques.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Rui, Y., Huang, T., Mehrotra, S.: Relevance feedback techniques in interactive content-based image retrieval. In: Storage and Retrieval for Image and Video Databases (SPIE 1998), pp. 25–36 (1998)
Porkaew, K., Ortega, M., Mehrotra, S.: Query reformulation for content based multimedia retrieval in mars. In: Proc. of ICMCS 1999, San Diego, CA, USA, pp. 747–751 (1999)
Wu, L., Faloutsos, C., Sycara, K., Payne, T.: Falcon: Feedback adaptive loop for content-based retrieval. In: Proc. of VLDB 2000, Cairo, Egypt, pp. 297–306 (2000)
Kim, D., Chung, C.: Qcluster: Relevance feedback using adaptive clustering for content-based image retrieval. In: Proc. of ACM SIGMOD 2003, San Diego, CA, USA, pp. 599–610 (2003)
Ishikawa, Y., Subramanya, R., Faloutsos, C.: MindReader: Querying databases through multiple examples. In: Proc. of VLDB 1998, pp. 218–227 (1998)
Rocchio, J.: Relevance feedback in information retrieval. In: Salton, G. (ed.) The SMART Retrieval System: Experiments in Automatic Document Processing, pp. 313–323. Prentice-Hall, Englewood Cliffs (1971)
Williamson, R.: Does relevance feedback improve document retrieval performance? In: ACM SIGIR 1978, pp. 151–170 (1978)
Liu, W., Dumais, S., Sun, Y., Zhang, H., Czerwinski, M.: Semi-automatic image annotation. In: Proc. of INTERACT 2001, pp. 326–333 (2001)
Jin, X., French, J.: Improving image retrieval effectiveness via multiple queries. In: Proc. of ACM MMDB 2003, New Orleans, LA, pp. 86–93 (2003)
Müller, H., Marchand-Maillet, S., Pun, T.: The truth about corel - evaluation in image retrieval. In: Lew, M., Sebe, N., Eakins, J.P. (eds.) CIVR 2002. LNCS, vol. 2383, pp. 38–49. Springer, Heidelberg (2002)
Yan, R., Jin, R., Hauptmann, A.: Multimedia search with pseudo-relevance feedback. In: CIVR 2003 (2003)
Westerveld, T., de Vries, A.P.: Experimental result analysis for a generative probabilistic image retrieval model. In: SIGIR 2003, pp. 135–142 (2003)
Liu, W., Su, Z., Li, S., Sun, Y.F., Zhang, H.J.: Performance evaluation protocol for content-based image retrieval algorithms/systems. In: IEEE CVPR Workshop on Empirical Evaluation Methods in Computer Vision (2001)
Fagin, R.: Combining fuzzy information: an overview. ACM SIGMOD Record, 109–118 (2002)
Fox, E., Shaw, J.: Combination of multiple searches. In: Proc. of TREC2 (1994)
Belkin, N., Cool, C., Croft, W., Callan, J.: The effect of multiple query representations on information retrieval performance. In: Proc. of ACM SIGIR 2003, pp. 339–346 (1993)
Shaw, J., Fox, E.: Combination of multiple searches. In: Proc. of TREC3 (1995)
Salton, G., Fox, E., Wu, H.: Extended boolean information retrieval. Comm. of the ACM 26, 1022–1036 (1983)
Korfhage, R.: Information Storage and Retrieval. John Wiley and Sons, New York (1994)
Wang, J., Du, Y.: Scalable integrated region-based image retrieval using irm and statistical clustering. In: Proc. of JCDL 2001, Roanoke, VA (2001)
French, J., Jin, X., Martin, W.: An empirical investigation of the scalability of a multiple viewpoint cbir system. In: Enser, P.G.B., Kompatsiaris, Y., O’Connor, N.E., Smeaton, A.F., Smeulders, A.W.M. (eds.) CIVR 2004. LNCS, vol. 3115, pp. 252–260. Springer, Heidelberg (2004)
French, J., Watson, J., Jin, X., Martin, W.: Using multiple image representations to improve the quality of content-based image retrieval. In: Tech. report CS-2003-10, Dept. of Computer Science, Univ. of Virginia (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jin, X., French, J., Michel, J. (2006). Toward Consistent Evaluation of Relevance Feedback Approaches in Multimedia Retrieval. In: Detyniecki, M., Jose, J.M., Nürnberger, A., van Rijsbergen, C.J. (eds) Adaptive Multimedia Retrieval: User, Context, and Feedback. AMR 2005. Lecture Notes in Computer Science, vol 3877. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11670834_16
Download citation
DOI: https://doi.org/10.1007/11670834_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-32174-3
Online ISBN: 978-3-540-32175-0
eBook Packages: Computer ScienceComputer Science (R0)