Skip to main content
SpringerLink
Log in

Adaptive relevance feedback for large-scale image retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Content-based image retrieval aims at substituting traditional indexing based on manual annotation by using automatically-extracted visual indexing features. Novel techniques are needed however to efficiently deal with the semantic gap (i.e. the partial match between the low-level features and the visual content). Here, we investigate a query-free retrieval approach first proposed by Ferecatu and Geman. This approach relies solely on an iterative relevance feedback mechanism that drives a heuristic sampling of the collection, and aims to take explicitly into account the semantic gap. Our contributions are related to three complementary aspects. First, we formalize a large-scale approach based on a hierarchical tree-like organization of the images computed off-line. Second, we propose a versatile modulation of the exploration/exploitation trade-off based on the consistency of the system internal states between successive iterations. Third, we elaborate a long-term optimization of the similarity metric based on the user searching session logs accumulated off-line. We implemented a web-application that integrates all our contributions, and distribute it under the AGPL Version 3 free software license. We organized user-based evaluation campaigns using ImageNet dataset, and show empirically that our contributions significantly improve the retrieval performance of the original framework, that they are complementary to each other, and that their overall integration is consistently beneficial.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. The web-application software is available at http://www.idiap.ch/software/imr/

References

  1. Abello J, Kobourov SG, Yusufov R (2005) Visualizing large graphs with compound-fisheye views and treemaps. In: Graph Drawing, Lecture Notes in Computer Science, vol 3383. Springer, pp 431–441

  2. Bederson BB (2001) PhotoMesa: A zoomable image browser using quantum treemaps and bubblemaps. In: Proceedings of the 14th ACM symposium on User interface software and technology, pp 71–80

  3. Buchsbaum AL, Westbrook JR (2000) Maintaining hierarchical graph views. In: Proceedings of the 11th ACM-SIAM symposium on Discrete algorithms, pp 566–575

  4. Campbell I, van Rijsbergen K (1996) The ostensive model of developing information needs. In: Proceedings of the International Conference on Conceptions of Library and Information Science: Integration in Perspective (CoLIS), pp 251–268

  5. Carson C, Thomas M, Belongie S, Hellerstein JM, Malik J (1999) BlobWorld: A system for region-based image indexing and retrieval. In: Proceedings of the 3th International Conference on Visual Information Systems, vol 1614, pp 509–517

  6. Chang E, Cheng KT, Lai WC, Wu CT, Chang C, Wu YL (2001) PBIR: Perception-based image retrieval – A system that can quickly capture subjective image query concepts. In: Proceedings of the 9th ACM International Conference on Multimedia, pp 611–614

  7. Chechik G, Sharma V, Shalit U, Bengio S (2010) Large scale online learning of image similarity through ranking. J Mach Learn Res 11:1109–1135

    MathSciNet  MATH  Google Scholar 

  8. Cox IJ, Miller ML, Minka TP, Papathomas TV, Yianilos PN (2000) The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments. IEEE Trans Image Process 9(1):20–37

    Article  Google Scholar 

  9. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: Ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60

    Article  Google Scholar 

  10. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: A Large-Scale Hierarchical Image Database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 248–255

  11. Emre Celebi M, Alp Aslandogan Y (2005) Human perception-driven, similarity-based access to image databases. In: Proceedings of the Artificial Intelligence Research Society Conference, pp 245–251

  12. Fang Y, Geman D (2005) Experiments in mental face retrieval. In: Proceedings of the 5th International Conference on Audio and Video-based Biometric Person Authentication, pp 637–646

  13. Ferecatu M, Geman D (2007) Interactive search for image categories by mental matching. In: Proceedings of the 11th IEEE International Conference on Computer Vision, pp 1–8

  14. Ferecatu M, Geman D (2009) A statistical framework for image category search from a mental picture. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(6):1087–1101

    Article  Google Scholar 

  15. Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: The QBIC system. Computer 28(9):23–32

    Article  Google Scholar 

  16. Han J, Ngan K, Li ML, Zhang HJ (2005) A memory learning framework for effective image retrieval. IEEE Trans Image Process 14:511–524

    Article  Google Scholar 

  17. Heesch D (2008) A survey of browsing models for content based image retrieval. Journal of Multimedia Tools and Applications 40(2):261–284

    Article  Google Scholar 

  18. Hoi CH, Lyu MR, Jin R (2006) A unified log-based relevance feedback scheme for image retrieval. IEEE Trans Knowl Data Eng 18:509–524

    Article  Google Scholar 

  19. Ishikawa Y, Subramanya R, Faloutsos C (1998) MindReader: Querying databases through multiple examples. In: Proceedings of 24rd International Conference on Very Large Data Bases, pp 218–227

  20. Li J, Wang JZ (2008) Real-time computerized annotation of pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(6):985–1002

    Article  Google Scholar 

  21. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  22. Rui Y, Huang TS, Ortega M, Mehrotra S (1998) Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Transactions on Circuits and Video Technology 8(5):644–655

    Article  Google Scholar 

  23. Shneiderman B (1992) Tree visualization with tree-maps: 2-d space-filling approach. ACM Trans Graph 11(1):92–99

    Article  MATH  Google Scholar 

  24. Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12): 1349–1380

    Article  Google Scholar 

  25. Smith JR, Chang SF (1996) VisualSEEk: a fully automated content-based image query system. In: Proceedings of the 4th ACM international conference on Multimedia, pp 87–98

  26. Suditu N, Fleuret F (2011) HEAT: Iterative relevance feedback with one million images. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2118–2125

  27. Suditu N, Fleuret F (2012) Iterative relevance feedback with adaptive exploration/exploitation trade-off. In: Proceedings of the ACM Conference on Information and Knowledge Management (CIKM), pp 1323–1331

  28. Urban J, Jose J, Van Rijsbergen CJ (2006) An adaptive technique for content-based image retrieval. Multimedia Tools and Applications Processing (MTAP) 31:1–28

    Article  Google Scholar 

  29. Weston J, Bengio S, Usunier N (2010) Large scale image annotation: learning to rank with joint word-image embeddings. J Mach Learn 81(1):21–35

    Article  MathSciNet  Google Scholar 

  30. Zhou XS, Huang TS (2003) Relevance feedback for image retrieval: A comprehensive review. Journal of Multimedia Systems 8(6):536–544

    Article  Google Scholar 

Download references

Acknowledgments

Nicolae Suditu was supported by the Hasler Foundation through the EMMA project. François Fleuret was supported in part by the European Community’s Seventh Framework Programme FP7 - Challenge 2 - Cognitive Systems, Interaction, Robotics - under grant agreement No 247022 - MASH. The authors would like to express their thanks to Prof.Dr. Donald Geman and Dr. Marin Ferecatu for their constructive feedback and fruitful discussions that served as inspiration.

Author information

Authors and Affiliations

  1. Idiap Research Institute, Rue Marconi 19, 1920, Martigny, Switzerland

    Nicolae Suditu & François Fleuret

Authors
  1. Nicolae Suditu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. François Fleuret
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicolae Suditu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Suditu, N., Fleuret, F. Adaptive relevance feedback for large-scale image retrieval. Multimed Tools Appl 75, 6777–6807 (2016). https://doi.org/10.1007/s11042-015-2610-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2610-9

Keywords

Search

Navigation