Fast Re-ranking of Visual Search Results by Example Selection

  • John SchavemakerEmail author
  • Martijn Spitters
  • Gijs Koot
  • Maaike de Boer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9256)


In this paper we present a simple, novel method to use state-of-the-art image concept detectors and publicly available image search engines to retrieve images for semantically more complex queries from local databases without re-indexing of the database. Our low-key, data-driven method for associative recognition of unknown, or more elaborate, concepts in images allows user selection of visual examples to tailor query results to the typical preferences of the user. The method is compared with a baseline approach using ConceptNet-based semantic expansion of the query phrase to known concepts, as set by the concepts of the image concept detectors. Using the output of the image concept detector as index for all images in the local image database, a quick nearest-neighbor matching scheme is presented that can match queries swiftly via concept output vectors. We show preliminary results for a number of query phrases followed by a general discussion.


Image retrieval Concept detectors Query expansion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    de Boer, M.H.T., Daniele, L., Brandt, P., Sappelli, M.: Applying semantic reasoning in image retrieval. In: ALLDATA 2015, The First International Conference on Big Data, Small Data, Linked Data and Open Data, pp. 69–74. IARIA (2015)Google Scholar
  2. 2.
    Bouma, H., Eendebak, P.T., Schutte, K., Azzopardi, G., Burghouts, G.J.: Incremental concept learning with few training examples and hierarchical classification. In: Proc. SPIE, vol. 9652 (2015)Google Scholar
  3. 3.
    Chatfield, K., Zisserman, A.: VISOR: towards on-the-fly large-scale object category retrieval. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part II. LNCS, vol. 7725, pp. 432–446. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  4. 4.
    Chen, X., Shrivastava, A., Gupta, A.: Neil: Extracting visual knowledge from web data. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1409–1416. IEEE (2013)Google Scholar
  5. 5.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009 (2009)Google Scholar
  6. 6.
    Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning object categories from google’s image search. In: Tenth IEEE International Conference on Computer Vision, ICCV 2005, vol. 2, pp. 1816–1823. IEEE (2005)Google Scholar
  7. 7.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint 1408.5093 (2014)
  8. 8.
    Li, L.J., Fei-Fei, L.: Optimol: automatic online picture collection via incremental model learning. International Journal of Computer Vision 88(2), 147–168 (2010)CrossRefGoogle Scholar
  9. 9.
    Muja, M., Lowe, D.G.: Flann, fast library for approximate nearest neighbors (2009)Google Scholar
  10. 10.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge (2014)Google Scholar
  11. 11.
    Schutte, K., Bouma, H., Schavemaker, J., Daniele, L., Sappelli, M., Koot, G., Eendebak, P., Azzopardi, G., Spitters, M., de Boer, M., Brandt, P.: Interactive detection of incrementally learned concepts in images with ranking and semantic query interpretation. In: Proc. of 13th International Workshop on Content-Based Multimedia Indexing (CBMI) (2015)Google Scholar
  12. 12.
    Shi, Z., Yang, Y., Hospedales, T.M., Xiang, T.: Weakly supervised learning of objects, attributes and their associations. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part II. LNCS, vol. 8690, pp. 472–487. Springer, Heidelberg (2014) Google Scholar
  13. 13.
    Snoek, C.G.M., Worring, M., Koelma, D.C., Arnold, W.M., Smeulders, M.: A learned lexicon-driven paradigm for interactive video retrieval. IEEE Transactions on Multimedia 9(2) (2007)Google Scholar
  14. 14.
    Speer, R., Havasi, C.: Representing general relational knowledge in ConceptNet 5. In: LREC, pp. 3679–3686 (2012)Google Scholar
  15. 15.
    Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(11), 1958–1970 (2008)CrossRefGoogle Scholar
  16. 16.
    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. CoRR abs/1411.4555 (2014).
  17. 17.
    Wang, X.J., Zhang, L., Jing, F., Ma, W.Y.: Annosearch: Image auto-annotation by search. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1483–1490. IEEE (2006)Google Scholar
  18. 18.
    Zhang, R., Zhang, Z., Li, M., Ma, W.Y., Zhang, H.J.: A probabilistic semantic model for image annotation and multimodal image retrieval. In: Tenth IEEE International Conference on Computer Vision, ICCV 2005, vol. 1, pp. 846–851. IEEE (2005)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • John Schavemaker
    • 1
    Email author
  • Martijn Spitters
    • 1
  • Gijs Koot
    • 1
  • Maaike de Boer
    • 1
    • 2
  1. 1.TNO Technical SciencesThe HagueThe Netherlands
  2. 2.Radboud UniversityNijmegenThe Netherlands

Personalised recommendations