Multimodal-Based Supervised Learning for Image Search Reranking

  • Shengnan Zhao
  • Jun MaEmail author
  • Chaoran Cui
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9098)


The aim of image search reranking is to rerank the images obtained by a conventional text-based image search engine to improve the search precision, diversity and so on. Current image reranking methods are often based on a single modality. However, it is hard to find a general modality which can work well for all kinds of queries. This paper proposes a multimodal-based supervised learning for image search reranking. First, for different modalities, different similarity graphs are constructed and different approaches are utilized to calculate the similarity between images on the graph. Exploiting the similarity graphs and the initial list, we integrate the multiple modality into query-independent reranking features, namely PageRank Pseudo Relevance Feedback, Density Feature, Initial Ranking Score Feature, and then fuse them into a 19-dimensional feature vector for each image. After that, the supervised method is employed to learn the weight of each reranking feature. The experiments constructed on the MSRA-MM Dataset demonstrate the improvement in robust and effectiveness of the proposed method.


Image search reranking Supervised reranking Multimodal learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tian, X., Lu, Y., Yang, L., Tian, Q.: Learning to judge image search results. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 363–372. ACM (2011)Google Scholar
  2. 2.
    Snoek, C.G.M., Worring, M., Smeulders, A.W.M.: Early versus late fusion in semantic video analysis. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 399–402. ACM (2005)Google Scholar
  3. 3.
    Wang, M., Yang, L., Hua, X.-S.: Msra-mm: Bridging research and industrial societies for multimedia information retrieval, Microsoft Research Asia. Tech. Rep. (2009)Google Scholar
  4. 4.
    Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 133–142. ACM (2002)Google Scholar
  5. 5.
    Yang, L., Hanjalic, A.: Supervised reranking for web image search. In: Proceedings of the International Conference on Multimedia, pp. 183–192. ACM (2010)Google Scholar
  6. 6.
    Yang, Y., Yang, L., Wu, G., Li, S.: A bag-of-objects retrieval model for web image search. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 49–58. ACM (2012)Google Scholar
  7. 7.
    Jing, Y., Baluja, S.: Visualrank: Applying pagerank to large-scale image search. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(11), 1877–1890 (2008)CrossRefGoogle Scholar
  8. 8.
    Tian, X., Yang, Y., Wang, J., Xiuqing, W., Hua, X.-S.: Bayesian visual reranking. IEEE Transactions on Multimedia 13(4), 639–652 (2011)CrossRefGoogle Scholar
  9. 9.
    Platt, J.C.: Fast training of support vector machines using sequential minimal optimization. In: Advances in Kernel Methods, pp. 185–208. MIT Press (1999)Google Scholar
  10. 10.
    Joachims, T.: Training linear svms in linear time. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 217–226. ACM (2006)Google Scholar
  11. 11.
  12. 12.
    Scott, D.W.: Multivariate density estimation: theory, practice, and visualization, vol. 383. John Wiley & Sons (2009)Google Scholar
  13. 13.
    Yan, R., Hauptmann, A., Jin, R.: Multimedia search with pseudo-relevance feedback. In: Bakker, E.M., Lew, M.S., Huang, T.S., Sebe, N., Zhou, X.S. (eds.) CIVR 2003. LNCS, vol. 2728, pp. 238–247. Springer, Heidelberg (2003)Google Scholar
  14. 14.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2007, pp. 1–8. IEEE (2007)Google Scholar
  15. 15.
    Wang, M., Li, H., Tao, D., Ke, L., Xindong, W.: Multimodal graph-based reranking for web image search. IEEE Transactions on Image Processing 21(11), 4649–4661 (2012)MathSciNetCrossRefGoogle Scholar
  16. 16.
  17. 17.
    Lee, S.M., Xin, J.H., Westland, S.: Evaluation of image similarity by histogram intersection. Color Research and Application 30(4), 265–274 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.School of Computer Science and TechnologyShandong UniversityJinanChina

Personalised recommendations