Tag-Based Social Image Search: Toward Relevant and Diverse Results

  • Kuiyuan Yang
  • Meng Wang
  • Xian-Sheng Hua
  • Hong-Jiang Zhang


Recent years have witnessed a great success of social media websites. Tag-based image search is an important approach to access the image content of interest on these websites. However, the existing ranking methods for tag-based image search frequently return results that are irrelevant or lack of diversity. This chapter presents a diverse relevance ranking scheme which simultaneously takes relevance and diversity into account by exploring the content of images and their associated tags. First, it estimates the relevance scores of images with respect to the query term based on both visual information of images and semantic information of associated tags. Then semantic similarities of social images are estimated based on their tags. Based on the relevance scores and the similarities, the ranking list is generated by a greedy ordering algorithm which optimizes Average Diverse Precision (ADP), a novel measure that is extended from the conventional Average Precision (AP). Comprehensive experiments and user studies demonstrate the effectiveness of the approach.


Semantic Similarity Average Precision Visual Similarity Image Search Ranking List 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Agrawal, R., Gollapudi, S., Halverson, A., Leong, S.: Diversifying search results. In: Proceedings of ACM International Conference on Web Search and Data Mining (2009) Google Scholar
  2. 2.
    Buckley, C., Voorhees, E.M.: Retrieval evaluation with incomplete information. In: SIGIR (2004) Google Scholar
  3. 3.
    Cai, D., He, X., Li, Z., Ma, W.-Y., Wen, J.-R.: Hierarchical clustering of WWW image search results using visual, textual and link information. In: Proceedings of ACM Multimedia, pp. 952–959 (2004) Google Scholar
  4. 4.
    Carbonell, J., Goldstein, J.: The use of MMR, diversity-based reranking for reordering documents and producing summaries. In: Proceedings of SIGIR, pp. 335–336 (1998) Google Scholar
  5. 5.
    Chen, H., Karger, D.R.: Less is more: probabilistic models for retrieving fewer relevant documents. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 436. ACM, New York (2006) Google Scholar
  6. 6.
    Cilibrasi, R., Vitanyi, P.M.B.: The google similarity distance. IEEE Trans. Knowl. Data Eng. 19, 370–383 (2007) CrossRefGoogle Scholar
  7. 7.
    Clarke, C.L.A., Kolla, M., Cormack, G.V., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I.: Novelty and diversity in information retrieval evaluation. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 659–666. ACM, New York (2008) CrossRefGoogle Scholar
  8. 8.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007) CrossRefMathSciNetGoogle Scholar
  9. 9.
    Goffman, W.: A searching procedure for information retrieval. In: Information Storage and Retrieval, vol. 2, pp. 73–78 (1964) Google Scholar
  10. 10.
    Hsu, W.H., Kennedy, L.S., Chang, S.-F.: Video search reranking via information bottleneck principle. In: Proceedings of ACM Multimedia, pp. 35–44 (2006) Google Scholar
  11. 11.
    Jaimes, A., Chang, S.-F., Loui, A.C.: Detection of non-identical duplicate consumer photographs. In: Proceedings of ACM Multimedia, pp. 16–20 (2003) Google Scholar
  12. 12.
    Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20(4), 446 (2002) CrossRefGoogle Scholar
  13. 13.
    Jing, F., Wang, C., Yao, Y., Deng, K., Zhang, L., Ma, W.-Y.: IGroup: web image search results clustering. In: Proceedings of ACM Multimedia, pp. 587–596 (2006) Google Scholar
  14. 14.
    Kennedy, L.S., Chang, S.F., Kozintsev, I.V.: To search or to label? predicting the performance of search-based automatic image classifiers. In: Proceedings of MIR, pp. 249–258 (2006) Google Scholar
  15. 15.
    Kennedy, L., Slaney, M., Weinberger, K.: Reliable tags using image similarity: mining specificity and expertise from large-scale multimedia databases. In: WSMC ’09: Proceedings of the 1st Workshop on Web-scale Multimedia Corpus, pp. 17–24. ACM, New York (2009) CrossRefGoogle Scholar
  16. 16.
    King, B.M., Minium, E.W.: Statistical reasoning in psychology and education. Wiley, New York (2003) Google Scholar
  17. 17.
    Li, J., Wang, J.: Real-time computerized annotation of pictures. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 985–1002 (2008) CrossRefGoogle Scholar
  18. 18.
    Li, X.R., Snoek, C.G.M., Worring, M.: Learning tag relevance by neighbor voting for social image retrieval. In: Proceedings of MIR, pp. 180–187 (2008) CrossRefGoogle Scholar
  19. 19.
    Liu, D., Wang, M., Yang, L., Hua, X.-S., Zhang, H.-J.: Tag quality improvement for social images. In: Proceedings of ICME, pp. 350–353 (2009) Google Scholar
  20. 20.
    Nah, F.F.-H.: A study on tolerable waiting time: how long are web users willing to wait. Behav. Inf. Technol. 23(3), 153–163 (2004) CrossRefGoogle Scholar
  21. 21.
    Qi, G.J., Hua, X.S., Rui, Y., Tang, J.H., Zha, Z.J., Zhang, H.J.: A joint appearance-spatial distance for kernel-based image categorization. In: Proceedings of CVPR, pp. 1–8 (2008) Google Scholar
  22. 22.
    Robertson, S.: The probability ranking principle in IR. J. Doc. 33(294), 294–304 (1977) CrossRefGoogle Scholar
  23. 23.
    Rui, Y., Huang, T.S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circuits Syst. Video Technol. 8(5), 644–655 (1999) Google Scholar
  24. 24.
    Santos, R.L.T., Macdonald, C., Ounis, I.: Exploiting query reformulations for Web search result diversification. In: Proceedings of the 19th International Conference on World Wide Web, pp. 881–890. ACM, New York (2010) CrossRefGoogle Scholar
  25. 25.
    Song, K., Tian, Y., Huang, T., Gao, W.: Diversifying the image retrieval results. In: Proceedings of ACM Multimedia, pp. 707–710 (2006) Google Scholar
  26. 26.
    Srinivasan, S.H., Sawant, N.: Finding near-duplicate images on the web using fingerprints. In: Proceedings of ACM Multimedia, pp. 881–884 (2008) Google Scholar
  27. 27.
    Sun, A., Bhowmick, S.S.: Image tag clarity: in search of visual-representative tags for social images. In: WSM ’09: Proceedings of the First SIGMM Workshop on Social Media, pp. 19–26. ACM, New York (2009) CrossRefGoogle Scholar
  28. 28.
    Van Leuken, R.H., Garcia, L., Olivares, X., Zwol, R.: Visual diversification of image search results. In: Proceedings of WWW, pp. 341–350 (2009) CrossRefGoogle Scholar
  29. 29.
    Wang, B., Li, Z., Li, M., Ma, W.-Y.: Large-scale duplicate detection for web image search. In: Proceedings of ICME, pp. 353–356 (2006) Google Scholar
  30. 30.
    Wang, M., Hua, X.-S., Tang, J., Hong, R.: Beyond distance measurement: constructing neighborhood similarity for video annotation. IEEE Trans. Multimed. 11(3), 465–476 (2009) CrossRefGoogle Scholar
  31. 31.
    Weinberger, K.Q., Slaney, M., Van Zwol, R.: Resolving tag ambiguity. In: MM ’08: Proceeding of the 16th ACM International Conference on Multimedia, pp. 111–120. ACM, New York (2008) CrossRefGoogle Scholar
  32. 32.
    Wu, L., Hua, X.-S., Ma, W.-Y., Yu, N., Li, S.: Flickr distance. In: Proceedings of ACM Multimedia, pp. 31–40 (2008) Google Scholar
  33. 33.
    Yahoo key scientific challenges program. http://research.yahoo.com/ksc/multimedia
  34. 34.
    Yang, K., Wang, M., Hua, X.-S., Zhang, H.-J.: Social image search with diverse relevance ranking. In: International MultiMedia Modeling Conference (MMM) (2010) Google Scholar
  35. 35.
    Zhai, C., Cohen, W.W., Lafferty, J.: Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Information Processing and Management, pp. 10–17 (2006) Google Scholar
  36. 36.
    Zhai, C., Lafferty, J.: A risk minimization framework for information retrieval. Inf. Process. Manag. 31–55 (2006) Google Scholar
  37. 37.
    Zhao, W.L., Ngo, C.W.: Scale-rotation invariant pattern entropy for keypoint-based near-duplicate detection. IEEE Trans. Image Process. 18(2), 412–423 (2009) CrossRefMathSciNetGoogle Scholar
  38. 38.
    Zhu, J., Hoi, S.C.H., Lyu, M.R., Yan, S.: Near-duplicate keyframe retrieval by nonrigid image matching. In: Proceedings of ACM Multimedia, pp. 41–50 (2008) Google Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Kuiyuan Yang
    • 1
  • Meng Wang
    • 2
  • Xian-Sheng Hua
    • 3
  • Hong-Jiang Zhang
    • 4
  1. 1.Department of AutomationThe University of Science and Technology of ChinaHefeiChina
  2. 2.AKiiRA Media Systems IncPalo AltoUSA
  3. 3.Media Computing GroupMicrosoft Research AsiaBeijingChina
  4. 4.Microsoft Advanced Technology CenterBeijingChina

Personalised recommendations