M-SBIR: An Improved Sketch-Based Image Retrieval Method Using Visual Word Mapping

  • Jianwei NiuEmail author
  • Jun Ma
  • Jie Lu
  • Xuefeng Liu
  • Zeyu Zhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10133)


Sketch-based image retrieval (SBIR) systems, which interactively search photo collections using free-hand sketches depicting shapes, have attracted much attention recently. In most existing SBIR techniques, the color images stored in a database are first transformed into corresponding sketches. Then, features of the sketches are extracted to generate the sketch visual words for later retrieval. However, transforming color images to sketches will normally incur loss of information, thus decreasing the final performance of SBIR methods. To address this problem, we propose a new method called M-SBIR. In M-SBIR, besides sketch visual words, we also generate a set of visual words from the original color images. Then, we leverage the mapping between the two sets to identify and remove sketch visual words that cannot describe the original color images well. We demonstrate the performance of M-SBIR on a public data set. We show that depending on the number of different visual words adopted, our method can achieve \(9.8\sim 13.6\%\) performance improvement compared to the classic SBIR techniques. In addition, we show that for a database containing multiple color images of the same objects, the performance of M-SBIR can be further improved via some simple techniques like co-segmentation.


SBIR Visual word Mapping Co-segmentation M-SBIR 



This work was supported by the National Natural Science Foundation of China (Grant Nos. 61572060, 61190125, 61472024) and CERNET Innovation Project 2015 (Grant No. NGII20151004).


  1. 1.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)CrossRefGoogle Scholar
  2. 2.
    Cao, Y., Wang, H., Wang, C., Li, Z., Zhang, L., Zhang, L.: Mindfinder: interactive sketch-based image search on millions of images. In: Proceedings of the International Conference on Multimedia, pp. 1605–1608. ACM (2010)Google Scholar
  3. 3.
    Chen, T., Cheng, M.M., Tan, P., Shamir, A., Hu, S.M.: Sketch2Photo: internet image montage. ACM Trans. Graph. (TOG) 28(5), 124 (2009)Google Scholar
  4. 4.
    Choy, S.K., Tong, C.S.: Statistical wavelet subband characterization based on generalized gamma density and its application in texture retrieval. IEEE Trans. Image Process. 19(2), 281–289 (2010)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: ideas, influences, and trends of the new age. ACM Comput. Surv. (CSUR) 40(2), 5 (2008)CrossRefGoogle Scholar
  6. 6.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Photosketch: a sketch based image query and compositing system. In: SIGGRAPH: Talks, p. 60. ACM (2009)Google Scholar
  7. 7.
    Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans. Vis. Comput. Graph. 17(11), 1624–1636 (2011)CrossRefGoogle Scholar
  8. 8.
    Faktor, A., Irani, M.: Co-segmentation by composition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1297–1304 (2013)Google Scholar
  9. 9.
    Fonseca, M.J., Ferreira, A., Jorge, J.A.: Content-based retrieval of technical drawings. Int. J. Comput. Appl. Technol. 23(2–4), 86–100 (2005)CrossRefGoogle Scholar
  10. 10.
    Hu, R., Barnard, M., Collomosse, J.: Gradient field descriptor for sketch based retrieval and localization. In: 17th IEEE International Conference on Image Processing (ICIP), pp. 1025–1028. IEEE (2010)Google Scholar
  11. 11.
    Hu, R., Collomosse, J.: A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117(7), 790–806 (2013)CrossRefGoogle Scholar
  12. 12.
    Krapac, J., Verbeek, J., Jurie, F.: Modeling spatial layout with fisher vectors for image categorization. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1487–1494. IEEE (2011)Google Scholar
  13. 13.
    Leung, W.H., Chen, T.: Trademark retrieval using contour-skeleton stroke classification. In: 2002 IEEE International Conference on Multimedia and Expo, vol. 2, pp. 517–520. IEEE (2002)Google Scholar
  14. 14.
    Nowak, E., Jurie, F., Triggs, B.: Sampling strategies for bag-of-features image classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 490–503. Springer, Heidelberg (2006). doi: 10.1007/11744085_38 CrossRefGoogle Scholar
  15. 15.
    Rajendran, R., Chang, S.F.: Image retrieval with sketches and compositions. In: IEEE International Conference on Multimedia and Expo, vol. 2, pp. 717–720. IEEE (2000)Google Scholar
  16. 16.
    Rui, Y., Huang, T.S., Ortega, M., Mehrotra, S.: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circ. Syst. Video Technol. 8(5), 644–655 (1998)CrossRefGoogle Scholar
  17. 17.
    Shih, J.L., Chen, L.H.: A new system for trademark segmentation and retrieval. Image Vis. Comput. 19(13), 1011–1018 (2001)CrossRefGoogle Scholar
  18. 18.
    Sivic, J., Zisserman, A.: Video Google: a text retrieval approach to object matching in videos. In: Ninth IEEE International Conference on Computer Vision, Proceedings, pp. 1470–1477. IEEE (2003)Google Scholar
  19. 19.
    Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  20. 20.
    Wang, C., Zhang, J., Yang, B., Zhang, L.: Sketch2Cartoon: composing cartoon images by sketching. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 789–790. ACM (2011)Google Scholar
  21. 21.
    Wang, J.J.Y., Bensmail, H., Gao, X.: Joint learning and weighting of visual vocabulary for bag-of-feature based tissue classification. Pattern Recogn. 46(12), 3249–3255 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jianwei Niu
    • 1
    Email author
  • Jun Ma
    • 1
  • Jie Lu
    • 1
  • Xuefeng Liu
    • 2
  • Zeyu Zhu
    • 3
  1. 1.State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and EngineeringBeihang UniversityBeijingChina
  2. 2.Hong Kong Polytechnic UniversityHung HomHong Kong
  3. 3.School of Electronics and InformationXi’an Jiaotong UniversityXi’anChina

Personalised recommendations