Skip to main content
SpringerLink
Log in

Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sketch-based Image Retrieval (SBIR) is one important branch of Content-based Image Retrieval (CBIR). SBIR means dealing with retrieval using simple edge or contour images. However, SBIR is more difficult than CBIR due to the lack of visual information, this makes the Bag-of-Words (BoW) or codebook in SBIR hard to construct. In this paper, we propose a novel SBIR framework based on Product Quantization (PQ) with sparse coding (SC) to construct an optimized codebook. By using state-of-the-art local descriptors, we transform sketch images into features and then build the optimized codebook using PQ-based SC. In the retrieval stage, we can obtain a better representation of the query sketch and testing images by the optimized codebook with coding quantization residuals, by which the information loss during feature encoding process can be reduced; similarity computing is implemented by comparing the feature histograms between a query sketch and the testing data for the final results. We demonstrate the superiority and effectiveness of the proposed SBIR by comparing it with several state-of-the-art methods on three public sketch datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://cybertron.cg.tu-berlin.de/eitz/tvcg_benchmark/index.html

  2. http://personal.ee.surrey.ac.uk/Personal/R.Hu/SBIR.html

  3. http://groups.inf.ed.ac.uk/calvin/datasets.html

References

  1. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522

    Article  Google Scholar 

  2. Cao Y, Wang H, Wang C, Li Z, Zhang L (2011) Edgel inverted index for large-scale sketch-based image search. In CVPR, pp 761–768

  3. Cao X, Zhang H, Liu S, Guo X, Lin L (2013) SYM-FISH: A Symmetry-Aware Flip Invariant Sketch Histogram Shape Descriptor. In ICCV, pp 313–320

  4. Chalechale A, Naghdy G, Mertins A (2005) Sketch-based image matching using angular partitioning. IEEE Trans Syst Man Cybern Part A 35(1):28–41

    Article  Google Scholar 

  5. Chang N, Fu K (1979) Query-by-pictorial-example. In COMPSAC, pp 325–330

  6. Cover T, Thomas J (1991) Elements of information theory. John Wiley & sons, Inc.

  7. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2011) Sketch-based image retrieval: benchmark and bag-of-features descriptors. IEEE Trans Vis Comput Graph 17(11):1624–1636

    Article  Google Scholar 

  8. Eitz M, Hildebrand K, Boubekeur T, Alexa M (2012) Sketch-based shape retrieval. ACM Trans Graph 4(31):1–10

    Google Scholar 

  9. Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  10. Ge T, He K, Ke Q, Sun J (2014) Optimized product quantization. IEEE Trans Pattern Anal Mach Intell 36(4):744–755

    Article  Google Scholar 

  11. Ge T, He K, Sun J (2014) Product Sparse Coding. In CVPR, pp 939–946

  12. Han Y, Yang Y, Ma Z, Shen H, Sebe N, Zhou X (2014) Image attribute adaptation. IEEE Trans Multimedia 16(4):1115–1126

    Article  Google Scholar 

  13. Han Y, Yang Y, Yan Y, Ma Z, Sebe N, Zhou X (2015) Semi-supervised feature selection via spline regression for video semantic recognition. IEEE Trans Neural Netw Learn Syst 26(2):252–264

    Article  Google Scholar 

  14. Hu R, Collomosse J (2013) A performance evaluation of gradient field HOG descriptor for sketch based image retrieval. Comput Vis Image Underst 117(7):790–806

    Article  Google Scholar 

  15. Hurtut T, Gousseau Y, Schmitt F, Cheriet F (2008) Pictorial analysis of line-drawings. In CAe, pp 123–130

  16. Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128

    Article  Google Scholar 

  17. Kalantidis Y, Avrithis Y (2014) Locally Optimized Product Quantization for Approximate Nearest Neighbor Search. In CVPR, pp 2321–2328

  18. Liu W, Wang J, Ji R, Jiang Y, Chang, S (2012) Supervised Hashing with Kernels. In CVPR, pp 2074–2081

  19. Martínez JM (2002) MPEG-7: overview of MPEG-7 description tools, part 2. IEEE Multimedia 9(3):83–93

    Article  Google Scholar 

  20. Pauleve L, Jegou H, Amsaleg L (2010) Locality sensitive hashing: a comparison of hash function types and querying mechanisms. Pattern Recogn Lett 31(11):1348–1358

    Article  Google Scholar 

  21. Rui Y, Huang TS, Chang SF (1999) Image retrieval: current techniques, promising directions, and open issues. J Vis Commun Image Represent 10(1):39–62

    Article  Google Scholar 

  22. Saavedra JM, Bustos B (2013) Sketch-based image retrieval using keyshapes. Multimed Tools Appl: 1–30

  23. Smeulders WM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  24. Wang X, Bai X, Liu W., Latecki L (2011) Feature context for image classification and object detection, In CVPR, pp 961–968

  25. Wang J, Bai X, You X, Liu W, Latecki LJ (2012) Shape matching and classification using height functions. Pattern Recogn Lett 33(2):134–143

    Article  Google Scholar 

  26. Witten A, Moffat A, Bell T (1999) Managing gigabytes: compressing and indexing documents and images. Morgan Kaufmann

  27. Won C, Park D, Park S (2002) Efficient use of mpeg-7 edge histogram descriptor. Etri J 24(1):23–30

    Article  Google Scholar 

  28. Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(5):723–742

    Article  Google Scholar 

  29. Yang Y, Yang Y, Shen H (2013) Effective transfer tagging from image to video. ACM Trans Multimed Comput Commun Appl 9(2):3

    Article  Google Scholar 

  30. Yang J, Yu K, Gong Y Huang T (2009) Linear spatial pyramid matching using sparse coding for image classification. In CVPR, pp 1794–1801

  31. Yang Y, Zha Z, Gao Y, Zhu X, Chua T (2014) Exploiting web images for semantic video indexing via robust sample-specific loss. IEEE Trans Multimedia 16(6):1677–1689

    Article  Google Scholar 

  32. Zhang X, Huang Z, Shen H, Yang Y, Li Z (2012) Automatic tagging by exploring tag information capability and correlation. World Wide Web 15(3):233–256

    Article  Google Scholar 

Download references

Acknowledgments

This work is partly supported by National Program on Key Basic Research Project (973 Program, under Grant 2013CB329301), the Major Project of National Social Science Fund (under Grant 14ZDB153), the NSFC (under Grant 61202166 and 61472276), and Doctoral Fund of Ministry of Education of China (under Grant 20120032120042).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Tianjin University, Tianjin, China

    Qiang Li, Yahong Han & Jianwu Dang

  2. Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China

    Yahong Han & Jianwu Dang

  3. School of Information Science, Japan Advanced Institute of Science and Technology, Ishikawa, Japan

    Jianwu Dang

Authors
  1. Qiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yahong Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianwu Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yahong Han.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Q., Han, Y. & Dang, J. Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals. Multimed Tools Appl 75, 2419–2434 (2016). https://doi.org/10.1007/s11042-015-2645-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2645-y

Keywords

Search

Navigation