Skip to main content
Log in

Contextual local primitives for binary patent image retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Local features and descriptors that perform well in the case of photographic images are often unable to capture the content of binary technical drawings due to their different characteristics. Motivated by this, a new local feature representation, the contextual local primitives, is proposed in this paper. It is based on the detection of the junction and end points, classification of the local primitives to local primitive words and establishment of the geodesic connections of the local primitives. We exploit the granulometric information of the binary patent images to set all the necessary parameters of the involved mathematical morphology operators and window size for the local primitive extraction, which makes the whole framework parameter free. The contextual local primitives and, their spatial areas as a histogram weighting factor are evaluated by performing binary patent image retrieval experiments. It is found that the proposed contextual local primitives perform better than the local primitives only, the SIFT description of the contextual Hessian points, the SIFT description of local primitives and state of the art local content capturing methods. Moreover, an analysis of the approach in the perspective of a general patent image retrieval system reveals of its being efficient in multiple aspects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Notes

  1. http://www.patexpert.org/.

  2. http://mklab.iti.gr/content/patent-database.

  3. http://www.ir-facility.org/prototypes/marec.

  4. http://mklab.iti.gr/project/patentbase.

  5. http://mklab.iti.gr/project/patentbase.

References

  1. Attali D, Boissonnat JD, Edelsbrunner H (2009) Stability and computation of medial axes-a state-of-the-art report. In: Mathematical foundations of scientific visualization, computer graphics, and massive data exploration, pp 109–125

  2. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522

    Article  Google Scholar 

  3. Bergevin R, Filiatrault A (2007) Enhancing contour primitives by pairwise grouping and relaxation. In: Proceedings of 4th international conference on image analysis and recognition (ICIAR), pp 222–233

  4. Bhatti NA, Hanbury A (2011) Detection and classification of local primitives in line drawings. In: Proceedings of the AAPR workshop

  5. Bhatti NA, Hanbury A (2011) Granulometry based detection of junction and end points in patent drawings. In: 2011 7th International symposium on image and signal processing and analysis (ISPA), pp 307–312

  6. Bhatti N, Hanbury A (2011) Morphology based spatial relationships between local primitives in line drawings. In: CIARP, pp 165–172

  7. Bhatti N, Hanbury A (2012) Image search in patents: a review. In: International journal on document analysis and recognition (IJDAR), pp 1–21

  8. Castanedo F (2013) A review of data fusion techniques. Sci World J 2013:1–19

    Article  Google Scholar 

  9. Choi MJ, Torralba A, Willsky AS (2012) A tree-based context model for object recognition. IEEE Trans Pattern Anal Mach Intell 34(2):240–252

    Article  Google Scholar 

  10. Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision (ECCV), pp 1–22

  11. Csurka G, Renders J, Jacquet G (2011) XRCE’s participation at patent image classification and image-based patent retrieval tasks of the Clef-IP 2011. In: V Petras, P Forner, PD Clough (eds) CLEF (Notebook Papers/Labs/Workshop)

  12. Das M, Manmatha R, Riseman EM (1999) Indexing flower patent images using domain knowledge. IEEE Intell Syst 14(5):24–33

    Article  Google Scholar 

  13. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60

    Article  Google Scholar 

  14. Deseilligny MP, Stamon G, Ching YS (1998) Veinerization: a new shape description for flexible skeletonization. IEEE Trans Pattern Anal Mach Intell 20(5):505–521

    Article  Google Scholar 

  15. Desolneux A, Moisan L, Morel JM (2004) Seeing, thinking and knowing. In: Carsetti A (ed). Kluwer Academic Publishers, Norwell

  16. Fonseca MJ, Ferreira A, Jorge JA (2004) Content-based retrieval of technical drawings. In: Special issue of international journal of computer applications in technology (IJCAT)

  17. Fonseca MJ, Ferreira A, Jorge JA (2009) Sketch-based retrieval of complex drawings using hierarchical topology and geometry. Comput Aided Des 41(12):1067–1081

    Article  Google Scholar 

  18. Förstner W (1999) Uncertain neighborhood relations of point sets and fuzzy delaunay triangulation. In: Mustererkennung, 21. DAGM-Symposium, pp 213–222

  19. Galleguillos C, Belongie S (2010) Context based object categorization: a critical survey. Comput Vis Image Underst 114(6):712–722

    Article  Google Scholar 

  20. Hanbury A, Bhatti N, Lupu M, Morzinger R (2011) Patent image retrieval: a survey. In: Proceedings of the patent inforamtion retrieval workshop (PaIR). ACM, pp 3–8

  21. Heikkila M, Pietikainen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern Recogn 42(3):425–436

    Article  MATH  Google Scholar 

  22. Huet B, Guarascio G, Kern NJ, Mérialdo B. (2001) Relational skeletons for retrieval in patent drawings. In: ICIP, pp 737–740

  23. Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition - volume 2, CVPR ’06, pp 2169–2178

  24. Leung WH, Chen T (2002) User-independent retrieval of free-form hand-drawn sketches. In: ICASSP, pp 2029–2032

  25. Li FF, Fergus R, Perona P (2007) Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput Vis Image Underst 106(1):59–70

    Article  Google Scholar 

  26. List J (2007) How drawings could enhance retrieval in mechanical and device patent searching. World Patent Inf 29(3):210–218

    Article  Google Scholar 

  27. Liu R, Wang Y, Baba T, Masumoto D (2010) Shape detection from line drawings with local neighborhood structure. Pattern Recogn 43(5):1907–1916

    Article  MATH  Google Scholar 

  28. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis 60(2):91–110

    Article  Google Scholar 

  29. Mahmoudi F, Shanbehzadeh J, Eftekhari-Moghadam A, Soltanian-Zadeh H (2003) Image retrieval based on shape similarity by edge orientation autocorrelogram. Pattern Recogn 1725–1736

  30. Maire M, Arbelaez P, Fowlkes C, Malik J (2008) Using contours to detect and localize junctions in natural images. In: CVPR, pp 1–8

  31. Mikolajczyk K (2002) Scale and affine invariant interest point detectors. PhD Thesis

  32. Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Gool LV (2005) A comparison of affine region detectors. Int J Comput Vis 65:43–47

    Article  Google Scholar 

  33. Newby GB (1997) Context-based statistical sub-spaces. In: TREC, pp 735–745

  34. Olson CF, Hoover SA, Soltman JL, Zhang S (2016) Complementary keypoint descriptors. Springer International Publishing, pp 341–352

  35. Papari G, Petkov N (2011) Edge and line oriented contour detection: state of the art. Image Vis Comput 29(2-3):79–103

    Article  Google Scholar 

  36. Park JH, Um BS (1999) A new approach to similarity retrieval of 2-D graphic objects based on dominant shapes. Pattern Recogn Lett 20(6):591–616

    Article  Google Scholar 

  37. Parker C, Chen T (2003) Hierarchical matching for retrieval of hand-drawn sketches. In: ICME, pp 29–32

  38. Heuel S, WF (1998) A dual, scalable and hierarchical representation for perceptual organization of binary images. In: Workshop on perceptual organization in computer vision. IEEE Computer Society

  39. Santosh KC, Wendling L, Lamiroy B (2010) Unified pairwise spatial relations: an application to graphical symbol retrieval. In: Proceedings of the 8th international conference on graphics recognition: achievements, challenges, and evolution, GREC’09, pp 163–174

  40. Shen J (2009) Stochastic modeling western paintings for effective classification. Pattern Recogn 42(2):293–301

    Article  MATH  Google Scholar 

  41. Shen J, Deng RH, Cheng Z, Nie L, Yan S (2015) On robust image spam filtering via comprehensive visual modeling. Pattern Recogn 48(10):3227–3238

    Article  Google Scholar 

  42. Shotton J, Johnson M, Cipolla R (2008) Semantic texton forests for image categorization and segmentation. In: CVPR. IEEE Computer Society

  43. Sidiropoulos P, Vrochidis S, Kompatsiaris I (2010) Adaptive hierarchical density histogram for complex binary image retrieval. In: CBMI, pp 1–6

  44. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  45. Soille P (2003) Morphological image analysis: principles and applications, 2nd edn. Springer-Verlag New York, Inc., Secaucus

    MATH  Google Scholar 

  46. Tiwari A, Bansal V (2004) Patseek: content based image retrieval system for patent database. In: ICEB, pp 1167–1171

  47. Tuytelaars T, Mikolajczyk K (2008) Local invariant feature detectors - a survey. Found Trends Comput Graph Vis

  48. Vrochidis S, Papadopoulos S, Moumtzidou A, Sidiropoulos P, Pianta E, Kompatsiaris I (2010) Towards content-based patent image retrieval: a framework perspective. World Patent Inf 32(2):94–106

    Article  Google Scholar 

  49. Wong SKM, Ziarko W, Wong PCN (1985) Generalized vector space model in information retrieval. In: SIGIR, pp 18–25

  50. Xie L, Shen J, Zhu L (2016) Online cross-modal hashing for web image retrieval. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, AAAI’16. AAAI Press, pp 294–300

  51. Yang M, Qiu G, Huang J, Elliman D (2006) Near-duplicate image recognition and content-based image retrieval using adaptive hierarchical geometric centroids. In: ICPR, pp 958–961

  52. Zhiyuan Z, Juan Z, Bin X (2007) An outward-appearance patent-image retrieval approach based on the contour-description matrix. In: FCST, pp 86–89

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Naeem Bhatti.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhatti, N., Hanbury, A. & Stottinger, J. Contextual local primitives for binary patent image retrieval. Multimed Tools Appl 77, 9111–9151 (2018). https://doi.org/10.1007/s11042-017-4808-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4808-5

Keywords

Navigation