Multimedia Tools and Applications

, Volume 77, Issue 7, pp 9111–9151 | Cite as

Contextual local primitives for binary patent image retrieval

  • Naeem Bhatti
  • Allan Hanbury
  • Julian Stottinger


Local features and descriptors that perform well in the case of photographic images are often unable to capture the content of binary technical drawings due to their different characteristics. Motivated by this, a new local feature representation, the contextual local primitives, is proposed in this paper. It is based on the detection of the junction and end points, classification of the local primitives to local primitive words and establishment of the geodesic connections of the local primitives. We exploit the granulometric information of the binary patent images to set all the necessary parameters of the involved mathematical morphology operators and window size for the local primitive extraction, which makes the whole framework parameter free. The contextual local primitives and, their spatial areas as a histogram weighting factor are evaluated by performing binary patent image retrieval experiments. It is found that the proposed contextual local primitives perform better than the local primitives only, the SIFT description of the contextual Hessian points, the SIFT description of local primitives and state of the art local content capturing methods. Moreover, an analysis of the approach in the perspective of a general patent image retrieval system reveals of its being efficient in multiple aspects.


Patent image retrieval Local features Local primitives Contextual features 


  1. 1.
    Attali D, Boissonnat JD, Edelsbrunner H (2009) Stability and computation of medial axes-a state-of-the-art report. In: Mathematical foundations of scientific visualization, computer graphics, and massive data exploration, pp 109–125Google Scholar
  2. 2.
    Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522CrossRefGoogle Scholar
  3. 3.
    Bergevin R, Filiatrault A (2007) Enhancing contour primitives by pairwise grouping and relaxation. In: Proceedings of 4th international conference on image analysis and recognition (ICIAR), pp 222–233Google Scholar
  4. 4.
    Bhatti NA, Hanbury A (2011) Detection and classification of local primitives in line drawings. In: Proceedings of the AAPR workshopGoogle Scholar
  5. 5.
    Bhatti NA, Hanbury A (2011) Granulometry based detection of junction and end points in patent drawings. In: 2011 7th International symposium on image and signal processing and analysis (ISPA), pp 307–312Google Scholar
  6. 6.
    Bhatti N, Hanbury A (2011) Morphology based spatial relationships between local primitives in line drawings. In: CIARP, pp 165–172Google Scholar
  7. 7.
    Bhatti N, Hanbury A (2012) Image search in patents: a review. In: International journal on document analysis and recognition (IJDAR), pp 1–21Google Scholar
  8. 8.
    Castanedo F (2013) A review of data fusion techniques. Sci World J 2013:1–19CrossRefGoogle Scholar
  9. 9.
    Choi MJ, Torralba A, Willsky AS (2012) A tree-based context model for object recognition. IEEE Trans Pattern Anal Mach Intell 34(2):240–252CrossRefGoogle Scholar
  10. 10.
    Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision (ECCV), pp 1–22Google Scholar
  11. 11.
    Csurka G, Renders J, Jacquet G (2011) XRCE’s participation at patent image classification and image-based patent retrieval tasks of the Clef-IP 2011. In: V Petras, P Forner, PD Clough (eds) CLEF (Notebook Papers/Labs/Workshop)Google Scholar
  12. 12.
    Das M, Manmatha R, Riseman EM (1999) Indexing flower patent images using domain knowledge. IEEE Intell Syst 14(5):24–33CrossRefGoogle Scholar
  13. 13.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  14. 14.
    Deseilligny MP, Stamon G, Ching YS (1998) Veinerization: a new shape description for flexible skeletonization. IEEE Trans Pattern Anal Mach Intell 20(5):505–521CrossRefGoogle Scholar
  15. 15.
    Desolneux A, Moisan L, Morel JM (2004) Seeing, thinking and knowing. In: Carsetti A (ed). Kluwer Academic Publishers, NorwellGoogle Scholar
  16. 16.
    Fonseca MJ, Ferreira A, Jorge JA (2004) Content-based retrieval of technical drawings. In: Special issue of international journal of computer applications in technology (IJCAT)Google Scholar
  17. 17.
    Fonseca MJ, Ferreira A, Jorge JA (2009) Sketch-based retrieval of complex drawings using hierarchical topology and geometry. Comput Aided Des 41(12):1067–1081CrossRefGoogle Scholar
  18. 18.
    Förstner W (1999) Uncertain neighborhood relations of point sets and fuzzy delaunay triangulation. In: Mustererkennung, 21. DAGM-Symposium, pp 213–222Google Scholar
  19. 19.
    Galleguillos C, Belongie S (2010) Context based object categorization: a critical survey. Comput Vis Image Underst 114(6):712–722CrossRefGoogle Scholar
  20. 20.
    Hanbury A, Bhatti N, Lupu M, Morzinger R (2011) Patent image retrieval: a survey. In: Proceedings of the patent inforamtion retrieval workshop (PaIR). ACM, pp 3–8Google Scholar
  21. 21.
    Heikkila M, Pietikainen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern Recogn 42(3):425–436CrossRefzbMATHGoogle Scholar
  22. 22.
    Huet B, Guarascio G, Kern NJ, Mérialdo B. (2001) Relational skeletons for retrieval in patent drawings. In: ICIP, pp 737–740Google Scholar
  23. 23.
    Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the 2006 IEEE computer society conference on computer vision and pattern recognition - volume 2, CVPR ’06, pp 2169–2178Google Scholar
  24. 24.
    Leung WH, Chen T (2002) User-independent retrieval of free-form hand-drawn sketches. In: ICASSP, pp 2029–2032Google Scholar
  25. 25.
    Li FF, Fergus R, Perona P (2007) Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput Vis Image Underst 106(1):59–70CrossRefGoogle Scholar
  26. 26.
    List J (2007) How drawings could enhance retrieval in mechanical and device patent searching. World Patent Inf 29(3):210–218CrossRefGoogle Scholar
  27. 27.
    Liu R, Wang Y, Baba T, Masumoto D (2010) Shape detection from line drawings with local neighborhood structure. Pattern Recogn 43(5):1907–1916CrossRefzbMATHGoogle Scholar
  28. 28.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis 60(2):91–110CrossRefGoogle Scholar
  29. 29.
    Mahmoudi F, Shanbehzadeh J, Eftekhari-Moghadam A, Soltanian-Zadeh H (2003) Image retrieval based on shape similarity by edge orientation autocorrelogram. Pattern Recogn 1725–1736Google Scholar
  30. 30.
    Maire M, Arbelaez P, Fowlkes C, Malik J (2008) Using contours to detect and localize junctions in natural images. In: CVPR, pp 1–8Google Scholar
  31. 31.
    Mikolajczyk K (2002) Scale and affine invariant interest point detectors. PhD ThesisGoogle Scholar
  32. 32.
    Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Gool LV (2005) A comparison of affine region detectors. Int J Comput Vis 65:43–47CrossRefGoogle Scholar
  33. 33.
    Newby GB (1997) Context-based statistical sub-spaces. In: TREC, pp 735–745Google Scholar
  34. 34.
    Olson CF, Hoover SA, Soltman JL, Zhang S (2016) Complementary keypoint descriptors. Springer International Publishing, pp 341–352Google Scholar
  35. 35.
    Papari G, Petkov N (2011) Edge and line oriented contour detection: state of the art. Image Vis Comput 29(2-3):79–103CrossRefGoogle Scholar
  36. 36.
    Park JH, Um BS (1999) A new approach to similarity retrieval of 2-D graphic objects based on dominant shapes. Pattern Recogn Lett 20(6):591–616CrossRefGoogle Scholar
  37. 37.
    Parker C, Chen T (2003) Hierarchical matching for retrieval of hand-drawn sketches. In: ICME, pp 29–32Google Scholar
  38. 38.
    Heuel S, WF (1998) A dual, scalable and hierarchical representation for perceptual organization of binary images. In: Workshop on perceptual organization in computer vision. IEEE Computer SocietyGoogle Scholar
  39. 39.
    Santosh KC, Wendling L, Lamiroy B (2010) Unified pairwise spatial relations: an application to graphical symbol retrieval. In: Proceedings of the 8th international conference on graphics recognition: achievements, challenges, and evolution, GREC’09, pp 163–174Google Scholar
  40. 40.
    Shen J (2009) Stochastic modeling western paintings for effective classification. Pattern Recogn 42(2):293–301CrossRefzbMATHGoogle Scholar
  41. 41.
    Shen J, Deng RH, Cheng Z, Nie L, Yan S (2015) On robust image spam filtering via comprehensive visual modeling. Pattern Recogn 48(10):3227–3238CrossRefGoogle Scholar
  42. 42.
    Shotton J, Johnson M, Cipolla R (2008) Semantic texton forests for image categorization and segmentation. In: CVPR. IEEE Computer SocietyGoogle Scholar
  43. 43.
    Sidiropoulos P, Vrochidis S, Kompatsiaris I (2010) Adaptive hierarchical density histogram for complex binary image retrieval. In: CBMI, pp 1–6Google Scholar
  44. 44.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  45. 45.
    Soille P (2003) Morphological image analysis: principles and applications, 2nd edn. Springer-Verlag New York, Inc., SecaucuszbMATHGoogle Scholar
  46. 46.
    Tiwari A, Bansal V (2004) Patseek: content based image retrieval system for patent database. In: ICEB, pp 1167–1171Google Scholar
  47. 47.
    Tuytelaars T, Mikolajczyk K (2008) Local invariant feature detectors - a survey. Found Trends Comput Graph VisGoogle Scholar
  48. 48.
    Vrochidis S, Papadopoulos S, Moumtzidou A, Sidiropoulos P, Pianta E, Kompatsiaris I (2010) Towards content-based patent image retrieval: a framework perspective. World Patent Inf 32(2):94–106CrossRefGoogle Scholar
  49. 49.
    Wong SKM, Ziarko W, Wong PCN (1985) Generalized vector space model in information retrieval. In: SIGIR, pp 18–25Google Scholar
  50. 50.
    Xie L, Shen J, Zhu L (2016) Online cross-modal hashing for web image retrieval. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, AAAI’16. AAAI Press, pp 294–300Google Scholar
  51. 51.
    Yang M, Qiu G, Huang J, Elliman D (2006) Near-duplicate image recognition and content-based image retrieval using adaptive hierarchical geometric centroids. In: ICPR, pp 958–961Google Scholar
  52. 52.
    Zhiyuan Z, Juan Z, Bin X (2007) An outward-appearance patent-image retrieval approach based on the contour-description matrix. In: FCST, pp 86–89Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Naeem Bhatti
    • 1
  • Allan Hanbury
    • 2
  • Julian Stottinger
    • 3
  1. 1.Department of ElectronicsQuaid-i-Azam UniversityIslamabadPakistan
  2. 2.Institute of Software Technology and Interactive SystemsVienna University of TechnologyViennaAustria
  3. 3.Department of Information Engineering and Computer ScienceUniversity of TrentoTrentoItaly

Personalised recommendations