This paper presents an overview of the work we have done over the last several years on object recognition in images from region-based image representation. The overview focuses on the following related problems: (1) discovery of a single 2D object category frequently occurring in a given image set; (2) learning a model of the discovered category in terms of its photometric, geometric, and structural properties; and (3) detection and segmentation of objects from the category in new images. Images in the given set are segmented, and then each image is represented by a region graph that captures hierarchy and neighbor relations among image regions. The region graphs are matched to extract the maximally matching subgraphs, which are interpreted as instances of the discovered category. A graph-union of the matching subgraphs is taken as a model of the category. Matching the category model to the region graph of a new image yields joint object detection and segmentation. The paper argues that using a hierarchy of image regions and their neighbor relations offers a number of advantages in solving (1)-(3), over the more commonly used point and edge features. Experimental results, also reviewed in this paper, support the above claims. Details of our methods as well of comparisons with other methods are omitted here, and can be found in the indicated references.


Equal Error Rate Graph Match Neighbor Relation Object Discovery Region Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Agarwal, S., Awan, A., Roth, D.: Learning to detect objects in images via a sparse, part-based representation. IEEE TPAMI 26(11), 1475–1490 (2004)Google Scholar
  2. 2.
    Ahmadyfard, A.R., Kittler, J.V.: Using relaxation technique for region-based object recognition. Image and Vision Computing 20(11), 769–781 (2002)CrossRefGoogle Scholar
  3. 3.
    Ahuja, N.: A transform for multiscale image segmentation by integrated edge and region detection. IEEE TPAMI 18(12), 1211–1235 (1996)MathSciNetGoogle Scholar
  4. 4.
    Ahuja, N., Todorovic, S.: Extracting texels in 2.1D natural textures. In: ICCV (2007)Google Scholar
  5. 5.
    Ahuja, N., Todorovic, S.: Learning the taxonomy and models of categories present in arbitrary images. In: ICCV (2007)Google Scholar
  6. 6.
    Ahuja, N., Todorovic, S.: Connected segmentation tree – a joint representation of region layout and hierarchy. In: CVPR (2008)Google Scholar
  7. 7.
    Arora, H., Ahuja, N.: Analysis of ramp discontinuity model for multiscale image segmentation. In: ICPR, vol. 4, pp. 99–103 (2006)Google Scholar
  8. 8.
    Basri, R., Jacobs, D.: Recognition using region correspondences. IJCV 25(2), 145–166 (1997)CrossRefGoogle Scholar
  9. 9.
    Borenstein, E., Ullman, S.: Class-specific, top-down segmentation. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 109–124. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  10. 10.
    Bouman, C.A., Shapiro, M.: A multiscale random field model for Bayesian image segmentation. IEEE Trans. Image Processing 3(2), 162–177 (1994)CrossRefGoogle Scholar
  11. 11.
    Brice, C.R., Fennema, C.L.: Scene analysis using regions. Artificial Intelligence 1, 205–226 (1970)CrossRefGoogle Scholar
  12. 12.
    Bunke, H., Allermann, G.: Inexact graph matching for structural pattern recognition. Pattern Rec. Letters 1(4), 245–253 (1983)zbMATHCrossRefGoogle Scholar
  13. 13.
    Bunke, H., Foggia, P., Guidobaldi, C., Vento, M.: Graph clustering using the weighted minimum common supergraph. In: Hancock, E.R., Vento, M. (eds.) GbRPR 2003. LNCS, vol. 2726, pp. 235–246. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  14. 14.
    Bunke, H., Jiang, X., Kandel, A.: On the minimum common supergraph of two graphs. Computing 65(1), 13–25 (2000)zbMATHMathSciNetGoogle Scholar
  15. 15.
    Bunke, H., Kandel, A.: Mean and maximum common subgraph of two graphs. Pattern Rec. Letters 21(2), 163–168 (2000)CrossRefGoogle Scholar
  16. 16.
    Canny, J.: A computational approach to edge detection. IEEE TPAMI 8(6), 679–698 (1986)Google Scholar
  17. 17.
    Cao, L., Fei-Fei, L.: Spatially coherent latent topic model for concurrent segmentation and classification of objects and scenes. In: ICCV (2007)Google Scholar
  18. 18.
    Darwish, A.M., Jain, A.K.: A rule based approach for visual pattern inspection. IEEE Trans. Pattern Analysis Machine Intelligence 10(1), 56–68 (1988)CrossRefGoogle Scholar
  19. 19.
    Demirci, M.F., Shokoufandeh, A., Keselman, Y., Bretzner, L., Dickinson, S.J.: Object recognition as many-to-many feature matching. Int. J. Computer Vision 69(2), 203–222 (2006)CrossRefGoogle Scholar
  20. 20.
    Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE TPAMI 28(4), 594–611 (2006)Google Scholar
  21. 21.
    Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: CVPR, vol. 2, pp. 264–271 (2003)Google Scholar
  22. 22.
    Fidler, S., Leonardis, A.: Towards scalable representations of object categories: Learning a hierarchy of parts. In: CVPR (2007)Google Scholar
  23. 23.
    Finch, A.M., Wilson, R.C., Hancock, E.R.: An energy function and continuous edit process for graph matching. Neural Computation 10(7) (1998)Google Scholar
  24. 24.
    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE TPAMI 6(6) (1984)Google Scholar
  25. 25.
    Glantz, R., Pelillo, M., Kropatsch, W.G.: Matching segmentation hierarchies. Int. J. Pattern Rec. Artificial Intelligence 18(3), 397–424 (2004)CrossRefGoogle Scholar
  26. 26.
    Gold, S., Rangarajan, A.: A graduated assignment algorithm for graph matching. IEEE TPAMI 18(4), 377–388 (1996)Google Scholar
  27. 27.
    Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009)Google Scholar
  28. 28.
    Gu, C., Lim, J.J., Arbelaez, P., Malik, J.: Recognition using regions. In: CVPR (2009)Google Scholar
  29. 29.
    Gupta, A., Nishimura, N.: Finding largest subtrees and smallest supertrees. Algorithmica 21(2), 183–210 (1998)zbMATHCrossRefMathSciNetGoogle Scholar
  30. 30.
    Jiang, H., Drew, M.S., Li, Z.N.: Matching by linear programming and successive convexification. IEEE TPAMI 29(6), 959–975 (2007)Google Scholar
  31. 31.
    Kittler, J., Hancock, E.R.: Contextual decision rule for region analysis. Image Vision Comput. 5(2), 145–153 (1987)CrossRefGoogle Scholar
  32. 32.
    Leibe, B., Leonardis, A., Schiele, B.: Combined object categorization and segmentation with an implicit shape model. In: Workshop on Statistical Learning in Computer Vision, ECCV, pp. 17–32 (2004)Google Scholar
  33. 33.
    Lim, J.J., Arbelaez, P., Gu, C., Malik, J.: Context by region ancestry. In: ICCV (2009)Google Scholar
  34. 34.
    Lindeberg, T.: Scale-Space Theory in Computer Vision. Kluwer Academic Publishers, Norwell (1994)Google Scholar
  35. 35.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  36. 36.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. PAMI 26, 530–549 (2004)Google Scholar
  37. 37.
    Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Gool, L.V.: A comparison of affine region detectors. IJCV 65(1/2), 43–72 (2005)CrossRefGoogle Scholar
  38. 38.
    Mumford, D., Shah, J.: Boundary detection by minimizing functionals. In: CVPR, pp. 22–26 (1985)Google Scholar
  39. 39.
    Opelt, A., Pinz, A., Zisserman, A.: Incremental learning of object detectors using a visual shape alphabet. In: CVPR, vol. 1, pp. 3–10 (2006)Google Scholar
  40. 40.
    Pelillo, M.: Matching free trees, maximal cliques, and monotone game dynamics. IEEE TPAMI 24(11), 1535–1541 (2002)Google Scholar
  41. 41.
    Pelillo, M., Siddiqi, K., Zucker, S.W.: Matching hierarchical structures using association graphs. IEEE TPAMI 21(11), 1105–1120 (1999)Google Scholar
  42. 42.
    Pelillo, M., Siddiqi, K., Zucker, S.W.: Many-to-many matching of attributed trees using association graphs and game dynamics. In: Arcelli, C., Cordella, L.P., Sanniti di Baja, G. (eds.) IWVF 2001. LNCS, vol. 2059, pp. 583–593. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  43. 43.
    Peng, J., Bhanu, B.: Closed-loop object recognition using reinforcement learning. IEEE Trans. Pattern Analysis Machine Intelligence 20(2), 139–154 (1998)CrossRefGoogle Scholar
  44. 44.
    Qiu, H., Hancock, E.R.: Graph matching and clustering using spectral partitions. Pattern Recognition 39(1), 22–34 (2006)CrossRefGoogle Scholar
  45. 45.
    Russell, B.C., Efros, A.A., Sivic, J., Freeman, W.T., Zisserman, A.: Using multiple segmentations to discover objects and their extent in image collections. In: CVPR, vol. 2, pp. 1605–1614 (2006)Google Scholar
  46. 46.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. IJCV 77(1-3), 157–173 (2008)CrossRefGoogle Scholar
  47. 47.
    Sebastian, T.B., Klein, P.N., Kimia, B.B.: Recognition of shapes by editing their shock graphs. IEEE Trans. Pattern Anal. Machine Intell. 26(5), 550–571 (2004)CrossRefGoogle Scholar
  48. 48.
    Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual cortex. In: CVPR, vol. 2, pp. 994–1000 (2005)Google Scholar
  49. 49.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE TPAMI 22(8), 888–905 (2000)Google Scholar
  50. 50.
    Shokoufandeh, A., Macrini, D., Dickinson, S., Siddiqi, K., Zucker, S.W.: Indexing hierarchical structures using graph spectra. IEEE TPAMI 27(7), 1125–1140 (2005)Google Scholar
  51. 51.
    Siddiqi, K., Shokoufandeh, A., Dickinson, S.J., Zucker, S.W.: Shock graphs and shape matching. IJCV 35(1), 13–32 (1999)CrossRefGoogle Scholar
  52. 52.
    Sudderth, E., Torralba, A., Freeman, W., Willsky, A.: Learning hierarchical models of scenes, objects, and parts. In: ICCV, vol. 2, pp. 1331–1338 (2005)Google Scholar
  53. 53.
    Tabb, M., Ahuja, N.: Multiscale image segmentation by integrated edge and region detection. IEEE Trans. Image Processing 6(5), 642–655 (1997)CrossRefGoogle Scholar
  54. 54.
    Todorovic, S., Ahuja, N.: Extracting subimages of an unknown category from a set of images. In: CVPR, vol. 1, pp. 927–934 (2006)Google Scholar
  55. 55.
    Todorovic, S., Ahuja, N.: Learning subcategory relevances to category recognition. In: CVPR (2008)Google Scholar
  56. 56.
    Todorovic, S., Ahuja, N.: Unsupervised category modeling, recognition, and segmentation in images. IEEE TPAMI 30(12), 1–17 (2008)Google Scholar
  57. 57.
    Todorovic, S., Ahuja, N.: Texel-based texture segmentation. In: ICCV (2009)Google Scholar
  58. 58.
    Todorovic, S., Ahuja, N.: Region-based hierarchical image matching. IJCV (to appear)Google Scholar
  59. 59.
    Torralba, A., Murphy, K., Freeman, W.: Sharing features: efficient boosting procedures for multiclass object detection. In: CVPR, vol. 2, pp. 762–769 (2004)Google Scholar
  60. 60.
    Torsello, A., Hancock, E.R.: Matching and embedding through edit-union of trees. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2352, pp. 822–836. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  61. 61.
    Torsello, A., Hancock, E.R.: Learning shape-classes using a mixture of tree-unions. IEEE Trans. PAMI 28(6), 954–967 (2006)Google Scholar
  62. 62.
    Torsello, A., Robles-Kelly, A., Hancock, E.R.: Discovering shape classes using tree edit-distance and pairwise clustering. IJCV 72(3), 259–285 (2007)CrossRefGoogle Scholar
  63. 63.
    Tu, Z., Yuille, A.: Shape matching and recognition - using generative models and informative features. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3023, pp. 195–209. Springer, Heidelberg (2004)Google Scholar
  64. 64.
    Weiss, I., Ray, M.: Recognizing articulated objects using a region-based invariant transform. IEEE Trans. Pattern Analysis Machine Intelligence 27(10), 1660–1665 (2005)CrossRefGoogle Scholar
  65. 65.
    Winn, J., Criminisi, A., Minka, T.: Object categorization by learned universal visual dictionary. In: ICCV, vol. 2, pp. 1800–1807 (2005)Google Scholar
  66. 66.
    Winn, J., Jojic, N.: Locus: learning object classes with unsupervised segmentation. In: ICCV, pp. 756–763 (2005)Google Scholar
  67. 67.
    Worthington, P.L., Hancock, E.R.: Object recognition using shape-from-shading. IEEE TPAMI 23(5), 535–542 (2001)Google Scholar
  68. 68.
    Worthington, P.L., Hancock, E.R.: Region-based object recognition using shape-from-shading. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 455–471. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  69. 69.
    Zhang, R., Zhang, Z.: Hidden semantic concept discovery in region based image retrieval. In: CVPR, vol. 2, pp. 996–1001 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Narendra Ahuja
    • 1
  • Sinisa Todorovic
    • 2
  1. 1.Department of Electrical and Computer Engineering, Coordinated Science Lab, and Beckman InstituteUniversity of Illinois Urbana-Champaign 
  2. 2.School of Electrical Engineering and Computer ScienceOregon State University 

Personalised recommendations