Skip to main content
SpringerLink
Log in

Unsupervised Learning of Graph and Hypergraph Matching

  • Chapter
  • First Online:
Unsupervised Learning in Space and Time

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

  • 1581 Accesses

Abstract

Graph and hypergraph matching are fundamental problems in computer vision, which have been receiving a steadily growing attention in the last two decades due to the recent modern algorithms and models that are both fast and accurate. Graph and hypergraph matching are successful in tackling many real-world tasks that require 2D and 3D feature matching and alignment. Matching is a general problem in vision and graphs have always been quintessential for modeling information that has both local and longer range connections. Therefore, graph matching will continue to influence computer vision and robotics research, crossing over many different mathematical and computational models, including Bayesian networks, conditional and Markov random fields, spectral graph models, and, more recently, deep neural networks. While graph matching is limited to using pairwise relationships, hypergraph matching permits the use of relationships between sets of features of any order. Consequently, it carries the promise to make matching more robust to changes in scale, deformations, and outliers. In this chapter, we present several key ideas, computational models, and optimization methods that introduced in the literature the possibility to learn graph and hypergraph matching in a semi-supervised or even completely unsupervised fashion. We start by describing two methods for solving graph and hypergraph matching based on spectral matching and integer projected fixed point methods, introduced in Chap. 1. Then, based on the same intuition (also discussed in Chap. 1) that correct assignments form a strong cluster of agreements, we follow up by introducing the main approach to learning as it applies to graph and hypergraph matching, as well as the more general Maximum A Posteriori (MAP) inference problem in probabilistic graphical models (MRFs and CRFs). In the last part of the chapter, we demonstrate experimentally the effectiveness of the algorithms presented and, whenever possible, provide extensive comparisons to other top published methods. Moreover, we show that our learning approach is general and can significantly boost the performance of different state-of-the-art matching algorithms in the literature.

The material presented in this chapter is based in large part on the following papers:

Leordeanu, Marius, and Martial Hebert. “A spectral technique for correspondence problems using pairwise constraints.” IEEE International Conference on Computer Vision (ICCV), 2005.

Leordeanu, Marius, Martial Hebert, and Rahul Sukthankar. “An integer projected fixed point method for graph matching and map inference.” In Advances in neural information processing systems (NIPS) 2009.

Leordeanu, Marius, Rahul Sukthankar, and Martial Hebert. “Unsupervised learning for graph matching.” International journal of computer vision 96, no. 1 (2012): 28–45.

Leordeanu, Marius, Andrei Zanfir, and Cristian Sminchisescu. “Semi-supervised learning and optimization for hypergraph matching.” IEEE International Conference on Computer Vision, 2011.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.vision.caltech.edu/archive.html

References

  1. Gold S, Rangarajan A (1996) A graduated assignment algorithm for graph matching. PAMI 18(4):377–388

    Article  Google Scholar 

  2. Berg A, Berg T, Malik J (2005) Shape matching and object recognition using low distortion correspondences. In: CVPR

    Google Scholar 

  3. Cour T, Srinivasan P, Shi J (2006) Balanced graph matching. In: NIPS

    Google Scholar 

  4. Schellewald C, Schnorr C (2005) Probabilistic subgraph matching based on convex relaxation. In: International conference on energy minimization methods in computer vision and pattern recognition

    Google Scholar 

  5. Torr P (2003) Solving Markov random fields using semidefinite programming. In: International conference on artificial intelligence and statistics

    Google Scholar 

  6. Leordeanu M, Hebert M, Sukthankar R (2009) An integer projected fixed point method for graph matching and map inference. In: NIPS

    Google Scholar 

  7. Cho M, Lee J, Lee KM (2010) Reweighted random walks for graph matching. In: ECCV

    Google Scholar 

  8. Egozi A, Keller Y, Guterman H (2013) A probabilistic approach to spectral graph matching. Pattern Anal Mach Intell 35(1)

    Google Scholar 

  9. Zhou F, la Torre FD (2012) Factorized graph matching. In: Computer vision and pattern recognition

    Google Scholar 

  10. Cho M, Lee K (2012) Progressive graph matching: making a move of graphs via probabilistic voting. In: Computer vision and pattern recognition

    Google Scholar 

  11. Tolias G, Avrithis Y (2011) Speeded-up, relaxed spatial matching. In: ICCV

    Google Scholar 

  12. Escolano F, Hancock E, Lozano M (2011) Graph matching through entropic manifold alignment. In: Computer vision and pattern recognition

    Google Scholar 

  13. Torresani L, Kolmogorov V, Rother C (2013) A dual decomposition approach to feature correspondence. Pattern Anal Mach Intell 35(2)

    Google Scholar 

  14. Liu Z, Qiao H, Yang X, Hoi S (2014) Graph matching by simplified convex-concave relaxation procedure. Int J Comput Vis 1–18

    Google Scholar 

  15. Zhou F, la Torre FD (2013) Deformable graph matching. In: Computer vision and pattern recognition, pp 2922–2929

    Google Scholar 

  16. Leordeanu M, Hebert M (2005) A spectral technique for correspondence problems using pairwise constraints. In: ICCV

    Google Scholar 

  17. Caetano T, Cheng L, Le Q, Smola AJ (2007) Learning graph matching. In: ICCV

    Google Scholar 

  18. Leordeanu M, Hebert M (2008) Smoothing-based optimization. In: CVPR

    Google Scholar 

  19. Leordeanu M, Sukthankar R, Hebert M (2009) Unsupervised learning for graph matching. IJCV 96(1)

    Google Scholar 

  20. Cho M, Alahari K, Ponce J (2013) Learning graphs to match. In: International conference on computer vision, pp 25–32

    Google Scholar 

  21. Zanfir A, Sminchisescu C (2018) Deep learning of graph matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2684–2693

    Google Scholar 

  22. Duchenne O, Bach F, Kweon I, Ponce J (2009) A tensor-based algorithm for high-order graph matching. In: CVPR

    Google Scholar 

  23. Chertok M, Keller Y (2010) Efficient high order matching. IEEE Trans Pattern Anal Mach Intell 32:2205–2215

    Google Scholar 

  24. Zass R, Shashua A (2008) Probabilistic graph and hypergraph matching. In: CVPR

    Google Scholar 

  25. Çeliktutan O, Wolf C, Sankur B, Lombardi E (2014) Fast exact hyper-graph matching with dynamic programming for spatio-temporal data. J Math Imaging Vis 1–21

    Google Scholar 

  26. Wang J, Chen S, Fuh C (2014) Attributed hypergraph matching on a Riemannian manifold. Mach Vis Appl 25(4)

    Google Scholar 

  27. Yan J, Zhang C, Zha H, Liu W, Yang X, Chu SM (2015) Discrete hyper-graph matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1520–1528

    Google Scholar 

  28. Nie WZ, Liu AA, Gao Z, Su YT (2015) Clique-graph matching by preserving global and local structure. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4503–4510

    Google Scholar 

  29. Liu AA, Nie WZ, Gao Y, Su YT (2016) Multi-modal clique-graph matching for view-based 3d model retrieval. IEEE Trans Image Process 25(5):2103–2116

    Article  MathSciNet  Google Scholar 

  30. Adamczewski K, Suh Y, Mu Lee K (2015) Discrete Tabu search for graph matching. In: Proceedings of the IEEE international conference on computer vision, pp 109–117

    Google Scholar 

  31. Jiang B, Tang J, Ding C, Luo B (2017) Binary constraint preserving graph matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4402–4409

    Google Scholar 

  32. Shi X, Ling H, Hu W, Xing J, Zhang Y (2016) Tensor power iteration for multi-graph matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5062–5070

    Google Scholar 

  33. Yan J, Cho M, Zha H, Yang X, Chu SM (2015) Multi-graph matching via affinity optimization with graduated consistency regularization. IEEE Trans Pattern Anal Mach Intell 38(6):1228–1242

    Google Scholar 

  34. Leordeanu MD (2010) Spectral graph matching, learning, and inference for computer vision. PhD Thesis, Carnegie Mellon University

    Google Scholar 

  35. Leordeanu M, Zanfir A, Sminchisescu C (2011) Semi-supervised learning and optimization for hypergraph matching. In: IEEE international conference on computer vision

    Google Scholar 

  36. Leordeanu M, Zanfir A, Sminchisescu C (2011) Semi-supervised learning and optimization for hypergraph matching. In: ICCV

    Google Scholar 

  37. Leordeanu M, Hebert M (2006) Efficient map approximation for dense energy functions. In: International conference on machine learning

    Google Scholar 

  38. Cour T, Shi J (2007) Solving Markov random fields with spectral relaxation. In: International conference on artificial intelligence and statistics

    Google Scholar 

  39. Ravikumar P, Lafferty J (2006) Quadratic programming relaxations for metric labeling and Markov random field map estimation. In: International conference on machine learning

    Google Scholar 

  40. Yan P, Khan SM, Shah M (2008) Learning 4D action feature models for arbitrary view action recognition. In: CVPR

    Google Scholar 

  41. Gaur U, Zhu Y, Song B, Roy-Chowdhury A (2011) A string of feature graphs model for recognition of complex activities in natural videos. In: International conference on computer vision, pp 2595–2602

    Google Scholar 

  42. Liu J, Luo J, Shah M (2009) Recognizing realistic actions from videos “in the wild”. In: CVPR

    Google Scholar 

  43. Nayak N, Sethi R, Song B, Roy-Chowdhury A (2011) Motion pattern analysis for modeling and recognition of complex human activities. In: Guide to video analysis of humans: looking at people

    Google Scholar 

  44. Zhu Y, Nayak N, Gaur U, Song B, Roy-Chowdhury A (2013) Modeling multi-object interactions using string of feature graphs. Comput Vis Image Underst 117(10)

    Google Scholar 

  45. Kim G, Faloutsos C, Hebert M (2008) Unsupervised modeling of object categories using link analysis techniques. In: CVPR

    Google Scholar 

  46. Kim G, Faloutsos C, Hebert M (2008) Unsupervised modeling and recognition of object categories with combination of visual contents and geometric similarity links. In: ACM international conference on multimedia information retrieval

    Google Scholar 

  47. Leordeanu M, Collins R, Hebert M (2005) Unsupervised learning of object features from video sequences. In: CVPR

    Google Scholar 

  48. Parikh D, Chen T (2007) Unsupervised learning of hierarchical semantics of objects (hSOs). In: CVPR

    Google Scholar 

  49. Parikh D, Chen T (2007) Unsupervised identification of multiple objects of interest from multiple images: discover. In: Asian conference on computer vision

    Google Scholar 

  50. Leordeanu M, Hebert M, Sukthankar R (2007) Beyond local appearance: category recognition from pairwise interactions of simple features. In: CVPR

    Google Scholar 

  51. de Aguiar E, Stoll C, Theobalt C, Ahmed N, Seidel HP, Thrun S (2008) Performance capture from sparse multi-view video. In: SIGGRAPH

    Google Scholar 

  52. Huhle B, Jenke P, Straer W (2008) On-the-fly scene acquisition with a handy multi-sensor system. Int J Intell Syst Technol Appl 5(3–4):255–263

    Google Scholar 

  53. Ren X (2007) Learning and matching line aspects for articulated objects. In: CVPR

    Google Scholar 

  54. Chertok M, Keller Y (2010) Spectral symmetry analysis. PAMI 32(7):1227–1238

    Google Scholar 

  55. Hays J, Leordeanu M, Efros A, Liu Y (2006) Discovering texture regularity as a higher-order correspondence problem. In: ECCV

    Google Scholar 

  56. Yi L, Kim VG, Ceylan D, Shen I, Yan M, Su H, Lu C, Huang Q, Sheffer A, Guibas L et al (2016) A scalable active framework for region annotation in 3d shape collections. ACM Trans Graph (TOG) 35(6):210

    Article  Google Scholar 

  57. Im K, Raschle NM, Smith SA, Ellen Grant P, Gaab N (2015) Atypical sulcal pattern in children with developmental dyslexia and at-risk kindergarteners. Cereb Cortex 26(3):1138–1148

    Article  Google Scholar 

  58. Im K, Pienaar R, Lee JM, Seong JK, Choi YY, Lee KH, Grant PE (2011) Quantitative comparison and analysis of sulcal patterns using sulcal graph matching: a twin study. Neuroimage 57(3):1077–1086

    Article  Google Scholar 

  59. Im K, Guimaraes A, Kim Y, Cottrill E, Gagoski B, Rollins C, Ortinau C, Yang E, Grant PE (2017) Quantitative folding pattern analysis of early primary sulci in human fetuses with brain abnormalities. Am J Neuroradiol 38(7):1449–1455

    Article  Google Scholar 

  60. Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32(7):1153

    Article  Google Scholar 

  61. Singh R, Xu J, Berger B (2008) Global alignment of multiple protein interaction networks with application to functional orthology detection. Proc Natl Acad Sci 105(35):12,763–12,768

    Google Scholar 

  62. Jiang ZP, Xi W, Li X, Tang S, Zhao JZ, Han JS, Zhao K, Wang Z, Xiao B (2014) Communicating is crowdsourcing: Wi-Fi indoor localization with CSI-based speed estimation. J Comput Sci Technol 29(4):589–604

    Article  Google Scholar 

  63. Hu Y, Liao S, Lei Z, Yi D, Li S (2013) Exploring structural information and fusing multiple features for person re-identification. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 794–799

    Google Scholar 

  64. Zhang S, Zhu Y, Roy-Chowdhury AK (2016) Context-aware surveillance video summarization. IEEE Trans Image Process 25(11):5469–5478

    Google Scholar 

  65. Mirzaalian H, Lee TK, Hamarneh G (2016) Skin lesion tracking using structured graphical models. Med Image Anal 27:84–92

    Article  Google Scholar 

  66. Kang U, Hebert M, Park S (2013) Fast and scalable approximate spectral graph matching for correspondence problems. Inf Sci 220:306–318

    Article  MathSciNet  Google Scholar 

  67. Park S, Park SK, Hebert M (2013) Fast and scalable approximate spectral matching for higher order graph matching. IEEE Trans Pattern Anal Mach Intell 36(3):479–492

    Article  Google Scholar 

  68. Yang X, Liu CK, Liu ZY, Qiao H, Wang BF, Wang ZD (2014) A probabilistic spectral graph matching algorithm for robust correspondence between lunar surface images. In: Proceeding of the 11th world congress on intelligent control and automation. IEEE, pp 385–390

    Google Scholar 

  69. Brendel W, Todorovic S (2010) Segmentation as maximum-weight independent set. In: NIPS

    Google Scholar 

  70. Jain A, Gupta A, Rodriguez M, Davis L (2013) Representing videos using mid-level discriminative patches. In: Computer vision and pattern recognition, pp 2571–2578

    Google Scholar 

  71. Semenovich D (2010) Tensor power method for efficient map inference in higher-order MRFs. In: ICPR

    Google Scholar 

  72. Monroy A, Bell P, Ommer B (2014) Morphological analysis for investigating artistic images. Image Vis Comput 32(6)

    Google Scholar 

  73. He L, Yang X, Liu ZY (2018) Generalizing integer projected graph matching algorithm for outlier problem. In: 2018 IEEE 3rd international conference on image vision and computing (ICIVC). IEEE, pp 50–54

    Google Scholar 

  74. Lu Y, Huang K, Liu CL (2016) A fast projected fixed-point algorithm for large graph matching. Pattern Recognit 60:971–982

    Article  Google Scholar 

  75. Zaslavskiy M, Bach F, Vert JP (2008) A path following algorithm for graph matching. In: ICSP

    Google Scholar 

  76. Zaslavskiy M (2010) Graph matching and its application in computer vision and bioinformatics. PhD thesis, lEcole nationale superieure des mines de Paris

    Google Scholar 

  77. Besag J (1986) On the statistical analysis of dirty pictures. J R Stat Soc 48(5):259–302

    Google Scholar 

  78. Frank M, Wolfe P (1956) An algorithm for quadratic programming. Nav Res Logist Q 3(1–2):95–110

    Article  MathSciNet  Google Scholar 

  79. LeBlanc LJ, Helgason RV, Boyce DE (1985) Improved efficiency of the frank-wolfe algorithm for convex network programs. Transp Sci 19(4):445–462

    Article  MathSciNet  Google Scholar 

  80. Mirzaalian H, Lee T, Hamarneh G (2012) Uncertainty-based feature learning for skin lesion matching using a high order MRF optimization framework. In: Medical image computing and computer-assisted intervention, pp 98–105

    Google Scholar 

  81. Leordeanu M, Zanfir A, Sminchisescu C (2013) Locally affine sparse-to-dense matching for motion and occlusion estimation. In: ICCV

    Google Scholar 

  82. Semenovich D, Sowmya A (2010) Tensor power method for efficient map inference in higher-order MRFs. In: ICPR

    Google Scholar 

  83. Neudecker Magnus (1999) Matrix differential calculus with applications in statistics and econometrics. Wiley

    Google Scholar 

  84. Cour T, Shi J, Gogin N (2005) Learning spectral graph segmentation. In: International conference on artificial intelligence and statistics

    Google Scholar 

  85. Bach F, Jordan M (2003) Learning spectral clustering. In: NIPS

    Google Scholar 

  86. Baratchart L, Berthod M, Pottier L (1998) Optimization of positive generalized polynomials under lp constraints. J Convex Anal 5(2):133

    MathSciNet  MATH  Google Scholar 

  87. Kofidis E, Regallia PA (2002) On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J Matrix Anal Appl 23(3)

    Google Scholar 

  88. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape context. PAMI 24(4):509–522

    Article  Google Scholar 

  89. Kumar S (2005) Models for learning spatial interactions in natural images for context-based classification. PhD thesis, The Robotics Institute, Carnegie Mellon University

    Google Scholar 

  90. Leordeanu M, Hebert M (2009) Unsupervised learning for graph matching. In: CVPR

    Google Scholar 

  91. Guo M, Chou E, Huang DA, Song S, Yeung S, Fei-Fei L (2018) Neural graph matching networks for Fewshot 3d action recognition. In: Proceedings of the European conference on computer vision (ECCV), pp 653–669

    Google Scholar 

  92. Li Y, Gu C, Dullien T, Vinyals O, Kohli P (2019) Graph matching networks for learning the similarity of graph structured objects. arXiv:190412787

  93. Martineau M, Raveaux R, Conte D, Venturini G (2018) Learning error-correcting graph matching with a multiclass neural network. Pattern Recognit Lett

    Google Scholar 

  94. Wang R, Yan J, Yang X (2019) Neural graph matching network: learning Lawler’s quadratic assignment problem with extension to hypergraph and multiple-graph matching. arXiv:191111308

  95. Xu K, Wang L, Yu M, Feng Y, Song Y, Wang Z, Yu D (2019) Cross-lingual knowledge graph alignment via graph matching neural network. arXiv:190511605

  96. Nowak A, Villar S, Bandeira AS, Bruna J (2018) Revised note on learning quadratic assignment with graph neural networks. In: 2018 IEEE data science workshop (DSW). IEEE, pp 1–5

    Google Scholar 

  97. Wang W, Song H, Zhao S, Shen J, Zhao S, Hoi SC, Ling H (2019) Learning unsupervised video object segmentation through visual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3064–3074

    Google Scholar 

  98. Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) DeepFlow: large displacement optical flow with deep matching. In: ICCV

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Polytechnic University of Bucharest, Bucharest, Romania

    Marius Leordeanu

Authors
  1. Marius Leordeanu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marius Leordeanu .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Leordeanu, M. (2020). Unsupervised Learning of Graph and Hypergraph Matching. In: Unsupervised Learning in Space and Time. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-42128-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-42128-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-42127-4

  • Online ISBN: 978-3-030-42128-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation