Finding Correspondence from Multiple Images via Sparse and Low-Rank Decomposition

  • Zinan Zeng
  • Tsung-Han Chan
  • Kui Jia
  • Dong Xu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


We investigate the problem of finding the correspondence from multiple images, which is a challenging combinatorial problem. In this work, we propose a robust solution by exploiting the priors that the rank of the ordered patterns from a set of linearly correlated images should be lower than that of the disordered patterns, and the errors among the reordered patterns are sparse. This problem is equivalent to find a set of optimal partial permutation matrices for the disordered patterns such that the rearranged patterns can be factorized as a sum of a low rank matrix and a sparse error matrix. A scalable algorithm is proposed to approximate the solution by solving two sub-problems sequentially: minimization of the sum of nuclear norm and l 1 norm for solving relaxed partial permutation matrices, followed by a binary integer programming to project each relaxed partial permutation matrix to the feasible solution. We verify the efficacy and robustness of the proposed method with extensive experiments with both images and videos.


Feature correspondence partial permutation low rank and sparse matrix decomposition 


  1. 1.
    Berg, A.C., Berg, T.L., Malik, J.: Shape matching and object recognition using low distortion correspondences. In: CVPR (2005)Google Scholar
  2. 2.
    Pollefeys, M., et al.: Detailed real-time urban 3d reconstruction from video. IJCV 78, 143–167 (2008)CrossRefGoogle Scholar
  3. 3.
    Enqvist, O., Josephson, K., Kahl, F.: Optimal correspondences from pairwise constraints. In: ICCV (2009)Google Scholar
  4. 4.
    Caetano, T.S., Caelli, T., Schuurmans, D., Barone, D.A.C.: Graphical models and point pattern matching. TPAMI 28, 1646–1663 (2006)CrossRefGoogle Scholar
  5. 5.
    Maciel, J., Costeira, J.P.: A global solution to sparse correspondence problems. TPAMI 25, 187–199 (2003)CrossRefGoogle Scholar
  6. 6.
    Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusion via graph cuts. In: ICCV (2001)Google Scholar
  7. 7.
    Liu, H., Yan, S.: Common visual pattern discovery via spatially coherent correspondences. In: CVPR (2010)Google Scholar
  8. 8.
    Torresesani, L., Kolmogorov, V., Rother, C.: Feature Correspondence Via Graph Matching: Models and Global Optimization. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 596–609. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  9. 9.
    Caetano, T.S., McAuley, J., Cheng, L., Le, Q.V., Smola, A.J.: Learning graph matching. TPAMI 31, 1048–1058 (2009)CrossRefGoogle Scholar
  10. 10.
    Li, H.S., Huang, J.Z., Zhang, S.T., Huang, X.L.: Optimal object matching via convexification and composition. In: ICCV (2011)Google Scholar
  11. 11.
    Jiang, H., Tian, T.P., Sclaroff, S.: Scale and rotation invariant matching using linearly augmented trees. In: CVPR (2011)Google Scholar
  12. 12.
    Jiang, H., Drew, M.S., Li, Z.N.: Matching by linear programming and successive convexification. TPAMI 29, 959–975 (2007)CrossRefGoogle Scholar
  13. 13.
    Cho, M., Lee, J., Lee, K.: Feature correspondence and deformable object matching via agglomerative correspondence clustering. In: ICCV (2009)Google Scholar
  14. 14.
    Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The Generalized PatchMatch Correspondence Algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Oliveira, R., Costeira, J., Xavier, J.: Optimal point correspondence throught the use of rank constraints. In: CVPR (2005)Google Scholar
  16. 16.
    Torki, M., Elgammal, A.: One-shot multi-set non-rigid feature-spatial matching. In: CVPR (2010)Google Scholar
  17. 17.
    Poelman, C.J., Kanade, T.: A paraperspective factorization method for shape and motion recovery. In: ECCV (1994)Google Scholar
  18. 18.
    Tomasi, C., Kanade, T.: Shape from motion from image streams under orthography: A factorization method. IJCV 9, 137–154 (1992)CrossRefGoogle Scholar
  19. 19.
    Candes, E., Li, X., Ma, Y., Wright, J.: Robust principle component analysis? Journal of ACM 58, 1–37 (2009)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Peng, Y.G., Ganesh, A.: Rasl: Robust alignment by sparse and low-rank decomposition for linearly correlated images. In: CVPR (2010)Google Scholar
  21. 21.
    Lin, Z.C., Chen, M.M., Ma, Y.: The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices. UIUC technical report UILU-ENG-09-2215 (2009)Google Scholar
  22. 22.
    Toh, K.C., Yun, S.W.: An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of Optimization, 615–640 (2010)Google Scholar
  23. 23.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Ecjstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3, 1–22 (2010)zbMATHCrossRefGoogle Scholar
  24. 24.
    Wolf, L., Hassne, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: CVPR (2011)Google Scholar
  25. 25.
    Everingham, M., Sivic, J., Zisserman, A.: “who are you?” - learning person specific classifiers from video. In: BMVC (2006)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91–110 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Zinan Zeng
    • 1
    • 2
  • Tsung-Han Chan
    • 2
  • Kui Jia
    • 2
  • Dong Xu
    • 1
  1. 1.School of Computer EngineeringNanyang Technological UniversitySingapore
  2. 2.Advanced Digital Sciences CenterSingapore

Personalised recommendations