As-Rigid-As-Possible Stereo under Second Order Smoothness Priors

  • Chi Zhang
  • Zhiwei Li
  • Rui Cai
  • Hongyang Chao
  • Yong Rui
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)


Imposing smoothness priors is a key idea of the top-ranked global stereo models. Recent progresses demonstrated the power of second order priors which are usually defined by either explicitly considering three-pixel neighborhoods, or implicitly using a so-called 3D-label for each pixel. In contrast to the traditional first-order priors which only prefer fronto-parallel surfaces, second-order priors encourage arbitrary collinear structures. However, we still can find defective regions in matching results even under such powerful priors, e.g., large textureless regions. One reason is that most of the stereo models are non-convex, where pixel-wise smoothness priors, i.e., local constraints, are too flexible to prevent the solution from trapping in bad local minimums. On the other hand, long-range spatial constraints, especially the segment-based priors, have advantages on this problem. However, segment-based priors are too rigid to handle curved surfaces. We present a mixture model to combine the benefits of these two kinds of priors, whose energy function consists of two terms 1) a Laplacian operator on the disparity map which imposes pixel-wise second-order smoothness; 2) a segment-wise matching cost as a function of quadratic surface, which encourages “as-rigid-as-possible” smoothness. To effectively solve the problem, we introduce an intermediate term to decouple the two subenergies, which enables an alternated optimization algorithm that is about an order of magnitude faster than PatchMatch [1]. Our approach is one of the top ranked models on the Middlebury benchmark at sub-pixel accuracy.


  1. 1.
    Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo - stereo matching with slanted support windows. In: BMVC, pp. 1–11 (2011)Google Scholar
  2. 2.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47(1-3), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  3. 3.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: DAGM-Symposium, pp. 214–223 (2007)Google Scholar
  4. 4.
    Ranftl, R., Gehrig, S., Pock, T., Bischof, H.: Pushing the limits of stereo using variational stereo estimation. In: Intelligent Vehicles Symposium, pp. 401–407 (2012)Google Scholar
  5. 5.
    Heise, P., Klose, S., Jensen, B., Knoll, A.: Pm-huber: Patchmatch with huber regularization for stereo matching. In: ICCV, pp. 2360–2367 (2013)Google Scholar
  6. 6.
    Klaus, A., Sormann, M., Karner, K.F.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: ICPR (3), pp. 15–18 (2006)Google Scholar
  7. 7.
    Hong, L., Chen, G.: Segment-based stereo matching using graph cuts. In: CVPR (1), pp. 74–81 (2004)Google Scholar
  8. 8.
    Tao, H., Sawhney, H.S., Kumar, R.: A global matching framework for stereo computation. In: ICCV, pp. 532–539 (2001)Google Scholar
  9. 9.
    Wang, Z.F., Zheng, Z.G.: A region based stereo matching algorithm using cooperative optimization. In: CVPR (2008)Google Scholar
  10. 10.
    Grimson, E.: From images to surfaces: a computational study of the human early visual system. MIT Press (1981)Google Scholar
  11. 11.
    Terzopoulos, D.: Multilevel computational processes for visual surface reconstruction. Computer Vision, Graphics, and Image Processing 24(1), 52–96 (1983)CrossRefGoogle Scholar
  12. 12.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press (1987)Google Scholar
  13. 13.
    Ishikawa, H., Geiger, D.: Rethinking the prior model for stereo. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 526–537. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Woodford, O.J., Torr, P.H.S., Reid, I.D., Fitzgibbon, A.W.: Global stereo reconstruction under second order smoothness priors. In: CVPR (2008)Google Scholar
  15. 15.
    Rother, C., Kolmogorov, V., Lempitsky, V.S., Szummer, M.: Optimizing binary mrfs via extended roof duality. In: CVPR (2007)Google Scholar
  16. 16.
    Olsson, C., Ulén, J., Boykov, Y.: In defense of 3d-label stereo. In: CVPR, pp. 1730–1737 (2013)Google Scholar
  17. 17.
    Besse, F., Rother, C., Fitzgibbon, A., Kautz, J.: Pmbp: Patchmatch belief propagation for correspondence field estimation. International Journal of Computer Vision, 1–12 (2012)Google Scholar
  18. 18.
    Forsyth, D.A., Ponce, J.: Computer Vision: A Modern Approach. Prentice Hall Professional Technical Reference (2002)Google Scholar
  19. 19.
    Yamaguchi, K., McAllester, D.A., Urtasun, R.: Robust monocular epipolar flow estimation. In: CVPR, pp. 1862–1869 (2013)Google Scholar
  20. 20.
    Zhou, X., Boulanger, P.: New eye contact correction using radial basis function for wide baseline videoconference system. In: Lin, W., Xu, D., Ho, A., Wu, J., He, Y., Cai, J., Kankanhalli, M., Sun, M.-T. (eds.) PCM 2012. LNCS, vol. 7674, pp. 68–79. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  21. 21.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  22. 22.
    Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)CrossRefGoogle Scholar
  23. 23.
    Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: CVPR (1), pp. 195–202 (2003)Google Scholar
  24. 24.
    Steinbrücker, F., Pock, T., Cremers, D.: Large displacement optical flow computation withoutwarping. In: ICCV, pp. 1609–1614 (2009)Google Scholar
  25. 25.
    Chen, Y., Davis, T.A., Hager, W.W., Rajamanickam, S.: Algorithm 887: Cholmod, supernodal sparse cholesky factorization and update/downdate. ACM Trans. Math. Softw. 35(3) (2008)Google Scholar
  26. 26.
    Nelder, J., Mead, R.: A simplex method for function minimization (1965)Google Scholar
  27. 27.
    OpenMP Architecture Review Board: OpenMP application program interface version 3.0 (May 2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Chi Zhang
    • 1
  • Zhiwei Li
    • 2
  • Rui Cai
    • 2
  • Hongyang Chao
    • 1
  • Yong Rui
    • 2
  1. 1.Sun Yat-Sen UniversityChina
  2. 2.Microsoft Research AsiaChina

Personalised recommendations