Continuous Markov Random Fields for Robust Stereo Estimation

  • Koichiro Yamaguchi
  • Tamir Hazan
  • David McAllester
  • Raquel Urtasun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


In this paper we present a novel slanted-plane model which reasons jointly about occlusion boundaries as well as depth. We formulate the problem as one of inference in a hybrid MRF composed of both continuous (i.e., slanted 3D planes) and discrete (i.e., occlusion boundaries) random variables. This allows us to define potentials encoding the ownership of the pixels that compose the boundary between segments, as well as potentials encoding which junctions are physically possible. Our approach outperforms the state-of-the-art on Middlebury high resolution imagery [1] as well as in the more challenging KITTI dataset [2], while being more efficient than existing slanted plane MRF methods, taking on average 2 minutes to perform inference on high resolution imagery.


Belief Propagation Stereo Vision Stereo Match Quadratic Potential Autonomous Driving 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1,2,3) (2002); Microsoft Research Technical Report MSR-TR-2001-81 (November 2001)Google Scholar
  2. 2.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? In: CVPR (2012)Google Scholar
  3. 3.
    Tappen, M., Freeman, W.: Comparison of graph cuts with belief propagation for stereo, using identical mrf parameters. In: ICCV (2003)Google Scholar
  4. 4.
    Birchfield, S., Tomasi, C.: Multiway cut for stereo and motion with slanted surfaces. In: ICCV (1999)Google Scholar
  5. 5.
    Hong, L., Chen, G.: Segment-based stereo matching using graph cuts. In: CVPR (2004)Google Scholar
  6. 6.
    Bleyer, M., Gelautz, M.: A layered stereo matching algorithm using image segmentation and global visibility constraints. ISPRS Journal of Photogrammetry and Remote Sensing 59(3), 128–150 (2005)CrossRefGoogle Scholar
  7. 7.
    Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: ICPR 2006 (2006)Google Scholar
  8. 8.
    Deng, Y., Yang, Q., Lin, X., Tang, X.: A symmetric patch-based correspondence model for occlusion handling. In: ICCV (2005)Google Scholar
  9. 9.
    Yang, Q., Engels, C., Akbarzadeh, A.: Near real-time stereo for weakly-textured scenes. In: British Machine Vision Conference (2008)Google Scholar
  10. 10.
    Trinh, H., McAllester, D.: Unsupervised learning of stereo vision with monocular cues. In: Proc. BMVC, Citeseer (2009)Google Scholar
  11. 11.
    Zitnick, C., Kanade, T.: A cooperative algorithm for stereo matching and occlusion detection. PAMI 22(7) (July 2000)Google Scholar
  12. 12.
    Kolmogorov, V., Zabih, R.: Multi-camera Scene Reconstruction via Graph Cuts. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part III. LNCS, vol. 2352, pp. 82–96. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Bleyer, M., Rother, C., Kohli, P.: Surface stereo with soft segmentation. In: CVPR (2010)Google Scholar
  14. 14.
    Waltz, D.L.: Generating semantic description from drawings of scenes with shadows. MIT Artificial Intelligence Laboratory (1972) Workin Paper 47Google Scholar
  15. 15.
    Malik, J.: Interpreting line drawings of curved objects. International Journal of Computer Vision 1, 73–103 (1987)CrossRefGoogle Scholar
  16. 16.
    Hoiem, D., Stein, A.N., Efros, A.A., Hebert, M.: Recovering occlusion boundaries from a single image. In: ICCV (2007)Google Scholar
  17. 17.
    Ashutosh Saxena, J.S., Ng, A.Y.: Depth estimation using monocular and stereo cues. In: IJCAI (2007)Google Scholar
  18. 18.
    Maire, M., Arbelaez, P., Fowlkes, C., Malik, J.: Using contours to detect and localize junctions in natural images. In: CVPR (2008)Google Scholar
  19. 19.
    Peng, J., Hazan, T., McAllester, D., Urtasun, R.: Convex max-product algorithms for continuous mrfs with applications to protein folding. In: International Conference on Machine Learning, ICML (2011)Google Scholar
  20. 20.
    Hazan, T., Urtasun, R.: A primal-dual message-passing algorithm for approximated large scale structured prediction. Advances in Neural Information Processing Systems 23, 838–846 (2010)Google Scholar
  21. 21.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: IJCV (2002)Google Scholar
  22. 22.
    Konolige, K.: Small vision system. hardware and implementation. In: International Symposium on Robotics Research, pp. 111–116 (1997)Google Scholar
  23. 23.
    Kanade, T., Okutomi, M.: A stereo matching algorithm with an adaptive window: Theory and experiment. In: ICRA (1994)Google Scholar
  24. 24.
    Yoon, K.J., Member, S., Kweon, I.S.: Adaptive support-weight approach for correspondence search. PAMI 28, 650–656 (2006)CrossRefGoogle Scholar
  25. 25.
    Hirschmueller, H.: Stereo processing by semiglobal matching and mutual information. PAMI 30, 328–341 (2008)CrossRefGoogle Scholar
  26. 26.
    Cech, J., Sara, R.: Efficient sampling of disparity space for fast and accurate matching. In: BenCOS (2007)Google Scholar
  27. 27.
    Kostkova, J.: Stratified dense matching for stereopsis in complex scenes. In: BMVC (2003)Google Scholar
  28. 28.
    Wang, L., Jin, H., Yang, R.: Search Space Reduction for MRF Stereo. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 576–588. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  29. 29.
    Veksler, O.: Reducing search space for stereo correspondence with graph cuts. In: BMVC (2006)Google Scholar
  30. 30.
    Geiger, A., Roser, M., Urtasun, R.: Efficient Large-Scale Stereo Matching. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part I. LNCS, vol. 6492, pp. 25–38. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  31. 31.
    Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. PAMI 25(7) (2003)Google Scholar
  32. 32.
    Hazan, T., Shashua, A.: Norm-product belief propagation: Primal-dual message-passing for approximate inference. IEEE Transactions on Information Theory 56(12), 6294–6316 (2010)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Koller, D., Lerner, U., Angelov, D.: A general algorithm for approximate inference and its application to hybrid bayes nets. In: UAI (1999)Google Scholar
  34. 34.
    Ihler, A., McAllester, D.: Particle belief propagation. In: AISTATS 2009 (2009)Google Scholar
  35. 35.
    Hazan, T., Urtasun, R.: A primal-dual message-passing algorithm for approximated large scale structured prediction. In: NIPS (2010)Google Scholar
  36. 36.
    Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions using graph cuts. In: ICCV, pp. 508–515 (2001)Google Scholar
  37. 37.
    Bradski, G.: The opencv library. Dr. Dobb’s Journal of Software Tools (2000)Google Scholar
  38. 38.
    Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: CVPR (2011)Google Scholar
  39. 39.
    Cech, J., Sanchez-Riera, J., Horaud, R.P.: Scene flow estimation by growing correspondence seeds. In: CVPR (2011)Google Scholar
  40. 40.
    Ranftl, R., Gehrig, S., Pock, T., Bischof, H.: Pushing the Limits of Stereo Using Variational Stereo Estimation. In: IEEE Intelligent Vehicles Symposium (2012)Google Scholar
  41. 41.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. IJCV 70(1) (October 2006)Google Scholar
  42. 42.
    Weiss, Y., Freeman, W.: Correctness of belief propagation in gaussian graphical models of arbitrary topology. Neural Computation 13(10) (2001)Google Scholar
  43. 43.
    Schwing, A., Hazan, T., Pollefeys, M., Urtasun, R.: Distributed message passing for large scale graphical models. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2011)Google Scholar
  44. 44.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: ECCV (1994)Google Scholar
  45. 45.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. In: PAMI (2011)Google Scholar
  46. 46.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels. Technical Report 149300 EPFL (June 2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Koichiro Yamaguchi
    • 1
    • 2
  • Tamir Hazan
    • 1
  • David McAllester
    • 1
  • Raquel Urtasun
    • 1
  1. 1.TTI ChicagoUSA
  2. 2.Toyota Central R&D Labs., Inc.Japan

Personalised recommendations