Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering
Conference paper
Abstract
This paper presents a new efficient algorithm for computing temporally consistent disparity maps from video footage. Our method is motivated by recent work [1] that achieves high quality stereo results by smoothing disparity costs with a fast edge-preserving filter. This previous approach was designed to work with single static image pairs and does not maintain temporal coherency of disparity maps when applied to video streams.
The main contribution of our work is to transfer this concept to the spatio-temporal domain in order to efficiently achieve temporally consistent disparity maps, where disparity changes are aligned with spatio-temporal edges of the video sequence. We further show that our method can be used as spatio-temporal regularizer for optical flow estimation. Our approach can be implemented efficiently, achieving real-time results for stereo matching. Quantitative and qualitative results demonstrate that our approach (i) considerably improves over frame-by-frame methods for both stereo and optical flow; and (ii) outperforms the state-of-the-art for local space-time stereo approaches.
Keywords
Stereo Match Optical Flow Estimation Video Volume Average Angular Error Cost Volume
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Download
to read the full conference paper text
References
- 1.Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast Cost-Volume Filtering for Visual Correspondence and Beyond. In: CVPR (2011)Google Scholar
- 2.Yoon, K.J., Kweon, I.S.: Locally Adaptive Support-Weight Approach for Visual Correspondence Search. In: CVPR (2005)Google Scholar
- 3.Hosni, A., Bleyer, M., Gelautz, M., Rhemann, C.: Local Stereo Matching Using Geodesic Support Weights. In: ICIP (2009)Google Scholar
- 4.Hosni, A., Bleyer, M., Gelautz, M.: Near Real-Time Stereo With Adaptive Support Weight Approaches. In: 3DPVT (2010)Google Scholar
- 5.He, K., Sun, J., Tang, X.: Guided Image Filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010)CrossRefGoogle Scholar
- 6.Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.A.: Real-time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid. In: Daniilidis, K. (ed.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 510–523. Springer, Heidelberg (2010)Google Scholar
- 7.Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1/2/3), 7–42 (2002), http://www.middlebury.edu/stereo/ CrossRefMATHGoogle Scholar
- 8.Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-Dimensional Scene Flow. In: ICCV (1999)Google Scholar
- 9.Leung, C., Appleton, B., Lovell, B.C., Sun, C.: An Energy Minimisation Approach to Stereo-Temporal Dense Reconstruction. In: ICPR (2004)Google Scholar
- 10.Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR (2010)Google Scholar
- 11.Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: CVPR (2010)Google Scholar
- 12.Salgado, A., Sánchez, J.: Temporal Constraints in Large Optical Flow Estimation. In: Moreno Díaz, R., Pichler, F., Quesada Arencibia, A. (eds.) EUROCAST 2007. LNCS, vol. 4739, pp. 709–716. Springer, Heidelberg (2007)CrossRefGoogle Scholar
- 13.Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High Accuracy Optical Flow Estimation Based on a Theory for Warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
- 14.Weickert, J., Schnörr, C.: Variational optic flow computation with a spatio-temporal smoothness constraint. In: JMIV, vol. 14, pp. 245–255 (2001)Google Scholar
- 15.Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: CVPR (1991)Google Scholar
- 16.Nagel, H.H.: Extending the ’Oriented Smoothness Constraint’ into the Temporal Domain and the Estimation of Derivatives of Optical Flow. In: Faugeras, O. (ed.) ECCV 1990. LNCS, vol. 427, pp. 139–148. Springer, Heidelberg (1990)CrossRefGoogle Scholar
- 17.Zhang, L., Curless, B., Seitz, S.M.: Spacetime Stereo: Shape Recovery for Dynamic Scenes. In: CVPR (2003)Google Scholar
- 18.Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high-resolution capture for modeling and animation. In: SIGGRAPH (2004)Google Scholar
- 19.Davis, J., Nehab, D., Ramamoorthi, R., Rusinkiewicz, S.: Spacetime stereo: a unifying framework for depth from triangulation. In: PAMI, vol. 27(2), pp. 296–302 (2005)Google Scholar
- 20.Jenkin, M., Tsotsos, J.: Applying temporal constraints to the dynamic stereo problem. In: CVGIP, vol. 33, pp. 16–32 (1986)Google Scholar
- 21.Williams, O., Isard, M., MacCormick, J.: Estimating Disparity and Occlusions in Stereo Video Sequences. In: CVPR (2005)Google Scholar
- 22.Larsen, E.S., Mordohai, P., Pollefeys, M., Fuchs, H.: Temporally consistent reconstruction from multiple video streams using enhanced belief propagation. In: ICCV (2007)Google Scholar
- 23.Bleyer, M., Gelatuz, M.: Temporally Consistent Disparity Maps from Uncalibrated Stereo Videos. In: ISPA (2009)Google Scholar
- 24.Zimmer, H., Bruhn, A., Weickert, J.: Optic Flow in Harmony. In: IJCV, vol. 93, pp. 368 – 388 (2011)Google Scholar
- 25.Sizintsev, M., Wildes, R.P.: Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching. In: CVPR (2009)Google Scholar
- 26.Steinbrücker, F., Pock, T., Cremers, D.: Large displacement optical flow computation without warping. In: ICCV (2009)Google Scholar
Copyright information
© Springer-Verlag Berlin Heidelberg 2011