Advertisement

SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation

  • René SchusterEmail author
  • Oliver Wasenmüller
  • Christian Unger
  • Georg Kuschk
  • Didier Stricker
Article
  • 5 Downloads

Abstract

State-of-the-art scene flow algorithms pursue the conflicting targets of accuracy, run time, and robustness. With the successful concept of pixel-wise matching and sparse-to-dense interpolation, we shift the operating point in this field of conflicts towards universality and speed. Avoiding strong assumptions on the domain or the problem yields a more robust algorithm. This algorithm is fast because we avoid explicit regularization during matching, which allows an efficient computation. Using image information from multiple time steps and explicit visibility prediction based on previous results, we achieve competitive performances on different data sets. Our contributions and results are evaluated in comparative experiments. Overall, we present an accurate scene flow algorithm that is faster and more generic than any individual benchmark leader.

Keywords

Scene flow Matching Occlusions Interpolation 

Notes

References

  1. Bailer, C., Taetz, B., & Stricker, D. (2015). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In International conference on computer vision (ICCV).Google Scholar
  2. Bailer, C., Taetz, B., & Stricker, D. (2019). Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 41, 1879–1892.CrossRefGoogle Scholar
  3. Basha, T., Moses, Y., & Kiryati, N. (2013). Multi-view scene flow estimation: A view centered variational approach. International Journal of Computer Vision (IJCV), 101, 6–21.MathSciNetCrossRefGoogle Scholar
  4. Behl, A., Jafari, O. H., Mustikovela, S. K., Alhaija, H. A., Rother, C., & Geiger, A. (2017). Bounding boxes, segmentations and object coordinates: How important is recognition for 3D scene flow estimation in autonomous driving scenarios? In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  5. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision (ECCV).Google Scholar
  6. Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012). A naturalistic open source movie for optical flow evaluation. In European conference on computer vision (ECCV).Google Scholar
  7. Chen, Q., & Koltun, V. (2016). Full flow: Optical flow estimation by global optimization over regular grids. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  8. Dollár, P., & Zitnick, C. L. (2013). Structured forests for fast edge detection. In International conference on computer vision (ICCV).Google Scholar
  9. Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  10. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  11. He, K., & Sun, J. (2012). Computing nearest-neighbor fields via propagation-assisted kD-trees. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  12. Herbst, E., Ren, X., & Fox, D. (2013). RGB-D flow: Dense 3-D motion estimation using color and depth. In International conference on robotics and automation (ICRA).Google Scholar
  13. Hirschmüller, H. (2008). Stereo processing by semiglobal matching and mutual information. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 30, 328–341.CrossRefGoogle Scholar
  14. Hu, Y., Li, Y., & Song, R. (2017). Robust interpolation of correspondences for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  15. Hu, Y., Song, R., & Li, Y. (2016). Efficient coarse-to-fine patchmatch for large displacement optical flow. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  16. Huguet, F., & Devernay, F. (2007). A variational method for scene flow estimation from stereo sequences. In International conference on computer vision (ICCV).Google Scholar
  17. Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., & Cremers, D. (2015). A primal-dual framework for real-time dense RGB-D scene flow. In International conference on robotics and automation (ICRA).Google Scholar
  18. Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. Transactions on Pattern Analysis and Machine Intelligence (PAMI), 33, 978–994.CrossRefGoogle Scholar
  19. Lv, Z., Beall, C., Alcantarilla, P. F., Li, F., Kira, Z., & Dellaert, F. (2016). A continuous optimization approach for efficient and accurate scene flow. In European conference on computer vision (ECCV).Google Scholar
  20. Lv, F., Lian, Q., Yang, G., Lin, G., Jialin Pan, S., & Duan, L. (2018). Domain adaptive semantic segmentation through structure enhancement. In European conference on computer vision (ECCV).Google Scholar
  21. Ma, W. C., Wang, S., Hu, R., Xiong, Y., & Urtasun, R. (2019). Deep Rigid Instance Scene Flow. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  22. Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In International conference on computer vision (ICCV).Google Scholar
  23. Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016). A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  24. Menze, M., & Geiger, A. (2015). Object scene flow for autonomous vehicles. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  25. Menze, M., Heipke, C., & Geiger, A. (2018). Object scene flow. Journal of Photogrammetry and Remote Sensing (JPRS), 140, 60–76.CrossRefGoogle Scholar
  26. Neoral, M., & Šochman, J. (2017). Object scene flow with temporal consistency. In Computer vision winter workshop (CVWW).Google Scholar
  27. Ošep, A., Hermans, A., Engelmann, F., Klostermann, D., Mathias, M., & Leibe, B. (2016). Multi-scale object candidates for generic object tracking in street scenes. In International conference on robotics and automation (ICRA).Google Scholar
  28. Ren, Z., Sun, D., Kautz, J., & Sudderth, E. B. (2017). Cascaded scene flow prediction using semantic segmentation. In International conference on 3D vision (3DV).Google Scholar
  29. Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015). EpicFlow: edge-preserving interpolation of correspondences for optical flow. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  30. Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vazquez, D., & Lopez, A. M. (2015). Vision-based offline–online perception paradigm for autonomous driving. In Winter conference on applications of computer vision (WACV).Google Scholar
  31. Saxena, R., Schuster, R., Wasenmüller O, & Stricker D (2019) PWOC-3D: Deep occlusion-aware end-to-end scene flow estimation. In Intelligent vehicles symposium (IV).Google Scholar
  32. Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018a) Combining stereo disparity and optical flow for basic scene flow. In Commercial vehicle technology symposium (CVT).Google Scholar
  33. Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018b). FlowFields++: Accurate optical flow correspondences meet robust interpolation. In International conference on image processing (ICIP).Google Scholar
  34. Schuster, R., Wasenmüller, O., Kuschk, G., Bailer, C., & Stricker, D. (2018c). SceneFlowFields: Dense interpolation of sparse scene flow correspondences. In Winter conference on applications of computer vision (WACV).Google Scholar
  35. Taniai, T., Sinha, S. N., & Sato, Y. (2017). Fast multi-frame stereo scene flow with motion segmentation. In Conference on computer vision and pattern recognition (CVPR).Google Scholar
  36. Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (1999). Three-dimensional scene flow. In International conference on computer vision (ICCV).Google Scholar
  37. Vogel, C., Roth, S., & Schindler, K. (2014). View-consistent 3d scene flow estimation over multiple frames. In European conference on computer vision (ECCV).Google Scholar
  38. Vogel, C., Schindler, K., & Roth, S. (2013). Piecewise rigid scene flow. In International conference on computer vision (ICCV).Google Scholar
  39. Vogel, C., Schindler, K., & Roth, S. (2015). 3D scene flow estimation with a piecewise rigid scene model. International Journal of Computer Vision (IJCV), 115, 1–28.MathSciNetCrossRefGoogle Scholar
  40. Wang, M., & Deng, W. (2018). Deep visual domain adaptation: A survey. Neurocomputing, 312, 135–153.CrossRefGoogle Scholar
  41. Wannenwetsch, A. S., Keuper, M., & Roth, S. (2017). ProbFlow: Joint optical flow and uncertainty estimation. In International conference on computer vision (ICCV).Google Scholar
  42. Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers. D. (2008). Efficient dense scene flow from sparse or dense stereo data. In European conference on computer vision (ECCV).Google Scholar
  43. Xu, P., Davoine, F., Bordes, J. B., Zhao, H., & Denœux, T. (2016). Multimodal information fusion for urban scene understanding. Machine Vision and Applications (MVA), 27, 331–349.CrossRefGoogle Scholar
  44. Yamaguchi, K., McAllester, D., & Urtasun, R. (2014). Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In European conference on computer vision (ECCV).Google Scholar
  45. Yoshida, T., Wasenmüller, O., & Stricker, D. (2017). Time-of-flight sensor depth enhancement for automotive exhaust gas. In International conference on image processing (ICIP).Google Scholar
  46. Zweig, S., & Wolf, L. (2017). InterpoNet, a brain inspired neural network for optical flow dense interpolation. In Conference on computer vision and pattern recognition (CVPR).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.DFKI - German Research Center for Artificial IntelligenceKaiserslauternGermany
  2. 2.BMW GroupMunichGermany

Personalised recommendations