FALDOI: A New Minimization Strategy for Large Displacement Variational Optical Flow

  • Roberto P. Palomares
  • Enric Meinhardt-Llopis
  • Coloma Ballester
  • Gloria Haro
Article

Abstract

We propose a large displacement optical flow method that introduces a new strategy to compute a good local minimum of any optical flow energy functional. The method requires a given set of discrete matches, which can be extremely sparse, and an energy functional which locally guides the interpolation from those matches. In particular, the matches are used to guide a structured coordinate descent of the energy functional around these keypoints. It results in a two-step minimization method at the finest scale which is very robust to the inevitable outliers of the sparse matcher and able to capture large displacements of small objects. Its benefits over other variational methods that also rely on a set of sparse matches are its robustness against very few matches, high levels of noise, and outliers. We validate our proposal using several optical flow variational models. The results consistently outperform the coarse-to-fine approaches and achieve good qualitative and quantitative performance on the standard optical flow benchmarks.

Keywords

Optical flow Variational methods Coordinate descent Sparse matches 

Notes

Acknowledgements

The authors would like to thank Gabriele Facciolo for all the discussions with him about the subject of this paper.

References

  1. 1.
    Alvarez, L., Deriche, R., Papadopoulo, T., Sánchez, J.: Symmetrical dense optical flow estimation with occlusions detection. In: European Conference on Computer Vision, pp. 721–735 (2002)Google Scholar
  2. 2.
    Alvarez, L., Weickert, J., Sánchez, J.: Reliable estimation of dense optical flow fields with large displacements. Int. J. Comput. Vis. 39(1), 41–56 (2000)CrossRefMATHGoogle Scholar
  3. 3.
    Ayvaci, A., Raptis, M., Soatto, S.: Sparse occlusion detection with optical flow. Int. J. Comput. Vis. 97(3), 322–338 (2012)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Bailer, C., Taetz, B., Stricker, D.: Flow fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4015–4023 (2015)Google Scholar
  5. 5.
    Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)CrossRefGoogle Scholar
  6. 6.
    Ballester, C., Garrido, L., Lazcano, V., Caselles, V.: A TV-L1 Optical Flow Method with Occlusion Detection. Lectures Notes in Computer Science, vol. 7476, p. 3140 (2012)Google Scholar
  7. 7.
    Bao, L., Yang, Q., Jin, H.: Fast edge-preserving patchmatch for large displacement optical flow. IEEE Trans. Image Process. 23(12), 4996–5006 (2014)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRefGoogle Scholar
  9. 9.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. Eur. Conf. Comput. Vis. 3024, 25–36 (2004)MATHGoogle Scholar
  10. 10.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)CrossRefGoogle Scholar
  11. 11.
    Buades, A., Facciolo, G.: Reliable multi-scale and multi-window stereo matching. SIAM J. Imaging Sci. 8(2), 888–915 (2015)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European Conference on Computer Vision, pp. 611–625 (2012)Google Scholar
  13. 13.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Chen, Z., Jin, H., Lin, Z., Cohen, S., Wu, Y.: Large displacement optical flow from nearest neighbor fields. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2443–2450 (2013)Google Scholar
  15. 15.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)Google Scholar
  16. 16.
    Demetz, O., Hafner, D., Weickert, J.: Morphologically invariant matching of structures with the complete rank transform. Int. J. Comput. Vis. 113(3), 220–232 (2015)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Dollár, P., Zitnick, C.L.: Structured forests for fast edge detection. In: IEEE International Conference on Computer Vision (ICCV), pp. 1841–1848 (2013)Google Scholar
  18. 18.
    Fortun, D., Bouthemy, P., Kervrann, C.: Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow. Comput. Vis. Image Underst. 145, 81–94 (2016)CrossRefMATHGoogle Scholar
  19. 19.
    Friedman, J., Hastie, T., Hfling, H., Tibshirani, R.: Pathwise coordinate optimization. Technical Report, Annals of Applied Statistics (2007)Google Scholar
  20. 20.
    Gadot, D., Wolf, L.: Patchbatch: a batch augmented loss for optical flow. CoRR abs/1512.01815 (2015). http://arxiv.org/abs/1512.01815
  21. 21.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  22. 22.
    Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Hafner, D., Demetz, O., Weickert, J.: Why is the census transform good for robust optic flow computation? In: Scale-Space and Variational Methods in Computer Vision, pp. 210–221 (2013)Google Scholar
  24. 24.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif Intell 17, 185–203 (1981)CrossRefGoogle Scholar
  25. 25.
    Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  26. 26.
    Kannala, J., Brandt, S.: Quasi-dense wide baseline matching using match propagation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  27. 27.
    Kennedy, R., Taylor, C.J.: Optical flow with geometric occlusion estimation and fusion of multiple frames. In: Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 364–377 (2015)Google Scholar
  28. 28.
    Krähenbühl, P., Koltun, V.: Efficient nonlocal regularization for optical flow. In: European Conference on Computer Vision (ECCV), pp. 356–369 (2012)Google Scholar
  29. 29.
    Leordeanu, M., Sukthankar, R., Sminchisescu, C.: Efficient closed-form solution to generalized boundary detection. In: European Conference on Computer Vision (ECCV), pp. 516–529 (2012)Google Scholar
  30. 30.
    Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: IEEE International Conference on Computer Vision, pp. 1721–1728 (2013)Google Scholar
  31. 31.
    Lhuillier, M., Quan, L.: Match propagation for image-based modeling and rendering. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1140–1146 (2002)CrossRefGoogle Scholar
  32. 32.
    Li, Y., Osher, S.: A new median formula with applications to PDE based denoising. Commun. Math. Sci. 7(3), 741–753 (2009). http://projecteuclid.org/euclid.cms/1256562821
  33. 33.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision (1999)Google Scholar
  34. 34.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  35. 35.
    Menze, M., Heipke, C., Geiger, A.: Discrete optimization for optical flow. In: Pattern Recognition, pp. 16–28 (2015)Google Scholar
  36. 36.
    Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Kadir, S.F., Van Goo, T.: An algorithm for total variation minimization and applications. Int. J. Comput. Vis. 65(1–2), 43–72 (2005)CrossRefGoogle Scholar
  37. 37.
    Müller, T., Rabe, C., Rannacher, J., Franke, U., Mester, R.: Illumination-robust dense optical flow using census signatures. In: Proceedings of the DAGM Conference on Pattern Recognition, pp. 236–245 (2011)Google Scholar
  38. 38.
    Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8(5), 565–593 (1986)CrossRefGoogle Scholar
  39. 39.
    Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vis. 67(2), 141–158 (2006)CrossRefGoogle Scholar
  41. 41.
    Palomares, R.P., Haro, G., Ballester, C.: A rotation-invariant regularization term for optical flow related problems. In: Computer Vision–ACCV, pp. 304–319. Springer (2014)Google Scholar
  42. 42.
    Ranftl, R., Bredies, K., Pock, T.: Non-local total generalized variation for optical flow estimation. Comput. Vis. ECCV 2014, 439–454 (2014)Google Scholar
  43. 43.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In: Computer Vision and Pattern Recognition (2015)Google Scholar
  44. 44.
    Rey Otero, I., Delbracio, M.: Anatomy of the SIFT Method. Image Process On Line 4, 370–396 (2014). doi: 10.5201/ipol.2014.82 CrossRefGoogle Scholar
  45. 45.
    Sánchez, J., Salgado, A., Monzón, N.: Preserving accurate motion contours with reliable parameter selection. In: IEEE International Conference on Image Processing (ICIP), pp. 209–213 (2014)Google Scholar
  46. 46.
    Stein, F.: Efficient computation of optical flow using the census transform. In: DGAM, pp. 79–86 (2004)Google Scholar
  47. 47.
    Steinbrücker, F., Pock, T., Cremers, D.: Advanced data terms for variational optic flow estimation. In: Proceedings of Vision, Modeling and Visualization (2009)Google Scholar
  48. 48.
    Steinbrücker, F., Pock, T., Cremers, D.: Large displacement optical flow computation without warping. In: International Conference on Computer Vision, pp. 1609–1614 (2009)Google Scholar
  49. 49.
    Stoll, M., Volz, S., Bruhn, A.: Adaptive integration of feature matches into variational optical flow methods. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) Computer Vision-ACCV 2012: 11th Asian Conference on Computer Vision, Part III, Revised Selected Papers, pp. 1–14 (2013)Google Scholar
  50. 50.
    Strekalovskiy, E., Chambolle, A., Cremers, D.: Convex relaxation of vectorial problems with coupled regularization. SIAM J. Imaging Sci. 7(1), 294–336 (2014)Google Scholar
  51. 51.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2432–2439 (2010)Google Scholar
  52. 52.
    Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2014)CrossRefGoogle Scholar
  53. 53.
    Timofte, R., Van Gool, L.: Sparse flow: Sparse matching for small to large displacement optical flow. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1100–1106 (2015)Google Scholar
  54. 54.
    Vogel, C., Roth, S., Schindler, K.: An evaluation of data costs for optical flow. In: Pattern recognition, pp. 343–353 (2013)Google Scholar
  55. 55.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: International Conference on Computer Vision (2013)Google Scholar
  56. 56.
    Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2464–2471 (2010)Google Scholar
  57. 57.
    Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic huber-l1 optical flow. In: BMVC, vol. 1, no. 2, p. 3 (2009)Google Scholar
  58. 58.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012)CrossRefGoogle Scholar
  59. 59.
    Yang, J., Li, H.: Dense, accurate optical flow estimation with piecewise parametric model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1019–1027 (2015)Google Scholar
  60. 60.
    Yoon, K.J., Kweon, I.S.: Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 650–656 (2006)CrossRefGoogle Scholar
  61. 61.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: European Conference on Computer Vision (ECCV), pp. 151–158 (1994)Google Scholar
  62. 62.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: Proceedings of the 29th DAGM Conference on Pattern Recognition, pp. 214–223 (2007)Google Scholar
  63. 63.
    Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93(3), 368–388 (2011)MathSciNetCrossRefMATHGoogle Scholar
  64. 64.
    Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., Seidel, H.P.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) Energy Minimization Methods in Computer Vision and Pattern Recognition, pp. 207–220. Springer Berlin Heidelberg (2009)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Roberto P. Palomares
    • 1
  • Enric Meinhardt-Llopis
    • 2
  • Coloma Ballester
    • 1
  • Gloria Haro
    • 1
  1. 1.Universitat Pompeu FabraBarcelonaSpain
  2. 2.École Normale Supérieure Paris-SaclayCachanFrance

Personalised recommendations