Highly Overparameterized Optical Flow Using PatchMatch Belief Propagation

  • Michael Hornáček
  • Frederic Besse
  • Jan Kautz
  • Andrew Fitzgibbon
  • Carsten Rother
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)


Motion in the image plane is ultimately a function of 3D motion in space. We propose to compute optical flow using what is ostensibly an extreme overparameterization: depth, surface normal, and frame-to-frame 3D rigid body motion at every pixel, giving a total of 9 DoF. The advantages of such an overparameterization are twofold: first, geometrically meaningful reasoning can be called upon in the optimization, reflecting possible 3D motion in the underlying scene; second, the ‘fronto-parallel’ assumption implicit in the use of traditional matching pixel windows is ameliorated because the parameterization determines a plane-induced homography at every pixel. We show that optimization over this high-dimensional, continuous state space can be carried out using an adaptation of the recently introduced PatchMatch Belief Propagation (PMBP) energy minimization algorithm, and that the resulting flow fields compare favorably to the state of the art on a number of small- and large-displacement datasets.


Optical flow large displacement 9 DoF PatchMatch PMBP 


  1. 1.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. Intl. J. of Comp. Vis. (2011)Google Scholar
  2. 2.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (2009)Google Scholar
  3. 3.
    Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized patchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  4. 4.
    Besse, F., Rother, C., Fitzgibbon, A., Kautz, J.: PMBP: PatchMatch belief propagation for correspondence field estimation. In: Proc. BMVC (2012)Google Scholar
  5. 5.
    Bleyer, M., Rhemann, C., Rother, C.: PatchMatch stereo-Stereo matching with slanted support windows. In: Proc. BMVC (2011)Google Scholar
  6. 6.
    Brox, T., Bregler, C., Malik, J.: Large displacement optical flow. In: Proc. CVPR (2009)Google Scholar
  7. 7.
    Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Intl. J. of Comp. Vis. (2012)Google Scholar
  8. 8.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM (1981)Google Scholar
  9. 9.
    Hartley, R., Zisserman, A.: Multiple view geometry in computer vision, vol. 2. Cambridge University Press (2000)Google Scholar
  10. 10.
    Heise, P., Klose, S., Jensen, B., Knoll, A.: PM-Huber: PatchMatch with Huber regularization for stereo matching. In: Proc. CVPR (2013)Google Scholar
  11. 11.
    Horn, B., Schunck, B.: Determining optical flow. Artificial Intelligence (1981)Google Scholar
  12. 12.
    Hornáček, M., Fitzgibbon, A., Rother, C.: SphereFlow: 6 DoF scene flow from RGB-D pairs. In: Proc. CVPR (June 2014)Google Scholar
  13. 13.
    Li, G., Zucker, S.W.: Surface geometric constraints for stereo in belief propagation. In: Proc. CVPR (2006)Google Scholar
  14. 14.
    Lowe, D.: Object recognition from local scale-invariant features. In: Proc. ICCV (1999)Google Scholar
  15. 15.
    Mac Aodha, O., Humayun, A., Pollefeys, M., Brostow, G.: Learning a confidence measure for optical flow. IEEE T-PAMI (2012)Google Scholar
  16. 16.
    Moisan, L., Stival, B.: A probabilistic criterion to detect rigid point matches between two images and estimate the fundamental matrix. Intl. J. of Comp. Vis. (2004)Google Scholar
  17. 17.
    Morel, J.M., Yu, G.: Asift: A new framework for fully affine invariant image comparison. SIAM Journal on Imaging Sciences (2009)Google Scholar
  18. 18.
    Nir, T., Bruckstein, A., Kimmel, R.: Over-parameterized variational optical flow. Intl. J. of Comp. Vis. (2008)Google Scholar
  19. 19.
    Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE T-PAMI (2004)Google Scholar
  20. 20.
    Rosman, G., Shem-Tov, S., Bitton, D., Nir, T., Adiv, G., Kimmel, R., Feuer, A., Bruckstein, A.: Over-parameterized optical flow using a stereoscopic constraint. Scale Space and Variational Methods in Computer Vision (2012)Google Scholar
  21. 21.
    Rother, C., Kolmogorov, V., Lempitsky, V., Szummer, M.: Optimizing binary MRFs via extended roof duality. In: Proc. CVPR (2007)Google Scholar
  22. 22.
    Sun, D., Roth, S., Black, M.: Secrets of optical flow estimation and their principles. In: Proc. CVPR (2010)Google Scholar
  23. 23.
    Trobin, W., Pock, T., Cremers, D., Bischof, H.: An unbiased second-order prior for high-accuracy motion estimation. Pattern Recognition (2008)Google Scholar
  24. 24.
    Valgaerts, L., Bruhn, A., Weickert, J.: A variational model for the joint recovery of the fundamental matrix and the optical flow. Pattern Recognition (2008)Google Scholar
  25. 25.
    Vogel, C., Schindler, K., Roth, S.: Piecewise rigid scene flow. In: Proc. ICCV (2013)Google Scholar
  26. 26.
    Wedel, A., Pock, T., Braun, J., Franke, U., Cremers, D.: Duality TV-L1 flow with fundamental matrix prior. Image and Vision Computing (2008)Google Scholar
  27. 27.
    Woodford, O., Torr, P., Reid, I., Fitzgibbon, A.: Global stereo reconstruction under second-order smoothness priors. IEEE T-PAMI (2009)Google Scholar
  28. 28.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE T-PAMI (2012)Google Scholar
  29. 29.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Generalized belief propagation. In: NIPS (2000)Google Scholar
  30. 30.
    Yoon, K., Kweon, I.: Adaptive support-weight approach for correspondence search. IEEE T-PAMI (2006)Google Scholar
  31. 31.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. Pattern Recognition (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Michael Hornáček
    • 1
  • Frederic Besse
    • 2
  • Jan Kautz
    • 2
  • Andrew Fitzgibbon
    • 3
  • Carsten Rother
    • 4
  1. 1.TU ViennaAustria
  2. 2.University College LondonUK
  3. 3.Microsoft Research CambridgeUK
  4. 4.TU DresdenGermany

Personalised recommendations