A Comparison of Isotropic and Anisotropic Second Order Regularisers for Optical Flow

  • Daniel MaurerEmail author
  • Michael Stoll
  • Sebastian Volz
  • Patrick Gairing
  • Andrés Bruhn
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10302)


In variational optical flow estimation, second order regularisation plays an important role, since it offers advantages in the context of non-fronto-parallel motion. However, in contrast to first order smoothness constraints, most second order regularisers are limited to isotropic concepts. Moreover, the few existing anisotropic concepts are lacking a comparison so far. Hence, our contribution is twofold. (i) First, we juxtapose general concepts for isotropic and anisotropic second order regularization based on direct second order methods, infimal convolution techniques, and indirect coupling models. For all the aforementioned strategies suitable optical flow regularisers are derived. (ii) Second, we show that modelling anisotropic second order smoothness terms gives an additional degree of freedom when penalising deviations from smoothness. This in turn allows us to propose a novel anisotropic strategy which we call double anisotropic regularisation. Experiments on the two KITTI benchmarks show the qualitative differences between the different strategies. Moreover, they demonstrate that the novel concept of double anisotropic regularisation is able to produce excellent results.


Higher order regularisation Variational methods Optical flow 



We thank the German Research Foundation (DFG) for financial support within project B04 of SFB/Transregio 161.


  1. 1.
    Braux-Zin, J., Dupont, R., Bartoli, A.: A general dense image matching framework combining direct and feature-based costs. In: Proceedings of International Conference on Computer Vision, pp. 185–192 (2013)Google Scholar
  2. 2.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24673-2_3 CrossRefGoogle Scholar
  4. 4.
    Chambolle, A., Lions, P.-L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chan, T., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Deterministic edge-preserving regularization in computed imaging. IEEE Trans. Image Process. 6(2), 298–311 (1997)CrossRefGoogle Scholar
  7. 7.
    Demetz, O., Stoll, M., Volz, S., Weickert, J., Bruhn, A.: Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 455–471. Springer, Cham (2014). doi: 10.1007/978-3-319-10590-1_30 Google Scholar
  8. 8.
    Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: Proceedings of International Conference on Computer Vision, pp. 993–1000 (2013)Google Scholar
  9. 9.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361 (2012)Google Scholar
  10. 10.
    Hafner, D., Schroers, C., Weickert, J.: Introducing maximal anisotropy into second order coupling models. In: Gall, J., Gehler, P., Leibe, B. (eds.) GCPR 2015. LNCS, vol. 9358, pp. 79–90. Springer, Cham (2015). doi: 10.1007/978-3-319-24947-6_7 CrossRefGoogle Scholar
  11. 11.
    Hewer, A., Weickert, J., Scheffer, T., Seibert, H., Diebels, S.: Lagrangian strain tensor computation with higher order variational models. In: Proceedings of British Machine Vision Conference, pp. 129.1–129.10 (2013)Google Scholar
  12. 12.
    Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  13. 13.
    Lefkimmiatis, S., Ward, J.P., Unser, M.: Hessian Schatten-norm regularization for linear inverse problems. IEEE Trans. Image Process. 22(5), 1873–1888 (2013)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Lellmann, J., Morel, J.-M., Schönlieb, C.-B.: Anisotropic third-order regularization for sparse digital elevation models. In: Kuijper, A., Bredies, K., Pock, T., Bischof, H. (eds.) SSVM 2013. LNCS, vol. 7893, pp. 161–173. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38267-3_14 CrossRefGoogle Scholar
  15. 15.
    Lenzen, F., Becker, F., Lellmann, J.: Adaptive second-order total variation: an approach aware of slope discontinuities. In: Kuijper, A., Bredies, K., Pock, T., Bischof, H. (eds.) SSVM 2013. LNCS, vol. 7893, pp. 61–73. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38267-3_6 CrossRefGoogle Scholar
  16. 16.
    Lysaker, M., Lundervold, A., Tai, X.C.: Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time. IEEE Trans. Image Process. 12(12), 1579–1590 (2003)CrossRefzbMATHGoogle Scholar
  17. 17.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3061–3070 (2015)Google Scholar
  18. 18.
    Nagel, H.-H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Patt. Anal. Mach. Intell. 8(5), 565–593 (1986)CrossRefGoogle Scholar
  19. 19.
    Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. Int. J. Comput. Vis. 76(2), 205–216 (2008)CrossRefGoogle Scholar
  20. 20.
    Perona, P., Malik, J.: Scale space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 2, 629–639 (1990)CrossRefGoogle Scholar
  21. 21.
    Ranftl, R., Gehrig, S., Pock, T., Bischof, H.: Pushing the limits of stereo using variational stereo estimation. In: IEEE Intelligent Vehicles Symposium, pp. 401–407 (2012)Google Scholar
  22. 22.
    Ranftl, R., Bredies, K., Pock, T.: Non-local total generalized variation for optical flow estimation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 439–454. Springer, Cham (2014). doi: 10.1007/978-3-319-10590-1_29 Google Scholar
  23. 23.
    Ranftl, R.: Higher-Order Variational Methods for Dense Correspondence Problems. Ph.D thesis. Graz University of Technology, Austria (2014)Google Scholar
  24. 24.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: EpicFlow: Edge-preserving interpolation of correspondences for optical flow. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1164–1172 (2015)Google Scholar
  25. 25.
    Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Scherzer, O.: Denoising with higher order derivatives of bounded variation and an application to parameter estimation. Computing 60(1), 1–27 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Sun, D., Roth, S., Lewis, J.P., Black, M.J.: Learning optical flow. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 83–97. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88690-7_7 CrossRefGoogle Scholar
  28. 28.
    Trobin, W., Pock, T., Cremers, D., Bischof, H.: An Unbiased second-order prior for high-accuracy motion estimation. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 396–405. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-69321-5_40 CrossRefGoogle Scholar
  29. 29.
    Vogel, O., Bruhn, A., Weickert, J., Didas, S.: Direct shape-from-shading with adaptive higher order regularisation. In: Sgallari, F., Murli, A., Paragios, N. (eds.) SSVM 2007. LNCS, vol. 4485, pp. 871–882. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-72823-8_75 CrossRefGoogle Scholar
  30. 30.
    Weickert, J., Welk, M., Wickert, M.: L 2-Stable nonstandard finite differences for anisotropic diffusion. In: Kuijper, A., Bredies, K., Pock, T., Bischof, H. (eds.) SSVM 2013. LNCS, vol. 7893, pp. 380–391. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38267-3_32 CrossRefGoogle Scholar
  31. 31.
    Weickert, J., Schnörr, C.: A theoretical framework for convex regularizers in PDE-based computation of image motion. Int. J. Comput. Vision 45(3), 245–264 (2001)CrossRefzbMATHGoogle Scholar
  32. 32.
    Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-\(L^1\) optic flow. In: Proceedings of British Machine Vision Conference (2009)Google Scholar
  33. 33.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-\(L^1\) optical flow. In: Proceedings of German Pattern Recognition, pp. 214–223 (2007)Google Scholar
  34. 34.
    Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93(3), 368–388 (2011)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Daniel Maurer
    • 1
    Email author
  • Michael Stoll
    • 1
  • Sebastian Volz
    • 1
  • Patrick Gairing
    • 1
  • Andrés Bruhn
    • 1
  1. 1.Computer Vision and Intelligent Systems Group, Institute for Visualization and Interactive SystemsUniversity of StuttgartStuttgartGermany

Personalised recommendations