Illumination-Aware Large Displacement Optical Flow

  • Michael StollEmail author
  • Daniel Maurer
  • Sebastian Volz
  • Andrés Bruhn
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10746)


The integration of feature matches for handling large displacements is one of the key concepts of recent variational optical flow methods. In this context, many existing approaches rely on confidence measures to identify locations where a poor initial match can potentially be improved by adaptively integrating flow proposals. One very intuitive confidence measure to identify such locations is the matching cost of the data term. Problems arise, however, in the presence of illumination changes, since brightness constancy does not hold and invariant constancy assumptions typically discard too much information for an identification of poor matches. In this paper, we suggest a pipeline approach that addresses the aforementioned problem in two ways. First, we propose a novel confidence measure based on the illumination-compensated brightness constancy assumption. By estimating illumination changes from a pre-computed flow this measure allows us to reliably identify poor matches even in the presence of varying illumination. Secondly, in contrast to many existing pipeline approaches, we propose to integrate only feature matches that have been obtained from dense variational methods. This in turn not only provides robust matches due to the inherent regularization, it also demonstrates that in many cases sparse descriptor matches are not needed for large displacement optical flow. Experiments on the Sintel benchmark and on common large displacement sequences demonstrate the benefits of our strategy. They show a clear improvement over the baseline method and a comparable performance as similar methods from the literature based on sparse feature matches.



We thank the German Research Foundation (DFG) for financial support within project B04 of SFB/Transregio 161.


  1. 1.
    Alvarez, L., Esclarín, J., Lefébure, M., Sánchez, J.: A PDE model for computing the optical flow. In: Proceedings of Congreso de Ecuaciones Diferenciales y Aplicaciones, pp. 1349–1356 (1999)Google Scholar
  2. 2.
    Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2010)CrossRefGoogle Scholar
  3. 3.
    Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized PatchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). CrossRefGoogle Scholar
  4. 4.
    Berg, A., Malik, J.: Geometric blur for template matching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 607–614 (2001)Google Scholar
  5. 5.
    Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 292–302 (1991)Google Scholar
  6. 6.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRefGoogle Scholar
  7. 7.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). CrossRefGoogle Scholar
  9. 9.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)CrossRefGoogle Scholar
  10. 10.
    Bruhn, A., Weickert, J.: A confidence measure for variational optic flow methods. In: Klette, R., Kozera, R., Noakes, L., Weickert, J. (eds.) Geometric Properties from Incomplete Data, Computational Imaging and Vision, vol. 31, pp. 283–297. Springer, Dordrecht (2006). CrossRefGoogle Scholar
  11. 11.
    Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of IEEE International Conference on Image Processing, pp. 168–172 (1994)Google Scholar
  12. 12.
    Chen, Z., Jin, H., Lin, Z., Cohen, S., Wu, Y.: Large displacement optical flow from nearest neighbor fields. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2443–2450 (2013)Google Scholar
  13. 13.
    Dalal, N., Triggs, B.: Histogram of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)Google Scholar
  14. 14.
    Demetz, O., Stoll, M., Volz, S., Weickert, J., Bruhn, A.: Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 455–471. Springer, Cham (2014). Google Scholar
  15. 15.
    Drayer, B., Brox, T.: Combinatorial regularization of descriptor matching for optical flow estimation. In: Proceedings of British Machine Vision Conference, pp. 42.1–42.12 (2015)Google Scholar
  16. 16.
    Förstner, W., Gülch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, pp. 281–305 (1987)Google Scholar
  17. 17.
    Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  18. 18.
    Lempitsky, V., Roth, S., Rother, C.: FusionFlow: discrete-continuous optimization for optical flow estimation. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  19. 19.
    Lowe, D., Bruckstein, A.M., Kimmel, R.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  20. 20.
    Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: SIFT flow: dense correspondence across different scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 28–42. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  21. 21.
    Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)CrossRefGoogle Scholar
  22. 22.
    Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. Int. J. Comput. Vis. 76(2), 205–216 (2008)CrossRefGoogle Scholar
  23. 23.
    Rashwan, H.A., Mohamed, M.A., García, M.A., Mertsching, B., Puig, D.: Illumination robust optical flow model based on histogram of oriented gradients. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 354–363. Springer, Heidelberg (2013). CrossRefGoogle Scholar
  24. 24.
    Sevilla-Lara, L., Sun, D., Learned-Miller, E.G., Black, M.J.: Optical flow estimation with channel constancy. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 423–438. Springer, Cham (2014). Google Scholar
  25. 25.
    Stoll, M., Volz, S., Bruhn, A.: Variational large displacement optical flow without feature matches. In: Pelillo, M., Hancock, E. (eds.) EMMCVPR 2017. LNCS, vol. 10746, pp. 79–92. Springer, Cham (2017)Google Scholar
  26. 26.
    Stoll, M., Volz, S., Bruhn, A.: Adaptive integration of feature matches into variational optical flow methods. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7726, pp. 1–14. Springer, Heidelberg (2013). CrossRefGoogle Scholar
  27. 27.
    Stoll, M., Volz, S., Maurer, D., Bruhn, A.: A time-efficient optimisation framework for parameters of optical flow methods. In: Sharma, P., Bianchi, F.M. (eds.) SCIA 2017. LNCS, vol. 10269, pp. 41–53. Springer, Cham (2017). CrossRefGoogle Scholar
  28. 28.
    Tu, Z., Poppe, R., Veltkamp, R.C.: Weighted local intensity fusion method for variational optical flow estimation. Pattern Recogn. 50, 223–232 (2016)CrossRefGoogle Scholar
  29. 29.
    Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: Proceedings of International Conference on Computer Vision, pp. 1116–1123 (2011)Google Scholar
  30. 30.
    Weickert, J., Schnörr, C.: A theoretical framework for convex regularizers in PDE-based computation of image motion. Int. J. Comput. Vis. 45(3), 245–264 (2001)CrossRefzbMATHGoogle Scholar
  31. 31.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: Proceedings of International Conference on Computer Vision, pp. 1385–1392 (2013)Google Scholar
  32. 32.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)CrossRefGoogle Scholar
  33. 33.
    Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., Seidel, H.-P.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 207–220. Springer, Heidelberg (2009). CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Michael Stoll
    • 1
    Email author
  • Daniel Maurer
    • 1
  • Sebastian Volz
    • 1
  • Andrés Bruhn
    • 1
  1. 1.University of StuttgartStuttgartGermany

Personalised recommendations