Continual Occlusion and Optical Flow Estimation

  • Michal NeoralEmail author
  • Jan Šochman
  • Jiří Matas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)


Two optical flow estimation problems are addressed: (i) occlusion estimation and handling, and (ii) estimation from image sequences longer than two frames. The proposed ContinualFlow method estimates occlusions before flow, avoiding the use of flow corrupted by occlusions for their estimation. We show that providing occlusion masks as an additional input to flow estimation improves the standard performance metric by more than 25% on both KITTI and Sintel. As a second contribution, a novel method for incorporating information from past frames into flow estimation is introduced. The previous frame flow serves as an input to occlusion estimation and as a prior in occluded regions, i.e. those without visual correspondences. By continually using the previous frame flow, ContinualFlow performance improves further by 18% on KITTI and 7% on Sintel, achieving top performance on KITTI and Sintel.



The research was supported by Toyota Motor Europe, CTU student grant SGS17/185/OHK3/3T/13 and the OP VVV MEYS project CZ.02.1.01/0.0/0.0/16_019/0000765 Research Center for Informatics.


  1. 1.
    Bailer, C., Taetz, B., Stricker, D.: Flow fields: dense correspondence fields for highly accurate large displacement optical flow estimation. In: ICCV (2015)Google Scholar
  2. 2.
    Bailer, C., Varanasi, K., Stricker, D.: CNN-based patch matching for optical flow with thresholded hinge embedding loss. In: CVPR, pp. 3250–3259 (2017)Google Scholar
  3. 3.
    Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. IJCV (2011). Scholar
  4. 4.
    Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: International Conference on Machine Learning, pp. 41–48. ACM (2009)Google Scholar
  5. 5.
    Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: CVPR (1991)Google Scholar
  6. 6.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). Scholar
  7. 7.
    Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR 2017. IEEE (2017)Google Scholar
  8. 8.
    Chen, Q., Koltun, V.: Full flow: optical flow estimation by global optimization over regular grids. In: CVPR (2016)Google Scholar
  9. 9.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: ICCV, pp. 2758–2766, December 2015Google Scholar
  10. 10.
    Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: CVPR (2016)Google Scholar
  11. 11.
    Garg, R., Pizarro, L., Rueckert, D., Agapito, L.: Dense multi-frame optic flow for non-rigid objects using subspace constraints. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6495, pp. 460–473. Springer, Heidelberg (2011). Scholar
  12. 12.
    Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. IJCV 104(3), 286–314 (2013)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Güney, F., Geiger, A.: Deep discrete flow. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10114, pp. 207–224. Springer, Cham (2017). Scholar
  14. 14.
    Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)CrossRefGoogle Scholar
  15. 15.
    Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. In: CVPR (2017)Google Scholar
  16. 16.
    Hui, T.W., Tang, X., Loy, C.C.: LiteFlowNet: a lightweight convolutional neural network for optical flow estimation. In: CVPR, June 2018Google Scholar
  17. 17.
    Hur, J., Roth, S.: MirrorFlow: exploiting symmetries in joint optical flow and occlusion estimation. ICCV (2017)Google Scholar
  18. 18.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)Google Scholar
  19. 19.
    Irani, M.: Multi-frame correspondence estimation using subspace constraints. IJCV 48(3), 173–194 (2002)CrossRefGoogle Scholar
  20. 20.
    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: ANIPS (2015)Google Scholar
  21. 21.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  22. 22.
    Kondermann, D., et al.: The HCI benchmark suite: stereo and flow ground truth with uncertainties for urban autonomous driving. In: CVPR (2016)Google Scholar
  23. 23.
    Maurer, D., Bruhn, A.: ProFlow: learning to predict optical flow. In: BMVC (2018)Google Scholar
  24. 24.
    Mayer, N., et al.: What makes good synthetic training data for learning disparity and optical flow estimation? IJCV 126(9), 942–960 (2018)CrossRefGoogle Scholar
  25. 25.
    Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: CVPR (2016)Google Scholar
  26. 26.
    Meister, S., Hur, J., Roth, S.: UnFlow: unsupervised learning of optical flow with a bidirectional census loss. In: AAAI (2018)Google Scholar
  27. 27.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)Google Scholar
  28. 28.
    Murray, D.W., Buxton, B.F.: Scene segmentation from visual motion using global optimization. PAMI 9, 220–228 (1987)CrossRefGoogle Scholar
  29. 29.
    Pang, J., Sun, W., Ren, J.S., Yang, C., Yan, Q.: Cascade residual learning: a two-stage convolutional neural network for stereo matching. In: CVPR (2017)Google Scholar
  30. 30.
    Pérez, J.S., Meinhardt-Llopis, E., Facciolo, G.: TV-L1 optical flow estimation. Image Process. Line 3, 137–150 (2013)CrossRefGoogle Scholar
  31. 31.
    Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: CVPR (2018)Google Scholar
  32. 32.
    Robust vision challenge team: robust vision challenge (2018). Accessed 8 July 2018
  33. 33.
    Schuster, R., Bailer, C., Wasenmüller, O., Stricker, D.: FlowFields++: accurate optical flow correspondences meet robust interpolation. In: ICIP (2018)Google Scholar
  34. 34.
    Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: CVPR, pp. 1098–1105 (2014)Google Scholar
  35. 35.
    Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR (2018)Google Scholar
  36. 36.
    Sundaram, N., Brox, T., Keutzer, K.: Dense point trajectories by GPU-accelerated large displacement optical flow. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 438–451. Springer, Heidelberg (2010). Scholar
  37. 37.
    Unger, M., Werlberger, M., Pock, T., Bischof, H.: Joint motion estimation and segmentation of complex scenes with label costs and occlusion modeling. In: CVPR, pp. 1878–1885. IEEE (2012)Google Scholar
  38. 38.
    Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: ICCV (2011)Google Scholar
  39. 39.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  40. 40.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV (2013)Google Scholar
  41. 41.
    Wulff, J., Sevilla-Lara, L., Black, M.J.: Optical flow in mostly rigid scenes. In: CVPR (2017)Google Scholar
  42. 42.
    Xiao, J., Cheng, H., Sawhney, H., Rao, C., Isnardi, M.: Bilateral filtering-based optical flow estimation with occlusion detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 211–224. Springer, Heidelberg (2006). Scholar
  43. 43.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Center for Machine Perception, Faculty of Electrical EngineeringCzech Technical University in PraguePragueCzech Republic

Personalised recommendations