Temporal Frame Interpolation (TFI)

Chapter
Part of the Springer Theses book series (Springer Theses)

Abstract

As discussed in the last chapter, the temporal scalability of standardized scalable video codecs, e.g., H.264/SVC [15] and SHVC [16], is limited to reducing the framerate. One of the main reasons that the framerate can not be increased is that the target-frame anchored motion is estimated in an opportunistic way, which means that it does not in general describe the “true” motion trajectory of objects in the scene. In contrast, in the motion anchoring strategies explored in this thesis, motion information is anchored at reference frames, and temporal frame interpolation (TFI) is the essential building block that allows us to form predictions of the target frames.

References

  1. 1.
    H. Schwarz, D. Marpe, T. Wiegand, Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans. Circuit Syst. Video Technol. 17(9), 1103–1120 (2007)CrossRefGoogle Scholar
  2. 2.
    P. Helle, H. Lakshman, M. Siekmann, J. Stegemann, T. Hinz, H. Schwarz, D. Marpe, T. Wiegand, A scalable video coding extension of HEVC, in Proceedings of the IEEE Data Compression Conference (2013)Google Scholar
  3. 3.
    S.H. Chan, T.Q. Nguyen, LCD motion blur: modeling, analysis, and algorithm. IEEE Trans. Image Process. 20(8), 2352–2365 (2011)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    B. Girod, A.M. Aaron, S. Rane, D. Rebollo-Monedero, Distributed video coding. Proc. IEEE 93(1), 71–83 (2005)Google Scholar
  5. 5.
    G. De Haan, P.W.A.C. Biezen, H. Huijgen, O.A. Ojo, True- motion estimation with 3-D recursive search block matching. IEEE Trans. Circuit Syst. Video Technol. 3(5), 368–379 (1993)CrossRefGoogle Scholar
  6. 6.
    A. Beric, G. De Haan, J. Van Meerbergen, R. Sethuraman, Towards an Efficient High Quality Picture-rate Up-converter, 2003Google Scholar
  7. 7.
    T. Ha, S. Lee, J. Kim, Motion compensated frame interpolation by new block-based motion estimation algorithm. IEEE Trans. Consum. Electron. 50(2), 752–759 (2004)CrossRefGoogle Scholar
  8. 8.
    Q. Lu, N. Xu, X. Fang, Motion-compensated frame interpolation with multiframe based occlusion handling. IEEE J. Disp. Technol. 11(4), (2015)Google Scholar
  9. 9.
    B.K. Horn, B.G. Schunck, Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  10. 10.
    T. Brox, A. Bruhn, N. Papenberg, J. Weickert, High accuracy optical flow estimation based on a theory for warping, in European Conference on Computer Vision (2004), pp. 25–36Google Scholar
  11. 11.
    D. Sun, S. Roth, J. Lewis, M.J. Black, Learning optical flow, in European Conference on Computer Vision (2008), pp. 83–97Google Scholar
  12. 12.
    A. Wedel, D. Cremers, T. Pock, H. Bischof, Structure-and motion- adaptive regularization for high accuracy optic flow, in Proceedings of the IEEE International Conference on Computer Vision (2009), pp. 1663–1668Google Scholar
  13. 13.
    L. Xu, J. Jia, Y. Matsushita, Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)CrossRefGoogle Scholar
  14. 14.
    J. Wulff, M.J. Black, Modeling blurred video with layers, in European Conference on Computer Vision (2014), vol. 8694, pp. 236–252Google Scholar
  15. 15.
    J. Revaud, P. Weinzaepfel, Z. Harchaoui, C. Schmid, EpicFlow: edge-preserving interpolation of correspondences for optical flow, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  16. 16.
    M. Menze, C. Heipke, A. Geiger, Discrete optimization for optical flow, in German Conference on Pattern Recognition (2015)Google Scholar
  17. 17.
    Q. Chen, V. Koltun, Full flow: optical flow estimation by global optimization over regular grid, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016), vol. 2016, pp. 4706–4714Google Scholar
  18. 18.
    S. Dikbas, Y. Altunbasak, Novel true-motion estimation algorithm and its application to motion-compensated temporal frame interpolation. IEEE Trans. Image Process. 22(8), 2931–2945 (2013)CrossRefMATHMathSciNetGoogle Scholar
  19. 19.
    B.-D. Choi, J.-W. Han, C.-S. Kim, S.-J. Ko, Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Trans. Circuit Syst. Video Technol. 17(4), 407–416 (2007)CrossRefGoogle Scholar
  20. 20.
    C. Wang, L. Zhang, Y. He, Y.-P. Tan, Frame rate up-conversion using trilateral filtering. IEEE Trans. Circuit Syst. Video Technol. 20(6), 886–893 (2010)CrossRefGoogle Scholar
  21. 21.
    A. Veselov, M. Gilmutdinov, Iterative hierarchical true motion estimation for temporal frame interpolation, in IEEE International Workshop on Multimedia Signal Processing (2014)Google Scholar
  22. 22.
    L.L. Rakêt, L. Roholm, A. Bruhn, J. Weickert, Motion compensated frame interpolation with a symmetric optical flow constraint. Adv. Vis. Comput. 447–457 (2012)Google Scholar
  23. 23.
    S.-G. Jeong, C. Lee, C.-S. Kim, Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization. IEEE Trans. Image Process. 22, 4497–4509 (2013)CrossRefMATHMathSciNetGoogle Scholar
  24. 24.
    Y. Chin, C.-J. Tsai, Dense true motion field compensation for video coding, in Proceedings of the IEEE International Conference on Image Processing (2013), pp. 1958–1961Google Scholar
  25. 25.
    D. Sun, S. Roth, M.J. Black, Secrets of optical flow estimation and their principles, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2010), pp. 2432–2439Google Scholar
  26. 26.
    D. Kim, H. Lim, H. Park, Iterative true motion estimation for motion-compensated frame interpolation. IEEE Trans. Circuit Syst. Video Technol. 23(3), 445–454 (2013)CrossRefGoogle Scholar
  27. 27.
    Y.-H. Cho, H.-Y. Lee, D.-S. Park, Temporal frame interpolation based on multiframe feature trajectory. IEEE Trans. Circuit Syst. Video Technol. 23(12), 2105–2115 (2013)CrossRefGoogle Scholar
  28. 28.
    D. Kim, H. Park, An efficient motion-compensated frame for high-resolution videos. IEEE J. Disp. Technol. 11(7), 580–588 (2015)CrossRefGoogle Scholar
  29. 29.
    E. Herbst, S. Seitz, S. Baker, Occlusion Reasoning for Temporal Interpolation using Optical Flow, Department of Computer Science and Engineering, University of Washington, Technical report. UW-CSE-09- 08-01, 2009Google Scholar
  30. 30.
    T. Stich, C. Linz, C. Wallraven, D. Cunningham, M. Magnor, Perception-motivated interpolation of image sequences. ACM Trans. Appl. Percept. 8(2), 1–25 (2011)CrossRefGoogle Scholar
  31. 31.
    W.R. Mark, L. McMillan, G. Bishop, Post-rendering 3D warping, in Proceedings of the Symposium on Interactive 3D Graphics (1997), pp. 7–16Google Scholar
  32. 32.
    R. Leonardi, A. Iocco, Time-varying motion estimation on a sequence of images. Multimed. Commun. Video Coding, 309–315 (1996)Google Scholar
  33. 33.
    P. Csillag, L. Boroczky, Estimation of accelerated motion for motion-compensated frame interpolation. Vis. Commun. Image Process. 604–614 (1996)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Electrical Engineering and TelecommunicationsUNSW SydneySydneyAustralia

Personalised recommendations