Multiframe Motion Coupling for Video Super Resolution

  • Jonas GeipingEmail author
  • Hendrik Dirks
  • Daniel Cremers
  • Michael Moeller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10746)


The idea of video super resolution is to use different view points of a single scene to enhance the overall resolution and quality. Classical energy minimization approaches first establish a correspondence of the current frame to all its neighbors in some radius and then use this temporal information for enhancement. In this paper, we propose the first variational super resolution approach that computes several super resolved frames in one batch optimization procedure by incorporating motion information between the high-resolution image frames themselves. As a consequence, the number of motion estimation problems grows linearly in the number of frames, opposed to a quadratic growth of classical methods and temporal consistency is enforced naturally.

We use infimal convolution regularization as well as an automatic parameter balancing scheme to automatically determine the reliability of the motion information and reweight the regularization locally. We demonstrate that our approach yields state-of-the-art results and even is competitive with machine learning approaches.



J.G. and M.M. acknowledge the support of the German Research Foundation (DFG) via the research training group GRK 1564 Imaging New Modalities. D.C. was partially funded by the ERC Consolidator grant 3D Reloaded.


  1. 1.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRefGoogle Scholar
  2. 2.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). CrossRefGoogle Scholar
  3. 3.
    Burger, M., Dirks, H., Schönlieb, C.B.: A variational model for joint motion estimation and image reconstruction. arXiv preprint arXiv:1607.03255 (2016)
  4. 4.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  5. 5.
    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76(2), 167–188 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Dirks, H.: A flexible primal-dual toolbox. arXiv preprint (2016).
  8. 8.
    Holler, M., Kunisch, K.: On infimal convolution of TV-type functionals and applications to video and image reconstruction. SIAM J. Imaging Sci. 7(4), 2258–2300 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2(2), 109–122 (2016)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, June 2016Google Scholar
  11. 11.
    Liao, R., Tao, X., Li, R., Ma, Z., Jia, J.: Video super-resolution via deep draft-ensemble learning. In: ICCV, pp. 531–539 (2015)Google Scholar
  12. 12.
    Liu, C., Sun, D.: On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 346–360 (2014)CrossRefGoogle Scholar
  13. 13.
    Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., Wu, E.: Handling motion blur in multi-frame super-resolution. In: CVPR, pp. 5224–5232 (2015)Google Scholar
  14. 14.
    Marquina, A., Osher, S.J.: Image super-resolution by TV-regularization and bregman iteration. J. Sci. Comput. 37(3), 367–382 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Mitzel, D., Pock, T., Schoenemann, T., Cremers, D.: Video super resolution using duality based TV-L1 optical flow. In: Pattern Recognition, pp. 432–441 (2009)Google Scholar
  16. 16.
    Möllenhoff, T., Laude, E., Moeller, M., Lellmann, J., Cremers, D.: Sublabel-accurate relaxation of nonconvex energies. In: CVPR, June 2016.
  17. 17.
    Mueller, J., Siltanen, S.: Linear and Nonlinear Inverse Problems with Practical Applications. Society for Industrial and Applied Mathematics, Philadelphia (2012)CrossRefzbMATHGoogle Scholar
  18. 18.
    Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)CrossRefGoogle Scholar
  19. 19.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the Mumford-Shah functional. In: ICCV, pp. 1133–1140. IEEE (2009)Google Scholar
  20. 20.
    Infognition Co., Ltd: Videoenhancer 2 software, version 2.1.
  21. 21. Redistributable Video Test Media Collection.
  22. 22.
    Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)Google Scholar
  23. 23.
    Sony Corporation: Sony 4K UHD surfing screen test demo. CC-BY LicenseGoogle Scholar
  24. 24.
    Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV 106(2), 115–137 (2014)CrossRefGoogle Scholar
  25. 25.
    Unger, M., Pock, T., Werlberger, M., Bischof, H.: A convex approach for variational super-resolution. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) DAGM 2010. LNCS, vol. 6376, pp. 313–322. Springer, Heidelberg (2010). CrossRefGoogle Scholar
  26. 26.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  27. 27.
    Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for TV-L 1 optical flow. In: Cremers, D., Rosenhahn, B., Yuille, A.L., Schmidt, F.R. (eds.) Statistical and Geometrical Approaches to Visual Motion Analysis. LNCS, vol. 5604, pp. 23–45. Springer, Heidelberg (2009). CrossRefGoogle Scholar
  28. 28.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007). CrossRefGoogle Scholar
  30. 30.
    Zhang, Z., Sze, V.: Fast: free adaptive super-resolution via transfer for compressed videos. ArXiv (2016)

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jonas Geiping
    • 1
    Email author
  • Hendrik Dirks
    • 2
  • Daniel Cremers
    • 3
  • Michael Moeller
    • 1
  1. 1.University of SiegenSiegenGermany
  2. 2.University of MünsterMünsterGermany
  3. 3.Technical University of MunichMunichGermany

Personalised recommendations