Skip to main content
SpringerLink
Log in

Noise-robust video super-resolution using an adaptive spatial-temporal filter

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we introduce a new interpolation-based super-resolution scheme for super-resolving a low-resolution video that contains large-scale local motions and/or heavy noise. Our scheme leverages an efficient space-time descriptor to adapt the interpolation kernel to the video’s spatial and temporal structures. Nevertheless, in the presence of large-scale local motions, the kernel suffers from tracking the motions incorrectly, leading to inaccurate temporal averaging. To address this problem, prior to computing the interpolation kernel, a mobile-neighborhood strategy that can identify the appropriate neighborhoods in adjacent frames is applied to neutralize the large-scale motions. Furthermore, we incorporate an adaptive sharpening technique into the kernel computation to remove the background noise and enhance the fine details simultaneously. Extensive experimental results on real-world videos show that the proposed method outperforms certain other state-of-the-art video super-resolution algorithms both visually and quantitatively, particularly in the presence of large-scale motions and/or heavy noise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. For simplicity, we will refer to this type of video SR approach as interpolation-based video SR approach.

  2. A “static” volume means a sequence of patches where the center pixel’s location in each is the same.

  3. All of these sequences (original, low-quality, and the processed sequences) appear in the first author’s website, at http://blog.sina.com.cn/s/blog_d71a34cb0101e2rt.html.

  4. This “Frog” video is downloaded from http://www.neatvideo.com/examples.html.

References

  1. Buades A, Coll B, Morel JM (2005) A non-local algorithm for image denoising. in Proc. IEEE Comput. Soc. Conf. on Computer Vision and Pattern Recognition pp. 60-65.

  2. Cheng M-H, Chen H-Y, Leou J-J (2011) Video super-resolution reconstruction using a mobile search strategy and adaptive patch size. Signal Process 91:1284–1297

    Article  Google Scholar 

  3. Dabov K et al (2007) Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Trans Image Process 16(8):2080–2095

    Article  MathSciNet  Google Scholar 

  4. Deledalle C-A, Denis L, Tupin F (2012) How to compare noisy patches? patch similarity beyond Gaussian noise. Int J Comput Vis 99:86–102

    Article  MathSciNet  MATH  Google Scholar 

  5. Farsiu S et al (2004) Advances and challenges in super-resolution. Int J Imaging Syst Technol 24(2):47–57

    Article  Google Scholar 

  6. Ferreira RU, Hung EM, de Queiroz RL (2012) Video super-resolution based on local invariant features matching. in: Proceedings of IEEE International Conference on Image Processing (ICIP) pp.877-880.

  7. Gao X, Wang Q, Tao X, Zhang K (2011) Zernike-moment-based image super resolution. IEEE Trans Image Process 20(10):2738–2747

    Article  MathSciNet  Google Scholar 

  8. Gupta G, Chakrabarti C (1995) Architectures for hierarchical and other block matching algorithms. IEEE Trans Circuits Syst Video Technol 5(6):477–489

    Article  Google Scholar 

  9. Hardie RC, Barnard KJ, Armstrong EE (1997) Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Trans Image Process 6(12):1621–1633

    Article  Google Scholar 

  10. http://www.infognition.com/videoenhancer/, Sep. 2012. Version 1.9.8

  11. Kim SH, Allebach JP (2005) Optimal unsharp mask for image sharpening and noise removal. J Electron Imaging 14:023005–1

    Article  Google Scholar 

  12. Kotera H, Wang H (2005) Multiscale image sharpening adaptive to edge profile. J Electron Imaging 14:013002–1

    Article  Google Scholar 

  13. Krinidis M, Nikolaidis N, Pitas I (2007) 2-D feature-point selection and tracking using 3-D physics-based deformable surfaces. IEEE Trans Circuits Syst Video Technol 17(7):876–888

    Article  Google Scholar 

  14. Lee I-H, Bose NK, Lin C-W (2010) Locally adaptive regularized super-resolution on video with arbitrary motion. in: Proceedings of IEEE International Conference on Image Processing (ICIP), pp.26-29.

  15. Liu C (2009) Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral Thesis. Massachusetts Institute of Technology

  16. Liu C, Sun D (2013) On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. to appear

  17. Liu F et al. (2008) Noisy video super resolution. ACM Int. Con. on Multimedia pp. 713-716

  18. Milanfar P (2011) Super resolution imaging. Taylor & Francis Group

  19. Park SC, Park MK, Kang MG (2003) Super-resolution image reconstruction: a technique overview. IEEE Signal Process Mag 20:21–36

    Article  Google Scholar 

  20. Polesel A, Ramponi G, Mathews VJ (2000) Image enhancement via adaptive unsharp 1 masking. IEEE Trans Image Process 9(3):505–510

    Article  Google Scholar 

  21. Protter M et al (2009) Generalizing the non-local-means to super-resolution reconstruction. IEEE Trans Image Process 18(1):36–51

    Article  MathSciNet  Google Scholar 

  22. Schulz RR, Stevenson RL (1996) Extraction of high-resolution frames from video sequences. IEEE Trans Image Process 5(6):996–1011

    Article  Google Scholar 

  23. Seo HJ, Milanfar P (2009) Static and space-time visual saliency detection by self-resemblance. J Vision 9(12):1–27

    Article  Google Scholar 

  24. Seo HJ, Milanfar P (2011) Action recognition from one example. IEEE Trans Pattern Anal Mach Intell 33(5):867–882

    Article  Google Scholar 

  25. Shahar O, Faktor A, Irani M (2011) Space-time super-resolution from a single video. 1 in Proc. IEEE Comput. Soc. Conf. on Computer Vision and Pattern Recognition, pp. 20-25.

  26. Shan Q et al (2008) Fast image/video upsampling. ACM Trans Graphics 27(5):1531–1537

    Article  Google Scholar 

  27. Shen H et al (2007) A MAP approach for joint motion estimation segmentation, and super resolution. IEEE Trans Image Process 16(2):479–490

    Article  MathSciNet  Google Scholar 

  28. Song H et al (2013) Adaptive regularization-based space-time super-resolution reconstruction. Signal Process Image Commun 28(7):763–778

    Article  Google Scholar 

  29. Su H, Wu Y, Zhou J (2012) Super-resolution without dense flow. IEEE Trans Image Process 21(4):1782–1795

    Article  MathSciNet  Google Scholar 

  30. Takeda H, Farsiu S, Milanfar P (2007) Kernel regression for image processing and reconstruction. IEEE Trans Image Process 16(2):349–366

    Article  MathSciNet  Google Scholar 

  31. Takeda H et al (2009) Super-resolution without explicit subpixel motion estimation. IEEE Trans Image Process 18(9):1958–1975

    Article  MathSciNet  Google Scholar 

  32. Vrigkasn M, Nikou C, Kondi LP (2013) Accurate image registration for MAP image super-resolution. Signal Process Image Commun 28(5):494–508

    Article  Google Scholar 

  33. Wang Z et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  34. Zhang H, Yang J, Zhang Y, Huang TS (2010) Non-local kernel regression for image and video restoration. in European Conference on Computer Vision (ECCV), 6313: 566-579.

  35. Zhang H, Yang J, Zhang Y, Huang TS (2013) Image and video restoration via non-local kernel regression. IEEE Trans Cybern 43(3):1035–1046

    Article  Google Scholar 

  36. Zhang L, Yuan Q, Shen H, Li P (2011) Multiframe image super-resolution adapted with local spatial information. J Opt Soc Am A 28(3):381–390

    Article  Google Scholar 

  37. Zhao W, Sawhney HS (2002) Is super-resolution with optical flow feasible? in 1 Proc. of European Conference on Computer Vision pp. 599-613.

  38. Zhong L et al. (2013) Handling noise in single image deblurring using directional filters. in Proc. IEEE Comput. Soc. Conf. on Computer Vision and Pattern Recognition.

  39. Zhou F, Yang W, Liao Q (2012) Interpolation-based image super-resolution using multi-surface fitting. IEEE Trans Image Process 21(7):3312–3318

    Article  MathSciNet  Google Scholar 

  40. Zhu X, Milanfar P (2011) Restoration for weakly blurred and strongly noisy images. IEEE Workshop on Applications of Computer Vision (WACV), pp. 103-109.

Download references

Acknowledgments

The work in this paper was supported by Brother Industries, Ltd., Japan. We would like to thank Mr. Masaki Kondo for the constructive comments and encouragement. We are also grateful to Mr. Zhou for sending us the code of [39].

Author information

Authors and Affiliations

  1. Department of Automation, Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing, 100084, China

    Jing Hu & Yupin Luo

Authors
  1. Jing Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yupin Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jing Hu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, J., Luo, Y. Noise-robust video super-resolution using an adaptive spatial-temporal filter. Multimed Tools Appl 74, 9259–9278 (2015). https://doi.org/10.1007/s11042-014-2079-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2079-y

Keywords

Search

Navigation