Abstract
Video denoising is an important and open problem, which is less treated than the single-image case. Most image sequence denoising techniques rely on still image denoising algorithms; however, it is possible to take advantage of the redundant information contained in the sequence to improve the denoising results. Most recent algorithms are patch based. These methods have two clearly differentiated steps: select similar patches to a reference one and estimate a noise-free version from this group. We review selection and estimation strategies. In particular, we show that the performance is improved by introducing motion compensation. We use as example a recent video denoising technique inspired by fusion algorithms that use motion compensation by regularized optical flow methods, which permits robust patch comparison in a spatiotemporal volume. The use of principal component analysis ensures the correct preservation of fine texture and details, provided that the noise is Gaussian and white, with known variance. Video acquired by any video camera or mobile phone undergoes several processings from the sensor to the final output. This processing, including at least demosaicking, white balance, gamma correction, filtering, and compression, makes a white noise model unrealistic. Indeed, real video captured in dark environments has a very poor quality, with severe spatially and temporally correlated noise. We discuss a denoising framework including realistic noise estimation, multiscale processing, variance stabilization, and white noise removal algorithms. We illustrate the performance of such a chain with real dark and compressed movie sequences.
The authors were supported by grant TIN2014-53772R and TIN2017-85572-P.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We used the Matlab implementations of Ji et al. (by Sibin, Yuhong, and Yu, 2013), and VIDOLSAT (from http://www.ifp.illinois.edu/~yoram). The rest of algorithms were implemented following the descriptions in the published papers. Remark that Gao et al. method is an extension to video sequences of the method described in [65].
References
Adams A, Gelfand N, Pulli K (2008) Viewfinder alignment. In: Computer graphics forum, vol 27, pp 597–606. Wiley Online Library
Alvarez L, Lions PL, Morel JM (1992) Image selective smoothing and edge detection by nonlinear diffusion II. SIAM J Numer Anal 29(3):845–866
Alvarez L, Weickert J, Sánchez J (2000) Reliable estimation of dense optical flow fields with large displacements. Int J Comput Vis 39(1):41–56
Arias P, Morel JM (2017) Video denoising via empirical Bayesian estimation of space-time patches. J Math Imaging Vis 1–24
Ayvaci A, Raptis M, Soatto S (2010) Occlusion detection and motion estimation with convex optimization. In: Advances in neural information processing systems, pp 100–108
Ballester C, Garrido L, Lazcano V, Caselles V (2012) A TV-L1 optical flow method with occlusion detection. In: Lecture notes in computer science, vol 7476. Springer, pp 31–40
Bennett E, McMillan L (2005) Video enhancement using per-pixel virtual exposures. In: ACM SIGGRAPH 2005 papers. ACM, p 852
Boulanger J, Kervrann C, Bouthemy P (2007) Space-time adaptation for patch-based image sequence restoration. IEEE Trans Pattern Anal Mach Intell 29(6):1096–1102
Brox T, Bruhn A, Papenberg N, Weickert J (2004) High accuracy optical flow estimation based on a theory for warping. In: European conference on computer vision. Springer, pp 25–36
Brox T, Malik J (2011) Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Mach Intell 33(3):500–513
Buades A, Coll B, Morel J (2005) A non local algorithm for image denoising. IEEE Comput Vis Pattern Recognit 2:60–65
Buades A, Coll B, Morel J (2008) Nonlocal image and movie denoising. Int J Comput Vis 76(2):123–139
Buades A, Duran J (2017) Flow-based video super-resolution with spatio-temporal patch similarity. In: Proceedings British machine vision conference 2017. BMVA
Buades A, Lisani JL (2017) Denoising of noisy and compressed video sequences. In: VISIGRAPP (4: VISAPP), pp 150–157
Buades A, Lisani JL (2017) Enhancement of noisy and compressed videos by optical flow and non-local denoising. IEEE Trans Image Process (submitted)
Buades A, Lisani JL, Miladinović M (2016) Patch-based video denoising with optical flow estimation. IEEE Trans Image Process 25(6):2573–2586
Buades A, Lou Y, Morel JM, Tang Z (2009) A note on multi-image denoising. In: International workshop on local and non-local approximation in image processing. IEEE, pp 1–15
Chambolle A (2004) An algorithm for total variation minimization and applications. J Math Imaging Vis 20:89–97
Colom M, Buades A, Morel JM (2014) Nonparametric noise estimation method for raw images. JOSA A 31(4):863–871
Colom M, Lebrun M, Buades A, Morel JM (2015) Nonparametric multiscale blind estimation of intensity-frequency dependent noise. IEEE Trans Image Process
Dabov K, Foi A, Egiazarian K (2007) Video denoising by sparse 3D transform-domain collaborative filtering. In: Proceedings of the 15th European signal processing conference, vol 1, p 7
Dabov K, Foi A, Katkovnik V, Egiazarian K et al (2009) BM3D image denoising with shape-adaptive principal component analysis. In: Proceedings of the workshop on signal processing with adaptive sparse structured representations, Saint-Malo, France
Delbracio M, Sapiro G (2015) Hand-held video deblurring via efficient Fourier aggregation. IEEE Trans Comput Imaging 1(4):270–283
Deledalle CA, Tupin F, Denis L (2010) Poisson NL-means: unsupervised non local means for Poisson noise. In: 17th IEEE international conference on image processing. IEEE, pp 801–804
Donoho D (1995) De-noising by soft-thresholding. IEEE Trans Inf Theory 41(3):613–627
Drago F, Myszkowski K, Annen T, Chiba N (2003) Adaptive logarithmic mapping for displaying high contrast scenes. In: Proceedings of EUROGRAPHICS, vol 22
Durand F, Dorsey J (2002) Fast bilateral filtering for the display of high-dynamic-range images. In: ACM transactions on graphics (TOG), vol 21. ACM, pp 257–266
Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15(12):3736–3745
Farsiu S, Robinson M, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344
Gao Y, Hu HM, Wu J (2015) Video denoising algorithm via multi-scale joint luma-chroma bilateral filter. In: 2015 visual communications and image processing (VCIP). IEEE, pp 1–4
Haro G, Buades A, Morel JM (2012) Photographing paintings by image fusion. SIAM J Imaging Sci 5(3):1055–1087
Horn B, Schunck B (1981) Determining optical flow. In: Technical symposium east. International Society for Optics and Photonics, pp 319–331
Ji H, Huang S, Shen Z, Xu Y (2011) Robust video restoration by joint sparse and low rank matrix approximation. SIAM J Imaging Sci 4(4):1122–1142
Joshi N, Cohen M (2010) Seeing Mt. Rainier: lucky imaging for multi-image denoising, sharpening, and haze removal. In: IEEE international conference on computational photography, pp 1–8
Jovanov L, Luong H, Ružic T, Philips W (2015) Multiview image sequence enhancement. In: SPIE/IS&T electronic imaging. International Society for Optics and Photonics, pp 93,990K–93,990K
Kim M, Park D, Han DK, Ko H (2015) A novel approach for denoising and enhancement of extremely low-light video. IEEE Trans Consum Electron 61(1):72–80
Kokaram AC (2013) Motion picture restoration: digital algorithms for artefact suppression in degraded motion picture film and video. Springer Science & Business Media
Lebrun M, Buades A, Morel JM (2013) A nonlocal Bayesian image denoising algorithm. SIAM J Imaging Sci 6(3):1665–1688
Lebrun M, Colom M, Buades A, Morel JM (2012) Secrets of image denoising cuisine. Acta Numerica 21:475–576
Lebrun M, Colom M, Morel JM (2015) The noise clinic: a blind image denoising algorithm. Image Process On Line 5:1–54
Liu C, Freeman W (2010) A high-quality video denoising algorithm based on reliable motion estimation. In: European conference on computer vision. Springer, pp 706–719
Liu C, Szeliski R, Kang S, Zitnick C, Freeman W (2008) Automatic estimation and removal of noise from a single image. IEEE Trans Pattern Anal Mach Intell 30(2):299–314
Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Maggioni M, Boracchi G, Foi A, Egiazarian K (2011) Video denoising using separable 4D nonlocal spatiotemporal transforms. In: IS&T/SPIE electronic imaging. International Society for Optics and Photonics, pp 787,003–787,003
Mairal J, Sapiro G, Elad M (2008) Learning multiscale sparse representations for image and video restoration. SIAM Multiscale Model Simul 7(1):214–241
Mäkitalo M, Foi A (2013) Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise. IEEE Trans Image Process 22(1):91–103
Orchard J, Ebrahimi M, Wong A (2008) Efficient non-local-means denoising using the SVD. In: Proceedings of the IEEE international conference on image processing
Ozkan MK, Sezan MI, Tekalp AM (1993) Adaptive motion-compensated filtering of noisy image sequences. IEEE Trans Circuits Syst Video Technol 3(4):277–290
Papenberg N, Bruhn A, Brox T, Didas S, Weickert J (2006) Highly accurate optic flow computation with theoretically justified warping. Int J Comput Vis 67(2):141–158
Ponomarenko NN, Lukin VV, Abramov SK, Egiazarian KO, Astola JT (2003) Blind evaluation of additive noise variance in textured images by nonlinear processing of block DCT coefficients. In: Electronic imaging. International Society for Optics and Photonics, pp 178–189
Rudin L, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms. Phys D 60(1–4):259–268
Sand P, Teller S (2008) Particle video: long-range motion estimation using point trajectories. Int J Comput Vis 80(1):72–91
Smith S, Brady J (1997) SUSAN: a new approach to low level image processing. Int J Comput Vis 23(1):45–78
Szeliski R (2006) Image alignment and stitching: a tutorial. Found Trends Comput Gr Vis 2(1):1–104
Tomasi C, Manduchi R (1998) Bilateral filtering for gray and color images. In: Sixth international conference on computer vision, pp 839–846
Wang YQ, Morel JM (2013) Sure guided gaussian mixture image denoising. SIAM J Imaging Sci 6(2):999–1034
Wedel A, Pock T, Zach C, Bischof H, Cremers D (2009) An improved algorithm for TV-L1 optical flow. In: Statistical and geometrical approaches to visual motion analysis. Springer, pp 23–45
Wen B, Ravishankar S, Bresler Y (2015) Video denoising by online 3d sparsifying transform learning. In: 2015 IEEE international conference on image processing (ICIP). IEEE, pp 118–122
Xu Q, Jiang H, Scopigno R, Sbert M (2010) A new approach for very dark video denoising and enhancement. In: 17th IEEE international conference on image processing. IEEE, pp 1185–1188
Yaroslavsky L, Egiazarian K, Astola J (2001) Transform domain image restoration methods: review, comparison, and interpretation. Proc SPIE 4304:155
Yaroslavsky LP (1985) Digital picture processing. Springer, New York Inc, Secaucus, NJ, USA
Yue H, Sun X, Yang J, Wu F (2015) Image denoising by exploring external and internal correlations. IEEE Trans Image Process 24(6):1967–1982
Zach C, Pock T, Bischof H (2007) A duality based approach for realtime TV-L1 optical flow. In: Pattern recognition. Springer, pp 214–223
Zhang L, Dong W, Zhang D, Shi G (2010) Two-stage image denoising by principal component analysis with local pixel grouping. Pattern Recognit 43(4):1531–1549
Zhang M, Gunturk BK (2008) Multiresolution bilateral filtering for image denoising. IEEE Trans Image Process 17(12):2324–2333. https://doi.org/10.1109/TIP.2008.2006658
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Buades, A., Lisani, J.L. (2018). Patch-Based Methods for Video Denoising. In: BertalmÃo, M. (eds) Denoising of Photographic Images and Video. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-96029-6_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-96029-6_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96028-9
Online ISBN: 978-3-319-96029-6
eBook Packages: Computer ScienceComputer Science (R0)