Advertisement

Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams

  • Sylvain Paris
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5303)

Abstract

Video streams are ubiquitous in applications such as surveillance, games, and live broadcast. Processing and analyzing these data is challenging because algorithms have to be efficient in order to process the data on the fly. From a theoretical standpoint, video streams have their own specificities – they mix spatial and temporal dimensions, and compared to standard video sequences, half of the information is missing, i.e. the future is unknown. The theoretical part of our work is motivated by the ubiquitous use of the Gaussian kernel in tools such as bilateral filtering and mean-shift segmentation. We formally derive its equivalent for video streams as well as a dedicated expression of isotropic diffusion. Building upon this theoretical ground, we adapt a number of classical algorithms to video streams: bilateral filtering, mean-shift segmentation, and anisotropic diffusion.

Keywords

Video Stream Video Streaming Anisotropic Diffusion Partial Derivative Equation Temporal Coherence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

Supplementary material(26,420 KB)

References

  1. 1.
    Zheng, Y., Lin, S., Kang, S.B.: Single-image vignetting correction. In: Proc. of the conf. on Computer Vision and Pattern Recognition, vol. 1, pp. 461–468. IEEE, Los Alamitos (2006)Google Scholar
  2. 2.
    Bennett, E.P., McMillan, L.: Video enhancement using per-pixel virtual exposures. ACM Transactions on Graphics 24, 845–852 (2005); Proc. of the ACM SIGGRAPH conf.CrossRefGoogle Scholar
  3. 3.
    Bennett, E.P., Mason, J.L., McMillan, L.: Multispectral bilateral video fusion. IEEE Transactions on Image Processing 16, 1185–1194 (2007)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Liu, C., Freeman, W.T., Szeliski, R., Kang, S.: Noise estimation from a single image. In: Proc. of the Computer Vision and Pattern Recognition Conf. IEEE, Los Alamitos (2006)Google Scholar
  5. 5.
    Li, Y., Sun, J., Shum, H.Y.: Video object cut and paste. ACM Transactions on Graphics 24, 595–600 (2005); Proc. of the ACM SIGGRAPH conf.CrossRefGoogle Scholar
  6. 6.
    Wang, J., Bhat, P., Colburn, R.A., Agrawala, M., Cohen, M.F.: Video cutout. ACM Transactions on Graphics 24 (2005); Proc. of the ACM SIGGRAPH conf.Google Scholar
  7. 7.
    Bae, S., Paris, S., Durand, F.: Two-scale tone management for photographic look. ACM Transactions on Graphics 25, 637–645 (2006); Proc. of the ACM SIGGRAPH conf. CrossRefGoogle Scholar
  8. 8.
    DeCarlo, D., Santella, A.: Stylization and abstraction of photographs. In: Proc. of the ACM SIGGRAPH conf. (2002)Google Scholar
  9. 9.
    Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Transactions on Graphics 25, 1221–1226 (2006); Proc. of the ACM SIGGRAPH conf. CrossRefGoogle Scholar
  10. 10.
    Chen, J., Paris, S., Durand, F.: Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics 26 (2007); Proc. of the ACM SIGGRAPH conf. Google Scholar
  11. 11.
    Wang, J., Xu, Y., Shum, H.Y., Cohen, M.F.: Video tooning. ACM Transactions on Graphics 23, 294–302 (2004); Proc. of the ACM SIGGRAPH conf.CrossRefGoogle Scholar
  12. 12.
    Wang, J., Thiesson, B., Xu, Y., Cohen, M.F.: Image and video segmentation by anisotropic mean shift. In: Proc. of the European Conf. on Computer Vision (2004)Google Scholar
  13. 13.
    Bousseau, A., Neyret, F., Thollot, J., Salesin, D.: Video watercolorization using bidirectional texture advection. ACM Transactions on Graphics 26 (2007); Proc. of the ACM SIGGRAPH conf. Google Scholar
  14. 14.
    Paris, S., Durand, F.: A topological approach to hierarchical segmentation using mean shift. In: Proc. of the IEEE conf. on Computer Vision and Pattern Recognition (2007)Google Scholar
  15. 15.
    Buades, A., Coll, B., Morel, J.M.: A non local algorithm for image denoising. In: Proc. of the conf. on Computer Vision and Pattern Recognition (2005)Google Scholar
  16. 16.
    Chen, J., Tang, C.K.: Spatio-temporal Markov random field for video denoising. In: Proc. of the IEEE conf. on Computer Vision and Pattern Recognition (2007)Google Scholar
  17. 17.
    Drori, I., Leyvand, T., Fleishman, S., Cohen-Or, D., Yeshurun, H.: Video operations in the gradient domain. Technical report, Tel-Aviv University (2004)Google Scholar
  18. 18.
    Chuang, Y.Y., Agarwala, A., Curless, B., Salesin, D., Szeliski, R.: Video matting of complex scenes. ACM Transactions on Graphics 21 (2002); Proc. of the ACM SIGGRAPH conf.Google Scholar
  19. 19.
    DeMenthon, D.: Spatio-temporal segmentation of video by hierarchical mean shift analysis. In: Proc. of the Statistical Methods in Video Processing Workshop (2002)Google Scholar
  20. 20.
    Zitnick, C.L., Jojic, N., Kang, S.B.: Consistent segmentation for optical flow estimation. In: Proc. of the International Conf. on Computer Vision (2005)Google Scholar
  21. 21.
    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. In: Proc. of the European Conf. on Computer Vision (2006)Google Scholar
  22. 22.
    Comaniciu, D., Meer, P.: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis Machine Intelligence 24, 603–619 (2002)CrossRefGoogle Scholar
  23. 23.
    Aubert, G., Kornprobst, P.: Mathematical problems in image processing: Partial Differential Equations and the Calculus of Variations. Applied Mathematical Sciences, vol. 147. Springer, Heidelberg (2002)zbMATHGoogle Scholar
  24. 24.
    Koenderink, J.J.: Scale-time. Biological Cybernetics 58 (1988)Google Scholar
  25. 25.
    ter Haar Romeny, B.M., Florack, L.M.J., Nielsen, M.: Scale-time kernels and models. In: Proc. of the conf. on Scale-Space and Morphology in Computer Vision (2001)Google Scholar
  26. 26.
    Lindeberg, T., Fagerström, D.: Scale-space with causal time direction. In: Proc. of European Conf. on Computer Vision (1996)Google Scholar
  27. 27.
    Lindeberg, T.: Linear spatio-temproal scale-space. In: Proc. of the International Conf. on Scale-Space Teory in Computer Vision (1997)Google Scholar
  28. 28.
    Klein, A., Sloan, P.P., Finkelstein, A., Cohen, M.F.: Stylized video cubes. In: Proc. of the ACM SIGGRAPH Symposium on Computer Animation (2002)Google Scholar
  29. 29.
    Kim, J., Woods, J.W.: Spatiotemporal adaptive 3-D Kalman filter for video. IEEE Transactions on Image Processing 6 (1997)Google Scholar
  30. 30.
    Patti, A.J., Tekalp, A.M., Sezan, M.I.: A new motion-compensated reduced-order model Kalman filter for space-varying restoration of progressive and interlaced video. IEEE Transactions on Image Processing 7 (1998)Google Scholar
  31. 31.
    Bennett, E.P., McMillan, L.: Computational time-lapse video. ACM Transactions on Graphics 26 (2007); Proc. of the ACM SIGGRAPH conf. Google Scholar
  32. 32.
    Koenderink, J.J.: The structure of images. Biological Cybernetics 50 (1984)Google Scholar
  33. 33.
    Guo, H.Y., Li, Y.Q., Wu, K.: Difference discrete variational principle, Euler-Lagrange cohomology and symplectic, multisymplectic structures. ArXiv Math. Physics e-prints (2001)Google Scholar
  34. 34.
    Deriche, R.: Recursively implementating the Gaussian and its derivatives. Technical Report RR-1893, INRIA (1993)Google Scholar
  35. 35.
    Aurich, V., Weule, J.: Non-linear gaussian filters performing edge preserving diffusion. In: Proc. of the DAGM Symposium (1995)Google Scholar
  36. 36.
    Smith, S.M., Brady, J.M.: SUSAN – a new approach to low level image processing. International Journal of Computer Vision 23, 45–78 (1997)CrossRefGoogle Scholar
  37. 37.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proc. of the International Conf. on Computer Vision, pp. 839–846. IEEE, Los Alamitos (1998)Google Scholar
  38. 38.
    Paris, S., Kornprobst, P., Tumblin, J., Durand, F.: A gentle introduction to bilateral filtering and its applications. In: Course at the ACM SIGGRAPH conf. (2007)Google Scholar
  39. 39.
    Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Transactions Pattern Analysis Machine Intelligence 12, 629–639 (1990)CrossRefGoogle Scholar
  40. 40.
    Horn, B.K.P.: Determining lightness from an image. Computer Graphics and Image Processing 3 (1974)Google Scholar
  41. 41.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Transactions on Graphics 22 (2003); Proc. of the ACM SIGGRAPH conf. Google Scholar
  42. 42.
    Levin, A., Zomet, A., Peleg, S., Weiss, Y.: Seamless image stitching in the gradient domain. In: Proc. of the European Conf. on Computer Vision (2006)Google Scholar
  43. 43.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. In: Proc. of the International Conf. on Computer Vision (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Sylvain Paris
    • 1
  1. 1.Adobe Systems, Inc.USA

Personalised recommendations