Abstract
This paper tackles the problem of scalable video indexing. We propose a new framework combining spatial and motion patch descriptors. The spatial descriptors are based on a multiscale description of the image and are called Sparse Multiscale Patches. We propose motion patch descriptors based on block motion that describe the motion in a Group of Pictures. The distributions of these sets of patches are compared combining weighted Kullback-Leibler divergences between spatial and motion patches. These divergences are estimated in a non-parametric framework using a k-th Nearest Neighbor estimator. We evaluate this weighted dissimilarity measure on selected videos from the ICOS-HD ANR project. Experiments show that the spatial part of the measure is relevant to detect different sequences, while its motion part allows to detect clips within a sequence. Experiments combining the spatial and temporal parts of the dissimilarity measure show its robustness to resampling and compression; thus exhibiting the spatial scalability of the method on heterogeneous networks.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
ICOS-HD (Scalable Joint Indexing and Compression for High-Definition Video Content) is a research project funded by ANR (French Research Agency).
Original HD sequences ©Warner Bros issued from the Dolby 4-4-4 Film Content Kit One.
References
Boltz S, Debreuve E, Barlaud M (2007) High-dimensional kullback-leibler distance for region-of-interest tracking: application to combining a soft geometric constraint with radiometry. In: CVPR. Minneapolis, USA
Hero AO, Ma B, Michel O, Gorman J (2001) Alpha-divergence for classification, indexing and retrieval. Technical report CSPL-328, University of Michigan
Laptev I, Pérez P (2007) Retrieving actions in movies. In: Proc. int. conf. comp. vis. (ICCV’07). Rio de Janeiro, Brazil, pp 1–8
Loftsgaarden D, Quesenberry C (1965) A nonparametric estimate of a multivariate density function. AMS 36:1049–1051
Mezaris V, Kompatsiaris I, Strintzis MG (2006) Object-based mpeg-2 video indexing and retrieval in a collaborative environment. Multimed Tools Appl 30:255–272
Morand C, Benois-Pineau J, Domenger J-Ph, Mansencal B (2007) Object-based indexing of compressed video content: from sd to hd video. In: IEEE VMDL/ICIAP. Modena, Italy
Piro P, Anthoine S, Debreuve E, Barlaud M (2008) Image retrieval via kullback-leibler divergence of patches of multiscale coefficients in the knn framework. In: CBMI. London, UK
Piro P, Anthoine S, Debreuve E, Barlaud M (2009) Sparse multiscale patches for image processing. In: ETVC. LNCS, vol 5416/2009. Springer, New York
Rothganger F, Lazebnik S, Schmid C, Ponce J (2007) Segmenting, modeling, and matching video clips containing multiple moving objects. IEEE Trans Pattern Anal Mach Intell 29(3):477–491
Schwarz H, Marpe D, Wiegand T (2007) Overview of the scalable video coding extension of the h.264/avc standard. IEEE Trans Circuits Syst Video Technol 17(9):1103–1120
Terrell DW, Scott GR (1992) Variable kernel density estimation. Ann Stat 20(3):1236–1265
Zhai Y, Liu J, Cao X, Basharat A, Hakeem A, Ali S, Shah M (2005) Video understanding and content-based retrieval. In: TRECVID05
Zong D, Chang SF (1999) An integrated approach for content-based video object segmentation and retrieval. IEEE Trans Circuits Syst Video Technol 9:1259–1268
Acknowledgements
The authors would like to acknowledge the contribution of W. Belhajali in the experiments conducted here.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work is supported by the French ANR grant “ICOS-HD”.
Rights and permissions
About this article
Cite this article
Piro, P., Anthoine, S., Debreuve, E. et al. Combining spatial and temporal patches for scalable video indexing. Multimed Tools Appl 48, 89–104 (2010). https://doi.org/10.1007/s11042-009-0350-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-009-0350-4