Streaming Hierarchical Video Segmentation

  • Chenliang Xu
  • Caiming Xiong
  • Jason J. Corso
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7577)


The use of video segmentation as an early processing step in video analysis lags behind the use of image segmentation for image analysis, despite many available video segmentation methods. A major reason for this lag is simply that videos are an order of magnitude bigger than images; yet most methods require all voxels in the video to be loaded into memory, which is clearly prohibitive for even medium length videos. We address this limitation by proposing an approximation framework for streaming hierarchical video segmentation motivated by data stream algorithms: each video frame is processed only once and does not change the segmentation of previous frames. We implement the graph-based hierarchical segmentation method within our streaming framework; our method is the first streaming hierarchical video segmentation method proposed. We perform thorough experimental analysis on a benchmark video data set and longer videos. Our results indicate the graph-based streaming hierarchical method outperforms other streaming video segmentation methods and performs nearly as well as the full-video hierarchical graph-based method.


Segmentation Result Approximation Framework Subsequence Versus Video Segmentation Hierarchical Segmentation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Megret, R., DeMenthon, D.: A Survey of Spatio-Temporal Grouping Techniques. Technical report, Language and Media Proc. Lab., U. of MD at College Park (2002)Google Scholar
  2. 2.
    Laptev, I.: On space-time interest points. IJCV (2005)Google Scholar
  3. 3.
    Grundmann, M., Kwatra, V., Han, M., Essa, I.: Efficient hierarchical graph-based video segmentation. In: CVPR (2010)Google Scholar
  4. 4.
    Brendel, W., Todorovic, S.: Video object segmentation by tracking regions. In: ICCV (2009)Google Scholar
  5. 5.
    Lezama, J., Alahari, K., Sivic, J., Laptev, I.: Track to the future: Spatio-temporal video segmentation with long-range motion cues. In: CVPR (2011)Google Scholar
  6. 6.
    Vazquez-Reina, A., Avidan, S., Pfister, H., Miller, E.: Multiple Hypothesis Video Segmentation from Superpixel Flows. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 268–281. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Wang, J., Xu, Y., Shum, H., Cohen, M.F.: Video tooning. In: ACM SIGGRAPH, pp. 574–583 (2004)Google Scholar
  8. 8.
    Bai, X., Sapiro, G.: Geodesic matting: A framework for fast interactive image and video segmentation and matting. IJCV 82(2), 113–132 (2009)CrossRefGoogle Scholar
  9. 9.
    Paris, S., Durand, F.: A topological approach to hierarchical segmentation using mean shift. In: CVPR (2007)Google Scholar
  10. 10.
    Xu, C., Corso, J.J.: Evaluation of super-voxel methods for early video processing. In: CVPR (2012)Google Scholar
  11. 11.
    Sharon, E., Galun, M., Sharon, D., Basri, R., Brandt, A.: Hierarchy and adaptivity in segmenting visual scenes. Nature 442(7104), 810–813 (2006)CrossRefGoogle Scholar
  12. 12.
    Corso, J.J., Sharon, E., Dube, S., El-Saden, S., Sinha, U., Yuille, A.: Efficient multilevel brain tumor segmentation with integrated bayesian model classification. TMI 27(5), 629–640 (2008)Google Scholar
  13. 13.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient Graph-Based Image Segmentation. IJCV 59(2), 167–181 (2004)CrossRefGoogle Scholar
  14. 14.
    Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the nyström method. TPAMI 26, 2004 (2004)Google Scholar
  15. 15.
    Paris, S.: Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 460–473. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: CVPR (2011)Google Scholar
  17. 17.
    Ren, X., Malik, J.: Learning a classification model for segmentation. In: ICCV, vol. 1, pp. 10–17 (2003)Google Scholar
  18. 18.
    Levinshtein, A., Stere, A., Kutulakos, K.N., Fleet, D.J., Dickinson, S.J., Siddiqi, K.: Turbopixels: Fast superpixels using geometric flows. TPAMI 31(12), 2290–2297 (2009)CrossRefGoogle Scholar
  19. 19.
    Veksler, O., Boykov, Y., Mehrani, P.: Superpixels and Supervoxels in an Energy Optimization Framework. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 211–224. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  20. 20.
    Moore, A.P., Prince, S.J.D., Warrell, J., Mohammed, U., Jones, G.: Superpixel lattices. In: CVPR (2008)Google Scholar
  21. 21.
    Mori, G., Ren, X., Efros, A.A., Malik, J.: Recovering human body configurations: Combining segmentation and recognition. In: CVPR, vol. 2, pp. 326–333 (2004)Google Scholar
  22. 22.
    Tighe, J., Lazebnik, S.: SuperParsing: Scalable Nonparametric Image Parsing with Superpixels. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 352–365. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  23. 23.
    Lee, Y.J., Kim, J., Grauman, K.: Key-segments for video object segmentation. In: ICCV (2011)Google Scholar
  24. 24.
    Muthukrishnan, S.: Data streams: Algorithms and applications. Foundations and Trends in Theoretical Computer Science 1(2) (2005)Google Scholar
  25. 25.
    Chen, A.Y.C., Corso, J.J.: Propagating multi-class pixel labels throughout video frames. In: Proc. of Western NY Image Proc. Workshop (2010)Google Scholar
  26. 26.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: TextonBoost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV 81(2), 2–23 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Chenliang Xu
    • 1
  • Caiming Xiong
    • 1
  • Jason J. Corso
    • 1
  1. 1.Computer Science and EngineeringSUNY at BuffaloUSA

Personalised recommendations