Multimedia Tools and Applications

, Volume 77, Issue 23, pp 31313–31329 | Cite as

On-line video multi-object segmentation based on skeleton model and occlusion detection

  • Guoheng Huang
  • Chi-Man PunEmail author


In this work, we propose an approach for on-line video multi object segmentation based on skeleton model and occlusion detection. We consider the multi-object segmentation in every frame as a multi-class region merging based object segmentation. We then generate the initial object superpixels automatically using a skeleton model from the second frame. Moreover, we also propose an initial background superpixel prediction scheme. In case the occlusion to affect the final segmentation result, we propose an occlusion detection model based on optical flow. The experimental results show that our method is both robust in segmenting multi objects and efficient in execution time.


Multi-object segmentation Skeleton model Occlusion detection Superpixel On-line 



This work was supported in part by the Research Committee of the University of Macau under Grants MYRG2015-00011-FST and MYRG2015-00012-FST, the Science and Technology Development Fund of Macau SAR under Grants 093/2014/A2 and 041/2017/A1, and the project (2018 - 2020, Video Multi-object Co-segmentation Based on Superpixel, National Natural Science Foundation of China (NSFC) Grant No. 61702111).


  1. 1.
    Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: Computer vision and pattern recognition. CVPR 2009. IEEE conference on, pp 1597–1604Google Scholar
  2. 2.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) SLIC Superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282CrossRefGoogle Scholar
  3. 3.
    Bai X, Wang J, Simons D, Sapiro G (2009) Video SnapCut: robust video object cutout using localized classifiers. ACM Trans Graph 28(3):1–11CrossRefGoogle Scholar
  4. 4.
    Bouguet J-Y (2000) Pyramidal implementation of the Lucas Kanade feature tracker: description of the algorithm. Available:
  5. 5.
    Brendel W, Todorovic S (2009) Video object segmentation by tracking regions. In: 2009 IEEE 12th international conference on computer vision, pp 833–840Google Scholar
  6. 6.
    Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619CrossRefGoogle Scholar
  7. 7.
    Dey TK, Zhao W (2002) Approximating the medial axis from the voronoi diagram with a convergence guarantee. In: Möhring R, Raman R (eds) Algorithms — ESA 2002: 10th annual European symposium Rome, Italy, September 17–21, 2002 proceedings. Springer, Berlin, pp 387–398CrossRefGoogle Scholar
  8. 8.
    Endres I, Hoiem D (2010) Category independent object proposals. In: Part V, Daniilidis K, Maragos P, Paragios N (eds) Computer vision – ECCV 2010: 11th European conference on computer vision, Heraklion, Crete, Greece, September 5–11, 2010, proceedings. Springer, Berlin, pp 575–588CrossRefGoogle Scholar
  9. 9.
    Grundmann M, Kwatra V, Han M, Essa I (2010) Efficient hierarchical graph-based video segmentation. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 2141–2148CrossRefGoogle Scholar
  10. 10.
    Horn BKP, Schunck BG (1980) Determining optical flow. Massachusetts Institute of Technology, CambridgeGoogle Scholar
  11. 11.
    Jian M, Lam K-M, Dong J (2014) Facial-feature detection and localization based on a hierarchical scheme. Inf Sci 262:1–14CrossRefGoogle Scholar
  12. 12.
    Jian M, Lam KM, Dong J, Shen L (2015) Visual-patch-attention-aware saliency detection. IEEE Trans Cybern 45(8):1575–1586CrossRefGoogle Scholar
  13. 13.
    Jian M, Qi Q, Dong J, Sun X, Sun Y, Lam K-M (2017) Saliency detection using quaternionic distance based weber local descriptor and level priors. Multimed Tools Appl 1–18. CrossRefGoogle Scholar
  14. 14.
    Jian M, Qi Q, Dong J, Yin Y, Lam KM (2018) Integrating QDWD with pattern distinctness and local contrast for underwater saliency detection. J Vis Commun Image Represent 53:31–41CrossRefGoogle Scholar
  15. 15.
    Jian MW, Dong JY, Ma J (2011) Image retrieval using wavelet-based salient regions. Imaging Sci J 59(4):219–231CrossRefGoogle Scholar
  16. 16.
    Lee YJ, Kim J, Grauman K (2011) Key-segments for video object segmentation. In: 2011 International conference on computer vision, pp 1995–2002Google Scholar
  17. 17.
    Levinshtein A, Stere A, Kutulakos KN, Fleet DJ, Dickinson SJ, Siddiqi K (2009) TurboPixels: fast Superpixels using geometric flows. IEEE Trans Pattern Anal Mach Intell 31(12):2290–2297CrossRefGoogle Scholar
  18. 18.
    Li F, Kim T, Humayun A, Tsai D, Rehg JM (2013) Video segmentation by tracking many figure-ground segments. In: 2013 IEEE international conference on computer vision, pp 2192–2199CrossRefGoogle Scholar
  19. 19.
    Ning J, Zhang L, Zhang D, Wu C (2010) Interactive image segmentation by maximal similarity based region merging. Pattern Recogn 43(2):445–456CrossRefGoogle Scholar
  20. 20.
    Ochs P, Malik J, Brox T (2014) Segmentation of moving objects by long term video analysis. IEEE Trans Pattern Anal Mach Intell 36(6):1187–1200CrossRefGoogle Scholar
  21. 21.
    Papazoglou A, Ferrari V (2013) Fast object segmentation in unconstrained video. In: 2013 IEEE international conference on computer vision, pp 1777–1784Google Scholar
  22. 22.
    Taylor B, Karasev V, Soattoc S (2015) Causal video object segmentation from persistence of occlusions. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4268–4276CrossRefGoogle Scholar
  23. 23.
    Varas D, Marques F (2014) Region-based particle filter for video object segmentation. In: 2014 IEEE conference on computer vision and pattern recognition, pp 3470–3477Google Scholar
  24. 24.
    Wenguan W, Jianbing S, Porikli F (2015) Saliency-aware geodesic video object segmentation. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3395–3402Google Scholar
  25. 25.
    Willert V, Eggert J, Clever S, Körner E (2005) Probabilistic color optical flow. In: Kropatsch WG, Sablatnig R, Hanbury A (eds) Pattern recognition: 27th DAGM symposium, Vienna, Austria, august 31–September 2, 2005. Proceedings. Springer, Berlin, pp 9–16CrossRefGoogle Scholar
  26. 26.
    Zhang X, Shan Y, Wei W, Zhu Z (2010) An image segmentation method based on improved watershed algorithm. In: Computational and information sciences (ICCIS), 2010 international conference on, pp 258–261Google Scholar
  27. 27.
    Zhang D, Javed O, Shah M (2013) Video object segmentation through spatially accurate and temporally dense extraction of primary object regions. In: 2013 IEEE conference on computer vision and pattern recognition (CVPR), pp 628–635CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Guangdong University of TechnologyGuangzhouChina
  2. 2.University of MacauMacauChina

Personalised recommendations