Advertisement

Labeling of partially occluded regions via the multi-layer CRF

  • Sergey Kosov
  • Kimiaki Shirahama
  • Marcin Grzegorzek
Article
  • 32 Downloads

Abstract

This work proposes a general multi-layer framework for image labeling, which targets the challenging problem of classifying the occluded parts of the 3D scene depicted in a 2D image. Our framework is based on the mixed graphical models, which explicitly encode causal relationship between the visible and occluded regions. Unlike other image labeling techniques where a single label is determined for each pixel, layered model assigns multiple labels to pixels. We propose a novel “Multi-Layer-CRF” framework that allows for the integration of sophisticated occlusion potentials into the model and enables the automatic inference of the layer decomposition. We use a special message-passing algorithm to perform maximum a posterior inference on mixed graphs and demonstrate the ability to infer the correct labels of occluded regions in both the aerial near-vertical dataset and urban street-view dataset. It is shown to increase the classification accuracy in occluded areas significantly.

Keywords

Conditional random fields Graphical models Classification Semantic segmentation Occlusions 

Notes

References

  1. 1.
    Bileschi SM (2006) Streetscenes: towards scene understanding in still images. Ph.D. thesis MITGoogle Scholar
  2. 2.
    Boykov Y, Jolly MP (2001) Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In: Proceedings of ICCV, pp 105–112Google Scholar
  3. 3.
    Breiman L. (2001) Random forests. Mach LearnGoogle Scholar
  4. 4.
    Chen YT, Liu X, Yang MH (2015) Multi-instance object segmentation with occlusion handling. In: Proceedings of CVPR, pp 3470–3478Google Scholar
  5. 5.
    Cramer M (2010) The DGPF test on digital aerial camera evaluation - overview and test design. PFG 2(2010):73–82Google Scholar
  6. 6.
    Criminisi A, Shotton J (2013) Decision forests in computer vision and medical image analysis. Springer, BerlinCrossRefGoogle Scholar
  7. 7.
    Duda R, Hart P (1972) Use of the hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15CrossRefzbMATHGoogle Scholar
  8. 8.
    Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645CrossRefGoogle Scholar
  9. 9.
    Frydenberg M (1990) The chain graph Markov property. Scand J Stat 17:333–353MathSciNetzbMATHGoogle Scholar
  10. 10.
    Guo R, Hoiem D (2012) Beyond the line of sight: labeling the underlying surfaces. In: Proceedings of ECCV, pp 761–774Google Scholar
  11. 11.
    Guo R, Hoiem D (2015) Labeling complete surfaces in scene understanding. IJCV 112(2):172–187MathSciNetCrossRefGoogle Scholar
  12. 12.
    Heitz G, Gal C (2010) Object separation in x-ray image sets. In: Proceedings of CVPR, pp 2093–2100Google Scholar
  13. 13.
    Hinz S, Baumgartner A (2003) Automatic extraction of urban road networks from multi-view aerial imagery. ISPRS J Photogramm Remote Sens 58:83–98CrossRefGoogle Scholar
  14. 14.
    Hoiem D, Efros A, Hebert M (2005) Geometric context from a single image. In: Proceedings of ICCV, pp 654–661Google Scholar
  15. 15.
    Kim BS, Kohli P, Savarese S (2013) 3D scene understanding by voxel-CRF. In: Proceedings of ICCV, pp 1425–1432Google Scholar
  16. 16.
    Kolmogorov V (2006) Convergent tree-reweighted message passing for energy minimization. IEEE Trans Pattern Anal Mach Intell 28(10):1568–1583CrossRefGoogle Scholar
  17. 17.
    Kosov S (2015) Direct graphical models C++ library. http://research.project-10.de/dgm/
  18. 18.
    Kosov S, Kohli P, Rottensteiner F, Heipke C (2013) A two-layer conditional random field for the classification of partially occluded objects. arXiv:1307.3043 [cs.CV]
  19. 19.
    Kosov S, Rottensteiner F, Heipke C, Leitloff J, Hinz S. (2012) 3D classification of crossroads from multiple aerial images using Markov random fields. In: Proceedings of ISPRS Congress, pp 479–484Google Scholar
  20. 20.
    Kramer O (2010) Iterated local search with Powell’s method: a memetic algorithm for continuous global optimization. Memetic Comput 2(1):69–83CrossRefGoogle Scholar
  21. 21.
    Kulkarni VY, Sinha PK (2013) Random forest classifiers: a survey and future research directions. Int J Adv Comput 36(1):1144–1153Google Scholar
  22. 22.
    Kumar S, Hebert M (2005) A hierarchical field framework for unified context-based classification. In: Proceedings of ICCV, pp 1284–1291Google Scholar
  23. 23.
    Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of ICML, pp 282–289Google Scholar
  24. 24.
    Lauritzen S, Wermuth N (1989) Graphical models for associations between variables, some of which are qualitative and some quantitative. Ann Stat 17(1):31–57MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Leibe B, Leonardis A, Schiele B (2008) Robust object detection with interleaved categorization and segmentation. IJCV 77:259–289CrossRefGoogle Scholar
  26. 26.
    Li SZ (2009) Markov random field modeling in image analysis, 3rd edn. Springer, BerlinzbMATHGoogle Scholar
  27. 27.
    Liu C, Kohli P, Furukawa Y (2016) Layered scene decomposition via the occlusion-CRF. In: Proceedings of CVPR, pp 165–173Google Scholar
  28. 28.
    Liu C, Yuen J, Torralba A (2011) Nonparametric scene parsing via label transfer. IEEE Trans Pattern Anal Mach Intell 33(12):2368–2382CrossRefGoogle Scholar
  29. 29.
    Prasad M, Zisserman A, Fitzgibbon A, Kumar P, Torr P (2006) Learning class-specific edges for object detection and segmentation. In: Proceedings of ICVGIP, pp 94–105Google Scholar
  30. 30.
    Richardson T, Spirtes P (2002) Ancestral graph Markov models. Ann Stat 30 (4):962–1030MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Schindler K (2012) An overview and comparison of smooth labeling methods for land-cover classification. IEEE Trans Geosci Remote Sens:4534–4545Google Scholar
  32. 32.
    Schnitzspan P, Fritz M, Roth S, Schiele B (2009) Discriminative structure learning of hierarchical representations for object detection. In: Proceedings of CVPR, pp 2238–2245Google Scholar
  33. 33.
    Silberman N, Shapira L, Gal R, Kohli P (2014) A contour completion model for augmenting surface reconstructions. In: Proceedings of ECCV, pp 488–503Google Scholar
  34. 34.
    Souly N, Shah M (2016) Scene labeling using sparse precision matrix. In: Proceedings of CVPR. IEEE Computer Society, pp 3650–3658Google Scholar
  35. 35.
    Souly N, Spampinato C, Shah M (2017) Semi supervised semantic segmentation using generative adversarial network. In: Proceedings of ICCV, pp 5689–5697Google Scholar
  36. 36.
    Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE Trans Pattern Anal Mach Intell 30(6):1068–1080CrossRefGoogle Scholar
  37. 37.
    Tighe J, Niethammer M, Lazebnik S (2014) Scene parsing with object instances and occlusion ordering. In: Proceedings of CVPR, pp 3748–3755Google Scholar
  38. 38.
    Tu Z, Bai X (2010) Auto-context and its application to high-level vision tasks and 3D brain image segmentation. IEEE Trans Pattern Anal Mach Intell 32(10):1744–1757CrossRefGoogle Scholar
  39. 39.
    Vishwanathan S, Schraudolph N, Schmidt M, Murphy K (2006) Accelerated training of conditional random fields with stochastic gradient methods. In: Proceedings of ICML, pp 969–976Google Scholar
  40. 40.
    Winn J, Shotton J (2006) The layout consistent random field for recognizing and segmenting partially occluded objects. In: Proceedings of CVPR, pp 37–44Google Scholar
  41. 41.
    Yang Y, Hallman S, Ramanan D, Fowlkes C (2012) Layered object models for image segmentation. IEEE Trans Pattern Anal Mach Intell 34:1731–1743CrossRefGoogle Scholar
  42. 42.
    Yin Z, Collins R (2007) Belief propagation in a 3D spatiotemporal MRF for moving object detection. In: Proceedings of CVPR, pp 1–8Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Research Group for Pattern Recognition, Department ETIUniversity of SiegenSiegenGermany

Personalised recommendations