Segmentation in Point Clouds from RGB-D Using Spectral Graph Reduction

  • Margret KeuperEmail author
  • Thomas Brox
Conference paper
Part of the Mathematics and Visualization book series (MATHVISUAL)


In this chapter, we tackle the problem of segmentation in point clouds from RGB-D data. In contrast to full point clouds, RGB-D data only provides a part of the volumetric information, the depth information of the one view given in the corresponding RGB image. Still, this additional information is valuable for the segmentation task as it helps disambiguating texture gradients from structure gradients. In order to create hierarchical segmentations, we combine a state-of-the-art method for natural RGB image segmentation based on spectral graph analysis with an RGB-D boundary detector. We show that spectral graph reduction can be employed in this case, facilitating the computation of RGB-D segmentations in large datasets.


Point Cloud Image Segmentation Random Forest Spectral Cluster Image Patch 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We acknowledge funding by the ERC Starting Grant VideoLearn.


  1. 1.
    Arbeláez, P., Maire, M., Fowlkes, C.C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE TPAMI 33 (5), 898–916 (2011)CrossRefGoogle Scholar
  2. 2.
    Arbeláez, P., Pont-Tuset, J., Barron, J., Marques, F., Malik, J.: Multiscale combinatorial grouping. In: CVPR, Columbus (2014)CrossRefGoogle Scholar
  3. 3.
    Bertasius, G., Shi, J., Torresani, L.: Deepedge: a multi-scale bifurcated deep network for top-down contour detection. CoRR (2014). abs/1412.1123Google Scholar
  4. 4.
    Canny, J.: A computational approach to edge detection. IEEE TPAMI 8, 679–698 (1986)CrossRefGoogle Scholar
  5. 5.
    Chen, X., Cai, D.: Large scale spectral clustering with landmark-based representation. In: AAAI, San Francisco (2011)Google Scholar
  6. 6.
    Chung, F.R.K.: Spectral graph theory, vol. 92. CBMS, Providence (1997)zbMATHGoogle Scholar
  7. 7.
    Dollár, P., Zitnick, C.L.: Structured forests for fast edge detection. In: ICCV, Sydney (2013)CrossRefGoogle Scholar
  8. 8.
    Dollár, P., Zitnick, C.L.: Fast edge detection using structured forests. CoRR (2014). abs/1406.5549Google Scholar
  9. 9.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. IJCV 59 (2), 167–181 (2004)CrossRefGoogle Scholar
  10. 10.
    Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the nyström method. IEEE TPAMI 26(2), 214–225 (2004)CrossRefGoogle Scholar
  11. 11.
    Galasso, F., Keuper, M., Brox, T., Schiele, B.: Spectral graph reduction for efficient image and streaming video segmentation. In: CVPR, Columbus (2014)CrossRefGoogle Scholar
  12. 12.
    Gupta, S., Arbelaez, P., Malik, J.: Perceptual organization and recognition of indoor scenes from RGB-D images. In: CVPR, Portland (2013)CrossRefGoogle Scholar
  13. 13.
    Gupta, S., Girshick, R., Arbelaez, P., Malik, J.: Learning rich features from RGB-D images for object detection and segmentation. In: ECCV, Zurich (2014)CrossRefGoogle Scholar
  14. 14.
    Isola, P., Zoran, D., Krishnan, D., Adelson, E.H.: Crisp boundary detection using pointwise mutual information. In: ECCV, Zurich (2014)CrossRefGoogle Scholar
  15. 15.
    Karpathy, A., Miller, S., Fei-Fei, L.: Object discovery in 3d scenes via shape analysis. In: ICRA, Karlsruhe (2013)CrossRefGoogle Scholar
  16. 16.
    Keuper, M., Bensch, R., Voigt, K., Dovzhenko, A., Palme, K., Burkhardt, H., Ronneberger, O.: Semi-supervised learning of edge filters for volumetric image segmentation. In: 32nd DAGM Symposium, Darmstadt. LNCS, pp. 462–471. Springer (2010)Google Scholar
  17. 17.
    Keuper, M., Padeken, J., Heun, P., Burkhardt, H., Ronneberger, O.: Mean shift gradient vector flow: a robust external force field for 3d active surfaces. In: ICPR, Istanbul (2010)Google Scholar
  18. 18.
    Keuper, M., Schmidt, T., Rodriguez-Franco, M., Schamel, W., Brox, T., Burkhardt, H., Ronneberger, O.: Hierarchical markov random fields for mast cell segmentation in electron microscopic recordings. In: ISBI, Chicago (2011)CrossRefGoogle Scholar
  19. 19.
    Kim, B., Kohli, P., Savarese, S.: 3d scene understanding by voxel-CRF. In: ICCV, Sydney (2013)CrossRefGoogle Scholar
  20. 20.
    Kundu, A., Li, Y., Dellaert, F., Li, F., Rehg, J.M.: Joint semantic segmentation and 3d reconstruction from monocular video. In: ECCV, Zurich (2014)CrossRefGoogle Scholar
  21. 21.
    Li, L., Socher, R., Fei-Fei, L.: Towards total scene understanding: classification, annotation and segmentation in an automatic framework. In: CVPR, Miami (2009)Google Scholar
  22. 22.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color and texture cues. IEEE TPAMI 26 (5), 530–549 (2004)CrossRefGoogle Scholar
  23. 23.
    Morath, V., Keuper, M., Rodriguez-Franco, M., Deswal, S., Fiala, G., Blumenthal, B., Kaschek, D., Timmer, J., Neuhaus, G., Ehl, S., Ronneberger, O., Schamel, W.: Semi-automatic determination of cell surface areas used in systems biology. Front. Biosci. Elite 5, 533–545 (2013)CrossRefGoogle Scholar
  24. 24.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: NIPS, Vancouver (2001)Google Scholar
  25. 25.
    Rangapuram, S.S., Hein, M.: Constrained 1-spectral clustering. In: AISTATS, La Palma (2012)Google Scholar
  26. 26.
    Ren, X., Bo, L.: Discriminatively trained sparse code gradients for contour detection. In: NIPS, Lake Tahoe (2012)Google Scholar
  27. 27.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. In: CVPR, San Juan (1997)Google Scholar
  28. 28.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE TPAMI 22 (8), 888–905 (2000)CrossRefGoogle Scholar
  29. 29.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: ECCV, Florence (2012)CrossRefGoogle Scholar
  30. 30.
    Taylor, C.J.: Towards fast and accurate segmentation. In: CVPR, Portland (2013)CrossRefGoogle Scholar
  31. 31.
    von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17 (4), 395-416 (2007)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Weiss, Y.: Segmentation using eigenvectors: a unifying view. In: ICCV, Corfu (1999)Google Scholar
  33. 33.
    Xiao, J., Owens, A., Torralba, A.: Sun3d: a database of big spaces reconstructed using sfm and object labels. In: ICCV, Sydney (2013)Google Scholar
  34. 34.
    Zheng, B., Zhao, Y., Yu, J.C., Ikeuchi, K., Zhu, S.: Beyond point clouds: scene understanding by reasoning geometry and physics. In: CVPR, Portland (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.University of FreiburgFreiburg im BreisgauGermany

Personalised recommendations