Interactive Segmentation of Textured and Textureless Objects

  • Karol HausmanEmail author
  • Dejan Pangercic
  • Zoltán-Csaba Márton
  • Ferenc Bálint-Benczédi
  • Christian Bersch
  • Megha Gupta
  • Gaurav Sukhatme
  • Michael Beetz
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 42)


This article describes interactive object segmentation for autonomous service robots acting in human living environments. The proposed system allows a robot to effectively segment textured and textureless objects in cluttered scenes by leveraging its manipulation capabilities. In this interactive perception approach, RGB and depth (RGB-D) camera based features are tracked while the robot actively induces motions into a scene using its arm. The robot autonomously infers appropriate arm movements which can effectively separate objects. The resulting tracked feature trajectories are assigned to their corresponding object by clustering. In the final step, we reconstruct the dense models of the objects from the previously clustered sparse RGB-D features. The approach is integrated with robotic grasping and is demonstrated on scenes consisting of various textured and textureless objects, showing the advantages of a tight integration between perception, cognition and action.


Point Cloud Object Segmentation Rigid Transformation Model Hypothesis Feature Trajectory 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Bergström N, Ek CH, Björkman M, Kragic D (2011) Scene understanding through interactive perception. In: 8th international conference on computer vision systems (ICVS). Sophia AntipolisGoogle Scholar
  2. Bersch C, Pangercic D, Osentoski S, Hausman K, Marton ZC, Ueda R, Okada K, Beetz M (2012) Segmentation of textured and textureless objects through interactive perception. In: RSS workshop on robots in clutter: manipulation, perception and navigation in human environments. Sydney, AustraliaGoogle Scholar
  3. Bouguet JY (2001) Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel Corp 5:1–10Google Scholar
  4. Brox T, Malik J (2010) Object segmentation by long term analysis of point trajectories. In: Proceedings of the 11th European conference on computer vision: Part V (ECCV’10). Springer, Berlin, pp 282–295Google Scholar
  5. Chang L, Smith JR, Fox D (2012) Interactive singulation of objects from a pile. In: 2012 IEEE international conference on robotics and automation (ICRA). IEEE, pp 3875–3882Google Scholar
  6. Cohen LB, Cashon CH (2003) Infant perception and cognition. Comprehensive handbook of psychology, volume 6: developmental psychology. Wiley, New York, pp 65–89 (Chap II. Infancy)Google Scholar
  7. Felzenszwalb PF, Huttenlocher DP (2004) Efficient graph-based image segmentation. Int J Comput Vis 59(2):167–181. doi: 10.1023/B:VISI.0000022288.19776.77
  8. Fischler M, Bolles R (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24:381–395CrossRefMathSciNetGoogle Scholar
  9. Fitzpatrick P (2003) First contact: an active vision approach to segmentation. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS 2003), vol 3. IEEE, pp 2161–2166Google Scholar
  10. Gupta M, Sukhatme G (2012) Using manipulation primitives for brick sorting in clutter. In: 2012 IEEE international conference on robotics and automation (ICRA). IEEE, pp 3883–3889Google Scholar
  11. Gupta M, Ruhr T, Beetz M, Sukhatme GS (2013) Interactive environment exploration in clutter. In: 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 5265–5272Google Scholar
  12. Hausman K, Balint-Benczedi F, Pangercic D, Marton ZC, Ueda R, Okada K, Beetz M (2013) Tracking-based interactive segmentation of textureless objects. In: IEEE international conference on robotics and automation (ICRA). Karlsruhe, Germany, Best Service Robotics Paper Award FinalistGoogle Scholar
  13. Hausman K, Corcos C, Mueller J, Sha F, Sukhatme GS (2014) Towards interactive object recognition. In: 3rd workshop on robots in clutter: perception and interaction in clutter, IEEE/RSJ international conference on intelligent robots and systems (IROS). Chicago, ILGoogle Scholar
  14. Heyer LJ, Kruglyak S, Yooseph S (1999) Exploring expression data: identification and analysis of coexpressed genes. Genome Res 9(11):1106–1115Google Scholar
  15. Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642CrossRefMathSciNetGoogle Scholar
  16. Jacobs DW (2001) Perceptual organization as generic object recognition. In: Shipley TF, Kellman PJ (eds) From fragments to objects—segmentation and grouping in vision. Elsevier, Amsterdam, pp 295–329 (Chap IV. Models of segmentation and grouping)Google Scholar
  17. Katz D, Brock O (2011) Interactive segmentation of articulated objects in 3D. In: Workshop on mobile manipulation at ICRAGoogle Scholar
  18. Kenney J, Buckley T, Brock O (2009) Interactive segmentation for manipulation in unstructured environments. In: Proceedings of the 2009 IEEE international conference on robotics and automation, ICRA’09Google Scholar
  19. Klappstein J, Vaudrey T, Rabe C, Wedel A, Klette R (2008) Moving object segmentation using optical flow and depth information. In: Proceedings of the 3rd Pacific rim symposium on advances in image and video technology (PSIVT’09). Springer, Berlin, pp 611–623Google Scholar
  20. Klingbeil E, Rao D, Carpenter B, Ganapathi V, Ng AY, Khatib O (2011) Grasping with application to an autonomous checkout robot. In: IEEE international conference on robotics and automation (ICRA). IEEE, Shanghai, China, pp 2837–2844Google Scholar
  21. Marton ZC, Pangercic D, Blodow N, Beetz M (2011) Combined 2D–3D categorization and classification for multimodal perception systems. Int J Robot Res 30(11):1378–1402Google Scholar
  22. Marton ZC, Balint-Benczedi F, Mozos OM, Blodow N, Kanezaki A, Goron LC, Pangercic D, Beetz M (2014) Part-based geometric categorization and object reconstruction in cluttered table-top scenes. J Intell Robot Syst 76(1):35–56Google Scholar
  23. Mishra AK, Aloimonos Y (2009) Active segmentation with fixation. In: Proceedings of the international conference on computer vision (ICCV). IEEEGoogle Scholar
  24. Mozos OM, Marton ZC, Beetz M (2011) Furniture models learned from the WWW—using web catalogs to locate and categorize unknown furniture pieces in 3D laser scans. Robot Autom Mag 18(2):22–32Google Scholar
  25. Rusu RB, Blodow N, Marton ZC, Soos A, Beetz M (2007) Towards 3D object maps for autonomous household robots. In: Proceedings of the 20th IEEE international conference on intelligent robots and systems (IROS). San Diego, CA, USAGoogle Scholar
  26. Rusu RB, Marton ZC, Blodow N, Dolha ME, Beetz M (2008) Functional object mapping of kitchen environments. In: Proceedings of the 21st IEEE/RSJ international conference on intelligent robots and systems (IROS). Nice, France, 22–26 Sept 2008Google Scholar
  27. Schiebener D, Morimoto J, Asfour T, Ude A (2013) Integrating visual perception and manipulation for autonomous learning of object representations. Adapt Behav 21(5):328–345Google Scholar
  28. Shi J, Tomasi C (1994) Good features to track. In: 1994 IEEE conference on computer vision and pattern recognition (CVPR’94), pp 593–600Google Scholar
  29. Suzuki S, Abe K (1985) Topological structural analysis of digitized binary images by border following. Comput Vis Graph Image Process 30(1):32–46CrossRefzbMATHGoogle Scholar
  30. van Hoof H, Kroemer O, Peters J (2014) Probabilistic segmentation and targeted exploration of objects in cluttered environments. IEEE Trans Robot 30(5):1198–1209. doi: 10.1109/TRO.2014.2334912
  31. Zhan Q, Liang Y, Xiao Y (2009) Color-based segmentation of point clouds. In: Laserscanning 2009 ISPRS WorkshopGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Karol Hausman
    • 1
    Email author
  • Dejan Pangercic
    • 2
  • Zoltán-Csaba Márton
    • 3
  • Ferenc Bálint-Benczédi
    • 4
  • Christian Bersch
    • 5
  • Megha Gupta
    • 1
  • Gaurav Sukhatme
    • 1
  • Michael Beetz
    • 4
  1. 1.University of Southern CaliforniaLos AngelesUSA
  2. 2.Robert Bosch LLCPalo AltoUSA
  3. 3.German Aerospace CenterOberpfaffenhofen-WesslingGermany
  4. 4.University of BremenBremenGermany
  5. 5.Google IncMountain ViewUSA

Personalised recommendations