Active Recognition and Manipulation for Mobile Robot Bin Picking

  • Dirk HolzEmail author
  • Matthias Nieuwenhuisen
  • David Droeschel
  • Jörg Stückler
  • Alexander Berner
  • Jun Li
  • Reinhard Klein
  • Sven Behnke
Conference paper
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 94)


Grasping individual objects from an unordered pile in a box has been investigated in stationary scenarios so far. In this work, we present a complete system including active object perception and grasp planning for bin picking with a mobile robot. At the core of our approach is an efficient representation of objects as compounds of simple shape and contour primitives. This representation is used for both robust object perception and efficient grasp planning. For being able to manipulate previously unknown objects, we learn object models from single scans in an offline phase. During operation, objects are detected in the scene using a particularly robust probabilistic graph matching. To cope with severe occlusions we employ active perception considering not only previously unseen volume but also outcomes of primitive and object detection. The combination of shape and contour primitives makes our object perception approach particularly robust even in the presence of noise, occlusions, and missing information. For grasp planning, we efficiently pre-compute possible grasps directly on the learned object models. During operation, grasps and arm motions are planned in an efficient local multiresolution height map. All components are integrated and evaluated in a bin picking and part delivery task.


Mobile bin picking contour and shape primitives active object perception grasp planning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ikeuchi, K., Horn, B.K.P., Nagata, S., Callahan, T., Feirigold, O.: Picking up an object from a pile of objects. In: Robotics Research: The First International Symposium, pp. 139–162. MIT Press (1984)Google Scholar
  2. 2.
    Rahardja, K., Kosaka, A.: Vision-based bin-picking: Recognition and localization of multiple complex objects using simple visual cues. In: Proc. IEEE Int. Conf. on Intelligent Robots and Systems (1996)Google Scholar
  3. 3.
    Liu, M.-Y., Tuzel, O., Veeraraghavan, A., Taguchi, Y., Marks, T.K., Chellappa, R.: Fast object localization and pose estimation in heavy clutter for robotic bin picking. Int. J. of Robotics Research 31(8), 951–973 (2012)CrossRefGoogle Scholar
  4. 4.
    Stückler, J., Holz, D., Behnke, S.: RoboCup@Home: Demonstrating everyday manipulation skills in RoboCup@Home. IEEE Robotics & Automation Magazine 19(2), 34–42 (2012)CrossRefGoogle Scholar
  5. 5.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition. Int. J. Comput. Vision 61(1), 55–79 (2005)CrossRefGoogle Scholar
  6. 6.
    Nieuwenhuisen, M., Stückler, J., Berner, A., Klein, R., Behnke, S.: Shape-primitive based object recognition and grasping. In: Proc. 7th German Conference on Robotics (2012)Google Scholar
  7. 7.
    Nieuwenhuisen, M., Droeschel, D., Holz, D., Stückler, J., Berner, A., Li, J., Klein, R., Behnke, S.: Mobile bin picking with an anthropomorphic service robot. In: Proc. IEEE Int. Conf. Robotics and Automation, pp. 2319–2326 (2013)Google Scholar
  8. 8.
    Papazov, C., Haddadin, S., Parusel, S., Krieger, K., Burschka, D.: Rigid 3D geometry matching for grasping of known objects in cluttered scenes. Int. J. of Robotics Research 31(4), 538–553 (2012)CrossRefGoogle Scholar
  9. 9.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Bley, F., Schmirgel, V., Kraiss, K.-F.: Mobile manipulation based on generic object knowledge. In: Proc. IEEE Int. Symp. on Robot and Human Interactive Communication (2006)Google Scholar
  11. 11.
    Choi, C., Taguchi, Y., Tuzel, O., Liu, M.-Y., Ramalingam, S.: Voting-based pose estimation for robotic assembly using a 3D sensor. In: Proc. IEEE Int. Conf. Robotics and Automation (2012)Google Scholar
  12. 12.
    Wahl, E., Hillenbrand, U., Hirzinger, G.: Surflet-pair-relation histograms: a statistical 3D-shape representation for rapid classification. In: Proc. Int. Conf. on 3-D Digital Imaging and Modeling (2003)Google Scholar
  13. 13.
    Drost, B., Ulrich, M., Navab, N., Ilic, S.: Model globally, match locally: Efficient and robust 3D object recognition. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (2010)Google Scholar
  14. 14.
    Kim, E., Medioni, G.: 3D object recognition in range images using visibility context. In: Proc. IEEE Int. Conf. on Intelligent Robots and Systems (2011)Google Scholar
  15. 15.
    Schnabel, R., Wessel, R., Wahl, R., Klein, R.: Shape recognition in 3D point-clouds. In: Proc. Int. Conf. on Computer Graphics, Visualization and Computer Vision (2008)Google Scholar
  16. 16.
    Li, Y., Wu, X., Chrysathou, Y., Sharf, A., Cohen-Or, D., Mitra, N.J.: Globfit: Consistently fitting primitives by discovering global relations. ACM Trans. on Graphics 30, 52:1–52:12 (2011)Google Scholar
  17. 17.
    Cohen, B.J., Subramanian, G., Chitta, S., Likhachev, M.: Planning for manipulation with adaptive motion primitives. In: Proc. IEEE Int. Conf. Robotics and Automation (2011)Google Scholar
  18. 18.
    Chitta, S., Jones, E.G., Ciocarlie, M., Hsiao, K.: Perception, planning, and execution for mobile manipulation in unstructured environments. IEEE Robotics & Automation Magazine 19(2), 58–71 (2012)CrossRefGoogle Scholar
  19. 19.
    Klingbeil, E., Rao, D., Carpenter, B., Ganapathi, V., Ng, A.Y., Khatib, O.: Grasping with application to an autonomous checkout robot. In: Proc. IEEE Int. Conf. Robotics and Automation (2011)Google Scholar
  20. 20.
    Chang, L., Smith, J.R., Fox, D.: Interactive singulation of objects from a pile. In: Proc. IEEE Int. Conf. Robotics and Automation (2012)Google Scholar
  21. 21.
    Gupta, M., Sukhatme, G.S.: Using manipulation primitives for brick sorting in clutter. In: Proc. IEEE Int. Conf. Robotics and Automation (2012)Google Scholar
  22. 22.
    Srinivasa, S.S., Ferguson, D., Helfrich, C.J., Berenson, D., Collet, A., Diankov, R., Gallagher, G., Hollinger, G., Kuffner, J., Van de Weghe, M.: HERB: a home exploring robotic butler. Autonomous Robots 28(1), 5–20 (2010)CrossRefGoogle Scholar
  23. 23.
    Bäuml, B., Schmidt, F., Wimböck, T., Birbach, O., Dietrich, A., Fuchs, M., Friedl, W., Frese, U., Borst, C., Grebenstein, M., Eiberger, O., Hirzinger, G.: Catching flying balls and preparing coffee: Humanoid Rollin’Justin performs dynamic and sensitive tasks. In: Proc. IEEE Int. Conf. Robotics and Automation (2011)Google Scholar
  24. 24.
    Vahrenkamp, N., Asfour, T., Dillmann, R.: Simultaneous grasp and motion planning: Humanoid robot ARMAR-III. IEEE Robotics & Automation Magazine 19(2), 43–57 (2012)CrossRefGoogle Scholar
  25. 25.
    Jain, A., Kemp, C.C.: EL-E: an assistive mobile manipulator that autonomously fetches objects from flat surfaces. Autonomous Robots 28(1), 45–64 (2010)CrossRefGoogle Scholar
  26. 26.
    bEETZ, M., Klank, U., Kresse, I., Maldonado, A., Mösenlechner, L., Pangercic, D., Rühr, T., Tenorth, M.: Robotic roommates making pancakes. In: Proc. Int. Conf. on Humanoid Robots (2011)Google Scholar
  27. 27.
    Mitra, N.J., Gelfand, N., Pottmann, H., Guibas, L.: Registration of point cloud data from a geometric optimization perspective. In: Symp. Geometry Processing (2004)Google Scholar
  28. 28.
    Bendels, G.H., Schnabel, R., Klein, R.: Detecting holes in point set surfaces. Journal of WSCG 14(1-3) (February 2006)Google Scholar
  29. 29.
    Schnabel, R., Wahl, R., Klein, R.: Efficient RANSAC for point-cloud shape detection. Computer Graphics Forum 26(2), 214–226 (2007)CrossRefGoogle Scholar
  30. 30.
    Berner, A., Bokeloh, M., Wand, M., Schilling, A., Seidel, H.-P.: A graph-based approach to symmetry detection. In: Proc. IEEE/EG Int. Symp. on Volume and Point-Based Graphics (2008)Google Scholar
  31. 31.
    Papazov, C., Burschka, D.: An efficient RANSAC for 3D object recognition in noisy and occluded scenes. In: Proc. Asian Conf. on Computer Vision (2011)Google Scholar
  32. 32.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, SGP 2006, pp. 61–70 (2006)Google Scholar
  33. 33.
    Holz, D., Behnke, S.: Sancta simplicitas – on the efficiency and achievable results of SLAM using ICP-based incremental registration. In: Proc. IEEE Int. Conf. Robotics and Automation, pp. 1380–1387 (2010)Google Scholar
  34. 34.
    Hornung, A., Wurm, K.M., Bennewitz, M., Stachniss, C., Burgard, W.: OctoMap: An efficient probabilistic 3D mapping framework based on octrees. Autonomous Robots (2013)Google Scholar
  35. 35.
    Şucan, I.A., Kavraki, L.E.: Kinodynamic motion planning by interior-exterior cell exploration. In: Chirikjian, G.S., Choset, H., Morales, M., Murphey, T. (eds.) Algorithmic Foundation of Robotics VIII. STAR, vol. 57, pp. 449–464. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  36. 36.
    Behnke, S.: Local multiresolution path planning. In: Polani, D., Browning, B., Bonarini, A., Yoshida, K. (eds.) RoboCup 2003. LNCS (LNAI), vol. 3020, pp. 332–343. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  37. 37.
    Fox, D.: Adapting the sample size in particle filters through KLD-sampling. I. J. Robotic Res. 22(12), 985–1004 (2003)CrossRefGoogle Scholar
  38. 38.
    Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. on Systems Science and Cybernetics 4(2), 100–107 (1968)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Dirk Holz
    • 1
    Email author
  • Matthias Nieuwenhuisen
    • 1
  • David Droeschel
    • 1
  • Jörg Stückler
    • 1
  • Alexander Berner
    • 1
  • Jun Li
    • 1
  • Reinhard Klein
    • 1
  • Sven Behnke
    • 1
  1. 1.Computer Science InstituteUniversity of BonnBonnGermany

Personalised recommendations