RoboSherlock: Unstructured Information Processing Framework for Robotic Perception

  • Michael BeetzEmail author
  • Ferenc Bálint-Benczédi
  • Nico Blodow
  • Christian Kerl
  • Zoltán-Csaba Márton
  • Daniel Nyga
  • Florian Seidel
  • Thiemo Wiedemeyer
  • Jan-Hendrik Worch
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 42)


A pressing question when designing intelligent autonomous systems is how to integrate the various subsystems concerned with complementary tasks. Robotic vision must provide task relevant information about the environment and the objects in it to various planning related modules. In most implementations of the traditional Perception–Cognition–Action paradigm these tasks are treated as quasi-independent modules that function as black boxes for each other. Often these subsystems are running in completely different frameworks, with a thin communication interface or middle-ware between them. While each subproblem poses specific requirements that can make fusing them more challenging, perception can benefit tremendously from a tight collaboration with cognition. In the following, a common framework for cognitive perception, based on the principle of unstructured information management (UIM) will be presented, called RoboSherlock. UIM has proven itself to be a powerful paradigm for scaling intelligent information and question answering systems towards real-world complexity. Complexity in UIM is handled by identifying (or hypothesizing) pieces of structured information by applying ensembles of experts for annotating information pieces, and by testing and integrating these isolated annotations into a comprehensive interpretation. RoboSherlock is an open source software framework for unstructured information processing in robot perception that demonstrates the potential of the paradigm for real-world scene perception.


Point Cloud Perception System Belief State Perception Task Entity Resolution 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Aydemir A, Sjöö K, Folkesson J, Pronobis A, Jensfelt P (2011) Search in the real world: active visual object search based on spatial relations. In: IEEE international conference on robotics and automation (ICRA), 2011. IEEE, pp 2818–2824Google Scholar
  2. Beetz M, Mösenlechner L, Tenorth M (2010) Cram—a cognitive robot abstract machine for everyday manipulation in human environments. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems. Taipei, Taiwan, pp 1012–1017Google Scholar
  3. Beetz M, Balint-Benczedi F, Blodow N, Nyga D, Wiedemeyer T, Marton ZC (2015) RoboSherlock: unstructured information processing for robot perception. In: IEEE international conference on robotics and automation (ICRA), Seattle, Washington, USA, nominations for Best Conference Paper Award and Best Service Robotics Paper AwardGoogle Scholar
  4. Blodow N (2014) Managing belief states for service robots—dynamic scene perception and spatio-temporal memory. Ph.D. thesis, Technische Universität München.
  5. Blodow N, Jain D, Marton ZC, Beetz M (2010) Perception and probabilistic anchoring for dynamic world state logging. In: 10th IEEE-RAS international conference on humanoid robots. Nashville, TN, USA, pp 160–166Google Scholar
  6. Blodow N, Marton ZC, Pangercic D, Rühr T, Tenorth M, Beetz M (2011) Inferring generalized pick-and-place tasks from pointing gestures. In: IEEE international conference on robotics and automation (ICRA), workshop on semantic perception, mapping and explorationGoogle Scholar
  7. Bohren J, Rusu RB, Jones EG, Marder-Eppstein E, Pantofaru C, Wise M, Mosenlechner L, Meeussen W, Holzer S (2011) Towards autonomous robotic butlers: lessons learned with the PR2. In: ICRA, Shanghai, ChinaGoogle Scholar
  8. Bradski G (2000) The OpenCV library. Dr Dobb’s journal of software toolsGoogle Scholar
  9. Collet Romea A, Martinez Torres M, Srinivasa S (2011) The MOPED framework: object recognition and pose estimation for manipulation. Int J Robot Res 30(10):1284–1306CrossRefGoogle Scholar
  10. Duncan K, Sarkar S, Alqasemi R, Dubey R (2013) Multi-scale superquadric fitting for efficient shape and pose recovery of unknown objects. In: IEEE international conference on robotics and automation (ICRA)Google Scholar
  11. Ferrucci D, Lally A (2004) UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng 10(3–4):327–348CrossRefGoogle Scholar
  12. Ferrucci D, Nyberg E, Allan J, Barker K, Brown E, Chu-Carroll J, Ciccolo A, Duboue P, Fan J, Gondek D, Hovy E, Katz B, Lally A, McCord M, Morarescu P, Murdock B, Porter B, Prager J, Strzalkowski T, Welty C, Zadrozny W (2009) Towards the open advancement of question answering systems. Technical report RC24789, IBM Research ReportGoogle Scholar
  13. Ferrucci D, Brown E, Chu-Carroll J, Fan J, Gondek D, Kalyanpur AA, Lally A, Murdock JW, Nyberg E, Prager J, Schlaefer N, Welty C (2010) Building Watson: an overview of the DeepQA project. AI Mag 31(3):59–79.
  14. Gould S, Russakovsky O, Goodfellow I, Baumstarck P, Ng AY, Koller D (2010) The stair vision library (v2.4).
  15. Hinterstoisser S, Lepetit V, Ilic S, Holzer S, Bradski G, Konolige K, , Navab N (2012) Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In: Asian conference on computer visionGoogle Scholar
  16. Kragic D, Vincze M (2009) Vision for robotics. Found Trends Robot 1(1):1–78. doi: 10.1561/2300000001
  17. Lai K, Bo L, Ren X, Fox D (2011) Sparse distance learning for object recognition combining RGB and depth information. In: Proceedings of international conference on robotics and automation (ICRA)Google Scholar
  18. Marton ZC, Pangercic D, Blodow N, Beetz M (2011) Combined 2D–3D categorization and classification for multimodal perception systems. Int J Robot Res 30(11):1378–1402CrossRefGoogle Scholar
  19. Marton ZC, Seidel F, Balint-Benczedi F, Beetz M (2012a) Ensembles of strong learners for multi-cue classification. Pattern recognition letters (PRL), Special issue on scene understandings and behaviours analysisGoogle Scholar
  20. Marton ZC, Seidel F, Beetz M (2012b) Towards modular spatio-temporal perception for task-adapting robots. In: Postgraduate conference on robotics and development of cognition (RobotDoC-PhD), a satellite event of the 22nd international conference on artificial neural networks (ICANN). Lausanne, SwitzerlandGoogle Scholar
  21. Mörwald T, Prankl J, Richtsfeld A, Zillich M, Vincze M (2010) Blort- the blocks world robotic vision toolbox. In: “Best practice algorithms in 3D perception and modeling for mobile manipulation workshop”—CD (in conjunction with the IEEE ICRA 2010)Google Scholar
  22. Muja M, Rusu RB, Bradski G, Lowe D (2011) REIN—A fast, robust, scalable REcognition INfrastructure. In: ICRA. Shanghai, ChinaGoogle Scholar
  23. Nyga D, Balint-Benczedi F, Beetz M (2014) PR2 Looking at things: ensemble learning for unstructured information processing with Markov logic networks. In: IEEE international conference on robotics and automation (ICRA). Hong Kong, ChinaGoogle Scholar
  24. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42:145–175CrossRefzbMATHGoogle Scholar
  25. Pangercic D, Tenorth M, Jain D, Beetz M (2010) Combining perception and knowledge processing for everyday manipulation. In: IEEE/RSJ international conference on intelligent robots and systems (IROS). Taipei, Taiwan, pp 1065–1071Google Scholar
  26. Pangercic D, Mathe K, Marton ZC, Goron LC, Opris MS, Schuster M, Tenorth M, Jain D, Ruehr T, Beetz M (2011) A robot that shops for and stores groceries. AAAI video competition (AIVC 2011).
  27. Pangercic D, Tenorth M, Pitzer B, Beetz M (2012) Semantic object maps for robotic housework—representation, acquisition and use. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS). Vilamoura, PortugalGoogle Scholar
  28. Richardson M, Domingos P (2006) Markov logic networks. Mach Learn 62(1–2):107–136. doi: 10.1007/s10994-006-5833-1 CrossRefGoogle Scholar
  29. Rühr T, Sturm J, Pangercic D, Cremers D, Beetz M (2012) A generalized framework for opening doors and drawers in kitchen environments. In: IEEE International conference on robotics and automation (ICRA). St. Paul, MN, USAGoogle Scholar
  30. Rusu RB, Cousins S (2011) 3D is here: point cloud library (PCL). In: IEEE international conference on robotics and automation (ICRA). Shanghai, China, pp 1–4Google Scholar
  31. Rusu RB, Meeussen W, Chitta S, Beetz M (2009) Laser-based perception for door and handle identification. In: Proceedings of the international conference on advanced robotics (ICAR), Munich, Best Paper AwardGoogle Scholar
  32. Rusu RB, Bradski G, Thibaux R, Hsu J (2010) Fast 3d recognition and pose using the viewpoint feature histogram. In: Proceedings of the 23rd IEEE/RSJ IROS. Taipei, TaiwanGoogle Scholar
  33. Tenorth M, Beetz M (2013) Knowrob—a knowledge processing infrastructure for cognition-enabled robots. Int J Robot Res (IJRR) 32(5):566–590CrossRefGoogle Scholar
  34. Tenorth M, Klank U, Pangercic D, Beetz M (2011) Web-enabled robots—robots that use the web as an information resource. Robot Autom Mag 18(2):58–68CrossRefGoogle Scholar
  35. Tenorth M, Profanter S, Balint-Benczedi F, Beetz M (2013) Decomposing cad models of objects of daily use and reasoning about their functional parts. In: IEEE/RSJ International conference on intelligent robots and systems (IROS). Tokyo Big Sight, JapanGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Michael Beetz
    • 1
    Email author
  • Ferenc Bálint-Benczédi
    • 1
  • Nico Blodow
    • 2
  • Christian Kerl
    • 3
  • Zoltán-Csaba Márton
    • 4
  • Daniel Nyga
    • 1
  • Florian Seidel
    • 2
  • Thiemo Wiedemeyer
    • 1
  • Jan-Hendrik Worch
    • 1
  1. 1.Institute for Artificial IntelligenceUniversität BremenBremenGermany
  2. 2.Intelligent Autonomous Systems GroupTechnische Universität MünchenMunichGermany
  3. 3.Computer Vision GroupTechnische Universität MünchenMunichGermany
  4. 4.German Aerospace CenterOberpfaffenhofen-WesslingGermany

Personalised recommendations