Abstract
As the tasks of autonomous manipulation robots get more complex, the tasking of the robots using natural-language instructions becomes more important. Executing such instructions in the way they are meant often requires robots to infer missing, and disambiguate given information using lots of common and commonsense knowledge. In this work, we report on Probabilistic Action Cores (Prac) (Nyga and Beetz, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012) – a framework for learning of and reasoning about action-specific probabilistic knowledge bases that can be learned from hand-labeled instructions to address this problem. In Prac, knowledge about actions and objects is compactly represented by first-order probabilistic models, which are used to learn a joint probability distribution over the ways in which instructions for a given action verb are formulated. These joint probability distributions are then used to compute the plan instantiation that has the highest probability of producing the intended action given the natural language instruction. Formulating plan interpretation as a conditional probability is a promising approach because we can at the same time infer the plan that is most appropriate for performing the instruction, the refinement of the parameters of the plan on the basis of the information given in the instruction, and automatically fill in missing parameters by inferring their most probable value from the distribution. Prac has been implemented as a web-based online service on the cloud-robotics platform openEASE [7].
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
All concept names refer to concept names provided by the NLTK toolbox (http://www.nltk.org).
- 2.
We are using a slightly modified notation, which technically does not precisely fit the previous formulations. We think this simplified notation better supports the understanding of reasoning considered in this paper.
- 3.
References
Anderson, J.: Constraint-directed improvisation for everyday activities. Ph.D. thesis (1995)
Artzi, Y., Zettlemoyer, L.: Weakly supervised learning of semantic parsers for mapping instructions to actions. Trans. Assoc. Comput. Linguist. 1(1), 49–62 (2013)
Bailey, D.: When push comes to shove: A computational model of the role of motor control in the acquisition of action verbs. Ph.D. thesis, University of California (1997)
Baker, C.F., Fillmore, C.J., Lowe, J.B.: The berkeley framenet project. Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics -. ACL ’98, vol. 1, pp. 86–90. Association for Computational Linguistics, Stroudsburg, PA, USA (1998)
Barker, R.: Ecological Psychology: Concepts and Methods for Studying the Environment of Human Behavior. Stanford University Press, Stanford (1968)
Beetz, M., Jain, D., Mösenlechner, L., Tenorth, M., Kunze, L., Blodow, N., Pangercic, D.: Cognition-enabled autonomous robot control for the realization of home chore task intelligence. Proc. IEEE 100(8), 2454–2471 (2012)
Beetz, M., Tenorth, M., Winkler, J.: Open-EASE – a knowledge processing service for robots and robotics/ai researchers. In: IEEE International Conference on Robotics and Automation (ICRA), Seattle, Washington, USA (2015). (Finalist for the Best Cognitive Robotics Paper Award)
De Marneffe, M., MacCartney, B., Manning, C.: Generating typed dependency parses from phrase structure parses. Proc. LREC 6, 449–454 (2006)
de Marneffe, M.-C., Manning, C.D.: The stanford typed dependencies representation. In: Coling 2008: Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation, CrossParser ’08, pp. 1–8. Stroudsburg, PA, USA (2008) (Association for Computational Linguistics)
Dzifcak, J., Scheutz, M., Baral, C., Schermerhorn, P.: What to do and how to do it: translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In: IEEE International Conference on Robotics and Automation, 2009. ICRA’09, pp. 4163–4168. IEEE (2009)
Feldman, J., Narayanan, S.: Embodied meaning in a neural theory of language. Brain Lang. 89(2), 385–392 (2004) (Language and MotorIntegration)
Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)
Firby, J.: Adaptive execution in complex dynamic Worlds. Technical report 672, Yale University, Department of Computer Science (1989)
Georgeff, M., Ingrand, F.: Decision making in an embedded reasing system. In: Proceedings of the Eleventh International Joint Conference on Artificial Intelligence, pp. 972–978. Detroit, MI (1989)
Kim, J., Mooney, R.J.: Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 433–444. Association for Computational Linguistics (2012)
Matuszek, C., Fox, D., Koscher, K.: Following directions using statistical machine translation. In: Proceeding of the 5th ACM/IEEE International Conference on Human-robot Interaction, pp. 251–258. ACM (2010)
Minsky, M.: A framework for representing knowledge. Technical Report Memo 306, MIT-AI Laboratory (1974)
Misra, D.K., Sung, J., Lee, K., Saxena, A.: Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. In: Proceedings of Robotics Science and Systems, Berkeley, USA (2014)
Mösenlechner, L., Beetz, M.: Parameterizing actions to have the appropriate effects. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), San Francisco, CA, USA (2011). Accessed 25–30 Sept 2011
Neo, E., Sakaguchi, T., Yokoi, K.: A natural language instruction system for humanoid robots integrating situated speech recognition, visual recognition and on-line whole-body motion generation. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2008. AIM 2008, pp. 1176–1182. IEEE (2008)
Nyga, D., Beetz, M.: Everything robots always wanted to know about housework (but were afraid to ask). In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vilamoura, Portugal (2012). Accessed 7–12 Oct 2012
Nyga, D., Beetz, M.: Reasoning about unmodelled concepts – incorporating class taxonomies in probabilistic relational models (2015). http://arxiv.org/abs/1504.05411
Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62(1–2), 107–136 (2006)
Ryu, J., Jung, Y., Kim, K., Myaeng, S.: Automatic extraction of human activity knowledge from method-describing web articles. In: Proceedings of the 1st Workshop on Automated Knowledge Base Construction, p. 16 (2010)
Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., Roy, N.: Understanding natural language commands for robotic navigation and mobile manipulation. In: Proceedings of the National Conference on Artificial Intelligence (AAAI) (2011)
Tenenbaum, J.B., Kemp, C., Griffiths, T.L., Goodman, N.D.: How to grow a mind: statistics, structure, and abstraction. Science 331(6022), 1279–1285 (2011)
Tenorth,M., Nyga, D., Beetz, M.: Understanding and executing instructions for everyday manipulation tasks from the world wide web. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1486–1491. Anchorage, AK, USA (2010). Accessed 3–8 May 2010
Acknowledgements
This work is supported by the EU FP7 Projects RoboHow (grant number 288533) and ACAT (grant number 600578).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Nyga, D., Beetz, M. (2018). Cloud-Based Probabilistic Knowledge Services for Instruction Interpretation. In: Bicchi, A., Burgard, W. (eds) Robotics Research. Springer Proceedings in Advanced Robotics, vol 3. Springer, Cham. https://doi.org/10.1007/978-3-319-60916-4_37
Download citation
DOI: https://doi.org/10.1007/978-3-319-60916-4_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60915-7
Online ISBN: 978-3-319-60916-4
eBook Packages: EngineeringEngineering (R0)