Abstract
As robots become more ubiquitous and capable of performing complex tasks, the importance of enabling untrained users to interact with them has increased. In response, unconstrained natural-language interaction with robots has emerged as a significant research area. We discuss the problem of parsing natural language commands to actions and control structures that can be readily implemented in a robot execution system. Our approach learns a parser based on example pairs of English commands and corresponding control language expressions. We evaluate this approach in the context of following route instructions through an indoor environment, and demonstrate that our system can learn to translate English commands into sequences of desired actions, while correctly capturing the semantic intent of statements involving complex control structures. The procedural nature of our formal representation allows a robot to interpret route instructions online while moving through a previously unknown environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Artzi, Y., Zettlemoyer, L.S.: Bootstrapping semantic parsers from conversations. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2011)
Beetz, M., Arbuckle, T., Belker, T., Bennewitz, M., Burgard, W., Cremers, A.B., Fox, D., Grosskreutz, H., Hähnel, D., Schulz, D.: Integrated plan-based control of autonomous service robots in human environments. IEEE Intelligent Systems 16(5) (2001)
Boutilier, C., Reiter, R., Soutchanski, M., Thrun, S.: Decision-theoretic, high-level agent programming in the situation calculus. In: Proc. of the National Conference on Artificial Intelligence, AAAI (2000)
Branavan, S.R.K., Zettlemoyer, L., Barzilay, R.: Reading between the lines: Learning to map high-level instructions to commands. In: ACL, pp. 1268–1277 (2010)
Burgard, W., Cremers, A.B., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S.: Experiences with an interactive museum tour-guide robot. Artificial Intelligence 114(1-2), 3–55 (1999)
Chen, D.L.: Fast online lexicon learning for grounded language acquisition. In: Proc. of the Annual Meetings of the Association for Computational Linguistics, ACL (2012)
Chen, D.L., Mooney, R.: Learning to interpret natural language navigation instructions from observations. In: Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI 2011), pp. 859–865 (2011)
Dzifcak, J., Scheutz, M., Baral, C., Schermerhorn, P.: What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In: Proc. of the 2009 IEEE Int’l Conf. on Robotics and Automation, ICRA 2009 (2009)
Ferrein, A., Lakemeyer, G.: Logic-based robot control in highly dynamic domains. Robotics and Autonomous Systems 56(11) (2008)
Friedman, S., Pasula, H., Fox, D.: Voronoi random fields: Extracting topological structure of indoor environments via place labeling. In: IJCAI, pp. 2109–2114 (2007)
Hähnel, D., Burgard, W., Lakemeyer, G.: GOLEX - bridging the gap between logic (GOLOG) and a real robot. In: Proc. of the German Conference on Artificial Intelligence (KI), Germany (1998)
Ingrand, F., Chatila, R., Alami, R., Robert, F.: PRS: A high level supervision and control language for autonomous mobile robots. In: Proc. of the IEEE International Conference on Robotics & Automation, ICRA (1996)
Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical turk. In: Proc. of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, CHI 2008. ACM (2008)
Kollar, T., Tellex, S., Roy, D., Roy, N.: Toward understanding natural language directions. In: HRI 2010: Proc. of the 5th Int’l Conf. on Human-Robot Interaction. ACM Press (2010)
Kress-Gazit, H., Fainekos, G., Pappas, G.: Translating structured english to robot controllers. Advanced Robotics 22(12) (2008)
Kress-Gazit, H., Wongpiromsarn, T., Topcu, U.: Correct, reactive robot control from abstraction and temporal logic specifications. IEEE Robotics and Automation Magazine, special issue on Formal Methods for Robotics and Automation 18(3), 65–74 (2011)
Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., Steedman, M.: Inducing probabilistic CCG grammars from logical form with higher-order unification. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2010)
Kwiatkowski, T., Zettlemoyer, L.S., Goldwater, S., Steedman, M.: Lexical generalization in CCG grammar induction for semantic parsing. In: Proc. of the Conf. on Empirical Methods in Natural Language Processing (2011)
Lai, K., Bo, L., Ren, X., Fox, D.: A scalable tree-based approach for joint object and pose recognition. In: The AAAI Conference on Artificial Intelligence, AAAI (2011)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
Lison, P., Kruijff, G.-J.M.: An integrated approach to robust processing of situated spoken dialogue. In: Proc. of SRSL 2009, the 2nd Workshop on Semantic Representation of Spoken Language, Athens, Greece, pp. 58–65. Association for Computational Linguistics (March 2009)
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., Fox, D.: A Joint Model of Language and Perception for Grounded Attribute Learning. In: Proc. of the 2012 International Conference on Machine Learning, Edinburgh, Scotland (June 2012)
Matuszek, C., Fox, D., Koscher, K.: Following directions using statistical machine translation. In: ACM/IEEE International Conference on Human-Robot Interaction, HRI (2010)
Mooney, R.: Learning to connect language and perception. In: Fox, D., Gomes, C.P. (eds.) Proc. of the Twenty-Third AAAI Conf. on Artificial Intelligence, AAAI 2008, Chicago, Illinois, pp. 1598–1601. AAAI Press (2008)
Shimizu, N., Haas, A.: Learning to Follow Navigational Route Instructions. In: Int’l Joint Conf. on Artificial Intelligence, IJCAI (2009)
Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In: EMNLP, pp. 254–263 (2008)
Steedman, M.: The Syntactic Process. MIT Press (2000)
Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee, A., Teller, S., Roy, N.: Approaching the symbol grounding problem with probabilistic graphical models. AI Magazine 32(4) (2012)
Tsuruoka, Y., Tsujii, J.: Iterative CKY parsing for probabilistic context-free grammars. In: Su, K.-Y., Tsujii, J., Lee, J.-H., Kwong, O.Y. (eds.) IJCNLP 2004. LNCS (LNAI), vol. 3248, pp. 52–60. Springer, Heidelberg (2005)
Wei, Y., Brunskill, E., Kollar, T., Roy, N.: Where to go: Interpreting natural directions using global inference. In: Proc. of the IEEE International Conference on Robotics & Automation, ICRA (2009)
Zettlemoyer, L.S., Collins, M.: Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In: Proc. of the Conf. on Uncertainty in Artificial Intelligence (2005)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Matuszek, C., Herbst, E., Zettlemoyer, L., Fox, D. (2013). Learning to Parse Natural Language Commands to a Robot Control System. In: Desai, J., Dudek, G., Khatib, O., Kumar, V. (eds) Experimental Robotics. Springer Tracts in Advanced Robotics, vol 88. Springer, Heidelberg. https://doi.org/10.1007/978-3-319-00065-7_28
Download citation
DOI: https://doi.org/10.1007/978-3-319-00065-7_28
Publisher Name: Springer, Heidelberg
Print ISBN: 978-3-319-00064-0
Online ISBN: 978-3-319-00065-7
eBook Packages: EngineeringEngineering (R0)