Capturing Frame-Like Object Descriptors in Human Augmented Mapping

  • Mohamadreza FaridghasemniaEmail author
  • Andrea Vanzo
  • Daniele Nardi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11946)


The model of an environment plays a crucial role in autonomous mobile robots, by providing them with the necessary task-relevant information. As robots become more intelligent, they need a richer and more expressive environment model. This model is a map that contains a structured description of the environment that can be used as the robot’s knowledge for several tasks, such as planning and reasoning. In this work, we propose a framework that allows to capture important environment descriptors, such as functionality and ownership of the robot’s surrounding objects, through verbal interaction. Specifically, we propose a corpus of verbal descriptions annotated with frame-like structures. We use the proposed dataset to train two multi-task neural architectures. We compare the two architectures through an experimental evaluation, discussing the design choices. Finally, we describe the creation of a simple interactive interface with our system, implemented through the trained model. The novelties of this work are: (i) the definition of a new problem, i.e., addressing different object descriptors, that plays a crucial role for the robot’s tasks accomplishment; (ii) a specialized corpus to support the creation of rich Semantic Maps; (iii) the design of different neural architectures, and their experimental evaluation over the proposed dataset; (iv) a simple interface for the actual usage of the proposed resources.


Natural Language understanding Semantic mapping Human robot interaction Neural networks Semantic mapping corpus Corpus annotator 


  1. 1.
    Bastianelli, E., Castellucci, G., Croce, D., Iocchi, L., Basili, R., Nardi, D.: Huric: a human robot interaction corpus. In: LREC, pp. 4519–4526 (2014)Google Scholar
  2. 2.
    Dinarelli, M., Quarteroni, S., Tonelli, S., Moschitti, A., Riccardi, G.: Annotating spoken dialogs: from speech segments to dialog acts and frame semantics. In: Proceedings of the 2nd Workshop on Semantic Representation of Spoken Language, pp. 34–41. Association for Computational Linguistics (2009)Google Scholar
  3. 3.
    Dukes, K.: Semantic annotation of robotic spatial commands. In: Language and Technology Conference (LTC) (2013)Google Scholar
  4. 4.
    Faruqui, M., Dodge, J., Jauhar, S.K., Dyer, C., Hovy, E., Smith, N.A.: Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166 (2014)
  5. 5.
    Fillmore, C.J.: Frames and the semantics of understanding. Quad. Semantica 6(2), 222–254 (1985)Google Scholar
  6. 6.
    Galindo, C., Fernández-Madrigal, J.-A., González, J., Saffiotti, A.: Robot task planning using semantic maps. Robot. Auton. Syst. 56(11), 955–966 (2008)CrossRefGoogle Scholar
  7. 7.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  8. 8.
    Kollar, T., Tellex, S., Roy, D., Roy, N.: Grounding verbs of motion in natural language commands to robots. In: Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics. STAR, vol. 79, pp. 31–47. Springer, Berlin (2014). Scholar
  9. 9.
    Lison, P., Tiedemann, J.: Opensubtitles 2016: extracting large parallel corpora from movie and tv subtitles (2016)Google Scholar
  10. 10.
    Matuszek, C., Herbst, E., Zettlemoyer, L., Fox, D.: Learning to parse natural language commands to a robot control system. In: Desai, J., Dudek, G., Khatib, O., Kumar, V. (eds.) Experimental Robotics. STAR, vol. 88, pp. 403–415. (2013). Scholar
  11. 11.
    Pronobis, A., Jensfelt, P.: Large-scale semantic mapping and reasoning with heterogeneous modalities. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 3515–3522. IEEE (2012)Google Scholar
  12. 12.
    Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: an open multilingual graph of general knowledge. In: AAAI, pp. 4444–4451 (2017)Google Scholar
  13. 13.
    Speer, R., Lowry-Duda, J.: Conceptnet at semeval-2017 task 2: extending word embeddings with multilingual relational knowledge. arXiv preprint arXiv:1704.03560 (2017)
  14. 14.
    Surmann, H., Nüchter, A., Hertzberg, J.: An autonomous mobile robot with a 3D laser range finder for 3D exploration and digitalization of indoor environments. Robot. Auton. Syst. 45(3–4), 181–198 (2003)CrossRefGoogle Scholar
  15. 15.
    Vanzo, A., Part, J.L., Yu, Y., Nardi, D., Lemon, O.: Incrementally learning semantic attributes through dialogue interaction. In: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 865–873. International Foundation for Autonomous Agents and Multiagent Systems (2018)Google Scholar
  16. 16.
    Walter, M.R., Hemachandra, S., Homberg, B., Tellex, S., Teller, S.: Learning semantic maps from natural language descriptions. Robot. Sci. Syst. (2013) Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mohamadreza Faridghasemnia
    • 1
    Email author
  • Andrea Vanzo
    • 2
  • Daniele Nardi
    • 1
  1. 1.Department of Computer, Control and Management Engineering “Antonio Ruberti”Sapienza University of RomeRomeItaly
  2. 2.Mathematical and Computer Science SchoolHeriot-Watt UniversityEdinburghUK

Personalised recommendations