Skip to main content
SpringerLink
Log in
Book cover

Biomimetic Neural Learning for Intelligent Robots pp 162–181Cite as

Grounding Neural Robot Language in Action

  • Chapter

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3575))

Abstract

In this paper we describe two models for neural grounding of robotic language processing in actions. These models are inspired by concepts of the mirror neuron system in order to produce learning by imitation by combining high-level vision, language and motor command inputs. The models learn to perform and recognise three behaviours, ‘go’, ‘pick’ and ‘lift’. The first single-layer model uses an adapted Helmholtz machine wake-sleep algorithm to act like a Kohonen self-organising network that receives all inputs into a single layer. In contrast, the second, hierarchical model has two layers. In the lower level hidden layer the Helmholtz machine wake-sleep algorithm is used to learn the relationship between action and vision, while the upper layer uses the Kohonen self-organising approach to combine the output of the lower hidden layer and the language input.

On the hidden layer of the single-layer model, the action words are represented on non-overlapping regions and any neuron in each region accounts for a corresponding sensory-motor binding. In the hierarchical model rather separate sensory- and motor representations on the lower level are bound to corresponding sensory-motor pairings via the top level that organises according to the language input.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Billard, A., Hayes, G.: Drama, a connectionist architecture for control and learning in autonomous robots. Behavior Journal 7(1), 35–64 (1999)

    Google Scholar 

  2. Arbib, M.: From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Science, 1–9 (2004)

    Google Scholar 

  3. Bailey, D., Chang, N., Feldman, J., Narayanan, S.: Extending embodied lexical development. In: Proceedings of the Nineteenth Annual Meeting of the Cognitive Science Conference (1998)

    Google Scholar 

  4. Buccino, G., Vogt, S., Ritzi, A., Fink, G., Zilles, K., Freund, H.-J., Rizzolatti, G.: Neural circuits underlying imitation learning of hand actions: An event-related fMRI study. Neuron. 42, 323–334 (2004)

    Article  Google Scholar 

  5. Burgard, W., Cremers, A.B., Fox, D., Hahnel, D., Lakemeyer, G., Schulz, D., Steiner, W., Thrun, S.: Experiences with an interactive museum tour-guide robot. Artificial Intelligence 114(1-2) (2000)

    Google Scholar 

  6. Dayan, P.: Helmholtz machines and wake-sleep learning. In: Arbib, M. (ed.) Handbook of Brain Theory and Neural Network. MIT Press, Cambridge (2000)

    Google Scholar 

  7. Demiris, Y., Hayes, G.: Imitation as a dual-route process featuring prediction and learning components: a biologically-plausible computational model. In: Dautenhaln, K., Nehaniv, C. (eds.) Imitation in animals and artifacts. MIT Press, Cambridge (2002)

    Google Scholar 

  8. Gallese, V., Goldman, A.: Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science 2(12), 493–501 (1998)

    Article  Google Scholar 

  9. Glenberg, A., Kaschak, M.: Grounding language in action. Psychonomic Bulletin and Review 9, 558–565 (2002)

    Article  Google Scholar 

  10. Harnad, S.: The symbol grounding problem. Physica D 42, 335–346 (1990)

    Article  Google Scholar 

  11. Harnad, S.: The symbol grounding problem. In: Encyclopedia of Cognitive Science, Dublin (2003)

    Google Scholar 

  12. Hauk, O., Pulvermüller, F.: Neurophysiological distinction of action words in the frontal lobe: An ERP study using minimum current estimates. Human Brain Mapping, 1–9 (2004)

    Google Scholar 

  13. Hayes, G., Demiris, J.: A robot controller using learning by imitation. In: Proceedings of the 2nd International Symposium on Intelligent Robotic Systems, Greijcnnnoble, France (1994)

    Google Scholar 

  14. Infantino, I., Chella, A., Dzindo, H., Macaluso, I.: A posture sequence learning system for an anthropomorphic robotic hand. In: Proceedings of the IROS 2003 Workshop on Robot Programming by Demonstration (2003)

    Google Scholar 

  15. Keysers, C., Kohler, E., Umiltà, M.A., Nanetti, L., Fogassi, L., Gallese, V.: Audio-visual mirror neurons and action recognition. Exp. Brain Res. 153, 628–636 (2003)

    Article  Google Scholar 

  16. Knight, R.: Contribution of human hippocampal region to novelty detection. Computer Speech and Language 383(6597), 256–259 (1996)

    Google Scholar 

  17. Kohler, E., Keysers, C., Umilta, M., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: Action representation in mirror neurons. Science 297, 846–848 (2002)

    Article  Google Scholar 

  18. Kohonen, T.: Self-Organizing Maps. Springer, Heidelberg (1997)

    MATH  Google Scholar 

  19. Opitz, B., Mecklinger, A., Friederici, A.D., von Cramon, D.Y.: The functional neuroanatomy of novelty processing: Integrating erp and fMRI results. Cerebral Cortex 9(4), 379–391 (1999)

    Article  Google Scholar 

  20. Penfield, W., Rasmussen, T.: The cerebral cortex of man. Macmillan, Cambridge (1950)

    Google Scholar 

  21. Pulvermüller, F.: Words in the brain’s language. Behavioral and Brain Sciences 22(2), 253–336 (1999)

    Article  Google Scholar 

  22. Pulvermüller, F., Assadollahi, R., Elbert, T.: Neuromagnetic evidence for early semantic access in word recognition. European Journal of Neuroscience 13, 201–205 (2001)

    Article  Google Scholar 

  23. Rizzolatti, G., Arbib, M.: Language within our grasp. Trends in Neuroscience 21(5), 188–194 (1998)

    Article  Google Scholar 

  24. Rizzolatti, G., Fogassi, L., Gallese, V.: Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Review 2, 661–670 (2001)

    Article  Google Scholar 

  25. Rizzolatti, G., Fogassi, L., Gallese, V.: Motor and cognitive functions of the ventral premotor cortex. Current Opinion in Neurobiology 12, 149–154 (2002)

    Article  Google Scholar 

  26. Rizzolatti, G., Luppino, G.: The cortical motor system. Neuron. 18(2), 889–901 (1995)

    Google Scholar 

  27. Rizzolatti, G., Luppino, G., Matelli, M.: The organization of the cortical motor system: New concepts. Electroencephalography and Clinical Neurophysiology 106, 283–296 (1998)

    Article  Google Scholar 

  28. Rolls, E.T.: The orbitofrontal cortex and reward. Cereb. Cortex 10(3), 284–294 (2000)

    Article  Google Scholar 

  29. Roy, D.: Learning visually grounded words and syntax of natural language sk. Computer Speech and Language 16(3) (2002)

    Google Scholar 

  30. Schaal, S.: Is imitation learning the route to humanoid robots. Trends in Cognitive Science 3(6), 233–242 (1999)

    Article  Google Scholar 

  31. Schaal, S., Ijspeert, A., Billard, A.: Computational approaches to motor learning by imitation. Transaction of the Royal Society of London: Serial B, Biological Sciences 358, 537–547 (2003)

    Article  Google Scholar 

  32. Thrun, S., Bennewitz, M., Burgard, W., Dellaert, F., Fox, D., Haehnel, D., Rosenberg, C., Roy, N., Schulte, J., Schulz, D.: Minerva: A second generation mobile tour-guide robot. In: Proceedings of the IEEE international conference on robotics and automation, ICRA 1999 (1999)

    Google Scholar 

  33. Umilta, M., Kohler, E., Gallese, V., Fogassi, L., Fadiga, L., Keysers, C., Rizzolatti, G.: I know what you are doing: A neurophysical study. Neuron. 31, 155–165 (2001)

    Article  Google Scholar 

  34. Vogt, P.: Grounding language about actions: Mobile robots playing follow me games. In: Meyer, J., Berthoz, A., Floreano, D., Roitblat, H., Wilson, S. (eds.) SAB 2000, Honolulu, Hawaii. MIT Press, Cambridge (2000)

    Google Scholar 

  35. Weber, C.: Self-organization of orientation maps, lateral connections, and dynamic receptive fields in the primary visual cortex. In: Dorffner, G., Bischof, H., Hornik, K. (eds.) ICANN 2001. LNCS, vol. 2130, p. 1147. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  36. Weber, C., Wermter, S., Zochios, A.: Robot docking with neural vision and reinforcement. Knowledge-Based Systems 17(2-4), 165–172 (2004)

    Article  Google Scholar 

  37. Wermter, S., Elshaw, M.: Learning robot actions based on self-organising language memory. Lecture Notes in Artificial Intelligent 16(5-6), 661–669 (2003)

    Google Scholar 

  38. Wermter, S., Weber, C., Elshaw, M., Panchev, C., Erwin, H., Pulvermüller, F.: Towards multimodal neural robot learning. Robotics and Autonomous Systems Journal 47, 171–175 (2004)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hybrid Intelligent Systems, School of Computing and Technology, University of Sunderland, St Peter’s Way, Sunderland, SR6 0DD, UK

    Stefan Wermter, Cornelius Weber & Mark Elshaw

  2. Institute of Neurophysiology, University of Parma, Via Volturno, 39/E, I-43100, Parma, Italy

    Vittorio Gallese

  3. Cognition and Brain Sciences Unit, Medical Research Council, 15 Chaucer Road, Cambridge, UK

    Friedemann Pulvermüller

Authors
  1. Stefan Wermter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cornelius Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Elshaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vittorio Gallese
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Friedemann Pulvermüller
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Sunderland, SR6 0DD, Sunderland, United Kingdom

    Stefan Wermter

  2. Institute of Neural Information Processing, University of Ulm, D-89069, Ulm, Germany

    Günther Palm

  3. Hybrid Intelligent Systems, SCAT, University of Sunderland, UK

    Mark Elshaw

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Wermter, S., Weber, C., Elshaw, M., Gallese, V., Pulvermüller, F. (2005). Grounding Neural Robot Language in Action. In: Wermter, S., Palm, G., Elshaw, M. (eds) Biomimetic Neural Learning for Intelligent Robots. Lecture Notes in Computer Science(), vol 3575. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11521082_10

Download citation

  • DOI: https://doi.org/10.1007/11521082_10

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-27440-7

  • Online ISBN: 978-3-540-31896-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation