Skip to main content

A Spiking Neural Network Model of Multi-modal Language Processing of Robot Instructions

  • Chapter
Biomimetic Neural Learning for Intelligent Robots

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3575))

Abstract

Presented is a spiking neural network architecture of human language instruction recognition and robot control. The network is based on a model of a leaky Integrate-And-Fire (lIAF) spiking neurone with Active Dendrites and Dynamic Synapses (ADDS) [1,2,3]. The architecture contains several main modules associating information across different modalities: an auditory system recognising single spoken words, a visual system recognising objects of different colour and shape, motor control system for navigation and motor control and a working memory. The main focus of this presentation is the working memory module whose function is sequential processing of word from a language instruction, task and goal representation and cross-modal association of objects and actions. We test the model with a robot whose goal is to recognise and execute language instructions. The work demonstrates the potential of spiking neurons for processing spatio-temporal patterns and the experiments present spiking neural networks as a paradigm which can be applied for modelling sequence detectors at word level for robot instructions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Panchev, C., Wermter, S.: Spike-timing-dependent synaptic plasticity from single spikes to spike trains. Neurocomputing (2004) (to appear)

    Google Scholar 

  2. Panchev, C., Wermter, S., Chen, H.: Spike-timing dependent competitive learning of integrate-and-fire neurons with active dendrites. In: Dorronsoro, J.R. (ed.) ICANN 2002. LNCS, vol. 2415, pp. 896–901. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  3. Panchev, C., Wermter, S.: Temporal sequence detection with spiking neurons: towards recognizing robot language instructions. Connection Science 17(1) (2005)

    Google Scholar 

  4. Elman, J.L., Bates, E.A., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., Plunkett, K.: Rethinking Innateness. MIT Press, Cambridge (1996)

    Google Scholar 

  5. Bates, E.: Plasticity, localization and language development. In: Broman, S., Fletcher, J. (eds.) The changing nervous system: Neurobehavioral consequences of early brain disorders. Oxford University Press, New York (1999)

    Google Scholar 

  6. Goodglass, H.: Understanding aphasia. Technical report, University of California. Academic Press, San Diego (1993)

    Google Scholar 

  7. Bates, E., Vicari, S., Trauner, D.: Neural mediation of language development: Perspectives from lesion studies of infants and children. In: Tager-Flusberg, H. (ed.) Neurodevelopmental disorders, pp. 533–581. MIT Press, Cambridge (1999)

    Google Scholar 

  8. Pulvermuller, F.: The Neuroscience of Language: On Brain Circuits of Words and Serial Order. Cambridge University Press, Cambridge (2002)

    Google Scholar 

  9. Bates, E., Thal, D., Trauner, D., Fenson, J., Aram, D., Eisele, J., Nass, R.: From first words to grammar in children with focal brain injury. Special issue on Origins of Communication Disorders, Developmental Neuropsychology, 275–343 (1997)

    Google Scholar 

  10. Stiles, J., Bates, E., Thal, D., Trauner, D., Reilly, J.: Linguistic, cognitive and affective development in children with pre- and perinatal focal brain injury: A ten-year overview from the san diego longitudinal project. In: Rovee-Collier, C., Hayne, H. (eds.) Advances in infancy research, pp. 131–163. Ablex, Norwood (1998)

    Google Scholar 

  11. Lenneberg, E.H.: Biological foundations of language. Wiley, New York (1962)

    Google Scholar 

  12. Stromswold, K.: The cognitive and neural bases of language acquisition. In: Gazzaniga, M.S. (ed.) The Cognitive Neuroscience. MIT Press, Cambridge (1995)

    Google Scholar 

  13. Jackson, J.H.: On affections of speech from disease of the brain. Brain 1, 304–330 (1878)

    Article  Google Scholar 

  14. Broca, P.: Remarques sur le siége de la faculté du langage articulé, suivies d’une observation d’aphemie (perte de la parole). Bulletin de la Société d’Anatomie 36, 330–357 (1861)

    Google Scholar 

  15. Wernike, C.: Der Aphasische Symtomenkomplex. Eine Psychologische Studie auf Anatomischer Basis. M. Cohn und Weigart, Breslau (1874)

    Google Scholar 

  16. Caplan, D.: Neurolinguisitcs and linguistic aphasiology: An introduction. Cambridge University Press, Cambridge (1987)

    Book  Google Scholar 

  17. Neininger, B., Pulvermüller, F.: The right hemisphere’s role in action word processing: a double case study. Neurocase 7, 303–320 (2001)

    Article  Google Scholar 

  18. Pulvermüller, F., Mohr, B.: The concept of transcortical cell assemblies: a key to the understanding of cortical lateralization and interhemispheric interaction. Neuroscience and Biobehavioral Reviews 20, 557–566 (1996)

    Article  Google Scholar 

  19. Ojemann, G.: Cortical organisation of language. Journal of Neuroscience 11(8), 20–39 (1991)

    Google Scholar 

  20. Price, C., Green, D., Studnitz, R.: A functional imaging study of translation and language switching. Brain 122, 2221–2235 (1999)

    Article  Google Scholar 

  21. Mazoyer, B.M., Tzourio, N., Frak, V., Syrota, A., Murayama, N., Levrier, O.: The cortical representation of speech. Journal of Cognitive Neuroscience 5, 467–479 (1993)

    Article  Google Scholar 

  22. Perani, D., Dehaene, S., Grassi, F., Cohen, L., Cappa, S., Dupoux, E.: Brain processing of native and foreign languages. Neuroreport 7, 2439–2444 (1996)

    Article  Google Scholar 

  23. Dehaene, S., Dupoux, E., Mehler, J., Cohen, L., Paulesu, E., Perani, D.: Anatomical variability in the cortical representation of first and second languages. Neuroreport 8, 3809–3815 (1997)

    Article  Google Scholar 

  24. Bavelier, D., Corina, D., Jezzard, P., Clark, V., Karni, A., Lalwani, A.: Hemispheric specialization for english and asl: left invariance-right variability. Neuroreport 9, 1537–1542 (1998)

    Article  Google Scholar 

  25. Kim, K.H., Relkin, N.R., Lee, K.M., Hirsch, J.: Distinct cortical areas associated with native and second languages. Nature 388, 171–174 (1997)

    Article  Google Scholar 

  26. Posner, M.I., DiGirolamo, G.J.: Flexible neural circuitry in word processing. Behavioral and Brain Sciences 22, 299–300 (1999)

    Article  Google Scholar 

  27. Freud, S.: Zur Auffassung der Aphasien. Franz Deuticke, Wien (1891); English tanslation: On Aphasia: A Critical Study. Translated by Stengel, E. International Universities Press, New York (1953)

    Google Scholar 

  28. Pulvermüller, F.: A brain perspective on language mechanisms: from discrete neuronal ensembles to serial order. Progress in Neurobiology 67(2), 85–111 (2002)

    Article  Google Scholar 

  29. Stein, B.E., Meredith, M.A.: Merging of the senses. MIT Press, Cambridge (1993)

    Google Scholar 

  30. Meredith, M.A.: On the neuronal basis for multisensory convergence: A brief overview. Cognitive Brain Research 14, 31–40 (2002)

    Article  Google Scholar 

  31. Shimojo, S., Shams, L.: Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology 11, 505–509 (2001)

    Article  Google Scholar 

  32. von Melchner, L., Pallas, S.L., Sur, M.: Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404, 871–876 (2000)

    Article  Google Scholar 

  33. Blake, R., Grossman, E.: Neural synergy between seeing and hearing, draft (2000)

    Google Scholar 

  34. Lappe, M.: Information transfer between sensory and motor networks. In: Moss, F., Gielen, S. (eds.) Handbook of bilogical physics, vol. 4. Elsevier Science, Amsterdam (2000)

    Google Scholar 

  35. Fuster, J.M., Bodner, M., Kroger, J.K.: Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405, 347–351 (2000)

    Article  Google Scholar 

  36. Zhou, Y.D., Fuster, J.M.: Visio-tactile cross-modal associations in cortical somatosensory cells. Proceedings of National Academy of Sciences U.S.A. 97, 9777–9782 (2000)

    Article  Google Scholar 

  37. Rizzolatti, G., Luppino, G., Matelli, M.: The organization of the cortical motor system: new concepts. Electroencephalography and Clinical Neurophysiology 106(4), 283–296 (1998)

    Article  Google Scholar 

  38. Barbas, H.: Connections underlying the synthesis of cognition, memory, and emotion in primate prefrontal cortices. Brain Research Bulletin 52(5), 319–330 (2000)

    Article  Google Scholar 

  39. Lewis, J.W., van Essen, D.C.: Corticocortical connections of visual, sensimotor, and multimodal processing areas in the parietal lobe of the macaque monkey. Journal of Computational Neurology 428, 112–137 (2000)

    Article  Google Scholar 

  40. Pandya, D.N., Yeterian, E.H.: Architecture and connections of cortical association areas. In: Peters, A., Jones, E. (eds.) Cerebral Cortex, Association and Auditory Cortices, vol. 4, pp. 3–61. Plenum Press, London (1985)

    Google Scholar 

  41. Deacon, T.W.: Cortical connections of the inferior arcuate sulcus cortex in the macaque brain. Brain Research 573, 8–26 (1992)

    Article  Google Scholar 

  42. Pulvermüller, F.: Brain reflections of words and their meaning. Trends in Cognitive Sciences 5(12), 517–524 (2001)

    Article  Google Scholar 

  43. Landauer, T.K., Dumais, S.T.: A solution to plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review 104, 211–254 (1997)

    Article  Google Scholar 

  44. Pulvermüller, F.: Words in the brain’s language. Behavioral and Brain Sciences 22, 253–336 (1999)

    Article  Google Scholar 

  45. Pulvermüller, F.: Words in the brain’s language. Behavioral and Brain Sciences 22, 253–336 (1999)

    Article  Google Scholar 

  46. Brent, M.R., Cartwright, T.A.: Distributional regularity and phonotactic constraints are useful for segmentation. Cognition 61(1-2), 93–125 (1996)

    Article  Google Scholar 

  47. Harris, Z.S.: From phonemes to morphemes. Language 28, 1–30 (1955)

    Article  Google Scholar 

  48. Redlich, A.N.: Redundancy reduction as a strategy for unsupervised learning. Neural Computation 3, 289–304 (1993)

    Article  Google Scholar 

  49. Shaffran, J.R., Aslin, R.N., Newport, E.L.: Statistical learning by 8-month-old infants. Science 274(5294), 1926–1934 (1996)

    Article  Google Scholar 

  50. Shannon, C.E., Weaver, W.: The Mathematical Theory of Communication. University of Illinois Press, Urbana (1963)

    MATH  Google Scholar 

  51. Locke, J.L.: Babbling and early speech: continuity and individual differences. First Language 9, 191–206 (1989)

    Article  Google Scholar 

  52. Locke, J.L.: The child’s path to spoken language. Harvard University Press, Cambridge (1993)

    Google Scholar 

  53. Marslen-Wilson, W., Tyler, L.K.: The temporal structure of spoken language understanding. Cognition 8, 1–71 (1980)

    Article  Google Scholar 

  54. Rugg, M.D.: Further study of electrophysiological correlates of lexical decision. Brain and Language 19, 142–194 (1983)

    Article  Google Scholar 

  55. Price, C.J., Wise, R.J.S., Frackowiak, R.S.J.: Demonstrating the implicit processing of visually presented words and pseudowords. Cerebral Cortex 6, 62–70 (1996)

    Article  Google Scholar 

  56. Korpilahti, P., Krause, C.M., Holopained, I., Lang, A.H.: Early and late mismatch negativity elicited by words and speech-like stimuli in children. Brain and Language 76, 332–371 (2001)

    Article  Google Scholar 

  57. Pulvermüller, F.T.K., Shtyrov, Y., Simola, J., Tiitinen, H., Alku, P., Alho, K., Martinkauppi, S., Ilmoniemi, R.J., Näätänen, R.: Memory traces for words as revealed by the mismatch negativity. Neuroimage 14, 607–623 (2001)

    Article  Google Scholar 

  58. Shtyrov, Y., Pulvermüller, F.: Neurophysiological evidence for memory traces for words in the human brain. Neuroreport 13, 521–525 (2002)

    Article  Google Scholar 

  59. Aboitiz, F., García, V.R.: The evolutionary origin of the language areas in the human brain. a neuroanatomical percpective. Brain Research Reviews 25, 381–396 (1997)

    Article  Google Scholar 

  60. Rizzolatti, G., Arbib, M.: Language within our grasp. Trends in Neurosciences 21, 188–194 (1998)

    Article  Google Scholar 

  61. Arbib, M.A.: The mirror neuron hypothesis for language ready brain. In: Angelo, C., Domenico, P. (eds.) Computational approaches to the evolution of language and communication, pp. 229–254. Springer, Berlin (2001)

    Google Scholar 

  62. Arbib, M.A.: The mirror system, imitation, and the evolution of language. In: Christopher, N., Kerstin, D. (eds.) Imitation in animals and artifacts, pp. 229–280. MIT Press, Cambridge (2002)

    Google Scholar 

  63. Arbib, M., Bota, M.: Language evolution: neural homologies and neuroinformatics. Neural Networks 16(9), 1237–1260 (2003)

    Article  Google Scholar 

  64. Arbib, M.: Rana computatrix to human language: towards a computational neuroethology of language evolution. Philosophical Transactions: Mathematical, Physical and Engineering Sciences 361(1811), 2345–2379 (2003)

    Article  MathSciNet  Google Scholar 

  65. Arbib, M.: From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences (2005)

    Google Scholar 

  66. Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G.: Action recognition in the premotor cortex. Brain 119, 593–609 (1996)

    Article  Google Scholar 

  67. Gallese, V., Keysers, C.: Mirror neurons: a sensorimotor representation system. Behavioral Brain Sciences 24(5), 983–984 (2001)

    Google Scholar 

  68. Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Paulesu, E., Perani, D., Fazio, F.: Localization of grasp representations in human by pet: 1. observation versus execution. Exp. Brain Res. 111, 246–252 (1996)

    Article  Google Scholar 

  69. Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Paulesu, E., Perani, D., Fazio, F.: Parietal cortex: from sight to action. Experimental Brain Research 11(2), 246–252 (1996)

    Google Scholar 

  70. Gallese, V., Goldman, A.: Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science 2, 493–501 (1998)

    Article  Google Scholar 

  71. Rizzolatti, G., Fogassi, L., Gallese, V.: Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Review 2, 661–670 (2001)

    Article  Google Scholar 

  72. Gallese, V.: Echo mirror neurons: Recognizing action by sound. In: Human Frontiers Science Program Workshop Mirror System: Humans, Monkeys and Models, University of Southern California, Los Angelis, CA (2001)

    Google Scholar 

  73. Kohler, E., Keysers, C., Umilta, M., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)

    Article  Google Scholar 

  74. Keysers, C., Kohler, E., Umiltà, M., Fogassi, L., Nanetti, L., Gallese, V.: Audio-visual mirror neurones and action recognition. Experimental Brain Research 153, 628–636 (2003)

    Article  Google Scholar 

  75. Zukow-Goldring, P., Arbib, M., Oztop, E.: Language and the mirror system: A perception/action based approach to cognitive development (2002) (in preparation)

    Google Scholar 

  76. Burns, B., Sutton, C., Morrison, C., Cohen, P.: Information theory and representation in associative word learning. In: Prince, C., Berthouze, L., Kozima, H., Bullock, D., Stojanov, G., Balkenius, C. (eds.) Proceedings Third International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Boston, MA, USA, pp. 65–71 (2003)

    Google Scholar 

  77. Oates, T.: Grounding Knowledge in Sensors: Unsupervised Learning for Language and Planning. PhD thesis, University of Massacgusetts, Amherst (2001)

    Google Scholar 

  78. Siskind, J.M.: Learning word-to-meaning mappings. In: Broeder, P., Murre, J. (eds.) Models of Language Acquisition: Inductive and Deductive Approaches, pp. 121–153. Oxford University Press, Oxford (2000)

    Google Scholar 

  79. Steels, L.: The Talking Heads Experiment. Words and Meanings, vol. 1. Antwerpen (1999)

    Google Scholar 

  80. Steels, L., Kaplan, F.: Bootstrapping grounded word semantics. In: Briscoe, T. (ed.) Linguistic evolution through language acquisition: formal and computational models. Cambridge University Press, Cambridge (1999)

    Google Scholar 

  81. Steels, L., Kaplan, F.: AIBO’s first words. the social learning of language and meaning (2001) (preprint)

    Google Scholar 

  82. Arbib, M.A., Grethe, J.S. (eds.): Computing the Brain: A Guide to Neuroinformatics. Academic Press, San Diego (2001)

    Google Scholar 

  83. Shtyrov, Y., Hauk, O., Pulvermüller, F.: Distributed neuronal networks for encoding catergory-specific semantic information: The mismatch negative to action words. European Journal of Neuroscience 19, 1–10 (2004)

    Article  Google Scholar 

  84. Wolfe, J., Cave, K.: The psychophysical evidence for a binding proble in human vision. Neuron. 24(1), 11–17 (1999)

    Article  Google Scholar 

  85. Triesman, A.: Solutions to the binding problem: Progress through controversy and convergence. Neuron. 24(1), 105–110 (1999)

    Article  Google Scholar 

  86. Treisman, A., Schmidt, H.: Illusory conjunctions in the perception of objects. Cognitive Psychology 14, 107–141 (1982)

    Article  Google Scholar 

  87. von der Malsburg, C.: The correlation theory of brain function. Technical report, Max-Planck-Institute for Biophysical Chemistry (1981) Internal Report 81-2

    Google Scholar 

  88. Rosenblatt, F.: Principles of Neurodynamics: Perceptions and the Theory of Brain Mechanisms. Spartan Books, Washington (1961)

    Google Scholar 

  89. Roskies, A.: The binding problem. Neuron. 24(1), 7–9 (1999)

    Article  Google Scholar 

  90. Ghose, G., Maunsell, J.: Specialized representations in the visual cortext: A role for binding. Neuron. 24(1), 79–85 (1999)

    Article  Google Scholar 

  91. Riesenhuber, M., Poggio, T.: Are cortical models really bound by the binding problem. Neuron. 24(1), 87–93 (1999)

    Article  Google Scholar 

  92. Barlow, H.B.: Single units and sensation: A neuron doctrine for perceptual psychology? Perception 1, 371–394 (1972)

    Article  Google Scholar 

  93. Van Essen, D., Gallant, J.: Neural mechanisms of form and motion processing in the primate visual system. Neuron. 13, 1–10 (1994)

    Article  Google Scholar 

  94. Kobatake, E., Tanaka, K.: Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortext. Journal of Neurophysiology 71, 856–857 (1994)

    Google Scholar 

  95. Gallant, J.L., Connor, C.E., Rakshit, S., Lewis, J., Van Essen, D.: Neural responses to polar, hyperbolic, and cartesian gratings in area v4 of the macaque monkey. Journal of Neurophysiology 76, 2718–2739 (1996)

    Google Scholar 

  96. Tallon-Baudry, C., Bertrand, O.: Oscillatory gamma activity in humans and its role in object representation. Trends in Cognitive Sciences 3, 151–162 (1999)

    Article  Google Scholar 

  97. Roelfsema, P.R., Engel, A.K., König, P., Singer, W.: Visio-motor integration is associated with zero time-lag synchronization among cortical areas. Nature 385, 157–161 (1997)

    Article  Google Scholar 

  98. Bressler, S.L., Coppola, R., Nakamura, R.: Episodic multiregional cortical coherence at multiple frequencies during visual task performance. Nature 366, 153–156 (1993)

    Article  Google Scholar 

  99. Moran, J., Desimone, R.: Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784 (1985)

    Article  Google Scholar 

  100. Reynolds, J., Desimone, R.: The role of neural mechanisms of attention in solving the binding problem. Neuron. 24(1), 19–29 (1999)

    Article  Google Scholar 

  101. Fuster, J.M.: Memory in the Cerebral Cortex. MIT Press, Cambridge (1995)

    Google Scholar 

  102. Goldman-Rakic, P.S.: Architecture of the prefrontal cortex and the central executive. In: Annals of the New York Academy of Science, New York, vol. 769, pp. 71–83 (1995)

    Google Scholar 

  103. Wilson, F.A., Ó Scalaide, S.P., Goldman-Rakic, P.S.: Dissociation of object and spatial processing domains in primate prefrontal cortex. Science 260, 1955–1958 (1993)

    Article  Google Scholar 

  104. Lisman, J., Idiart, M.A.P.: Storage of 7+/-2 short-term memories in oscillatory subcycles. Science 267, 1512–1515 (1995)

    Article  Google Scholar 

  105. Idiart, M.A.P., Lisman, J.: Short-term memory as a single cell phenomenon. In: Bower, J.M. (ed.) The Neurobiology of Computation: Proceedings of the third annual computational and neural systems conference. Kluwer Academic Publishers, Dordrecht (1995)

    Google Scholar 

  106. Jensen, O., Lisman, J.E.: Novel lists of 7 | 2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory. Learning and Memory 3, 257–263 (1996)

    Article  Google Scholar 

  107. Jensen, O., Lisman, J.E.: Theta/gamma networks with slow nmda channels learn sequences and encode episodic memory: role of nmda channels in recall. Learning and Memory (1996)

    Google Scholar 

  108. Jensen, O., Lisman, J.E.: Hippocampal sequence-encoding driven by a cortical multi-item working memroy buffer. Trends in Neurosciences 28(2), 67–72 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Panchev, C. (2005). A Spiking Neural Network Model of Multi-modal Language Processing of Robot Instructions. In: Wermter, S., Palm, G., Elshaw, M. (eds) Biomimetic Neural Learning for Intelligent Robots. Lecture Notes in Computer Science(), vol 3575. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11521082_11

Download citation

  • DOI: https://doi.org/10.1007/11521082_11

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-27440-7

  • Online ISBN: 978-3-540-31896-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics