Abstract
Presented is a spiking neural network architecture of human language instruction recognition and robot control. The network is based on a model of a leaky Integrate-And-Fire (lIAF) spiking neurone with Active Dendrites and Dynamic Synapses (ADDS) [1,2,3]. The architecture contains several main modules associating information across different modalities: an auditory system recognising single spoken words, a visual system recognising objects of different colour and shape, motor control system for navigation and motor control and a working memory. The main focus of this presentation is the working memory module whose function is sequential processing of word from a language instruction, task and goal representation and cross-modal association of objects and actions. We test the model with a robot whose goal is to recognise and execute language instructions. The work demonstrates the potential of spiking neurons for processing spatio-temporal patterns and the experiments present spiking neural networks as a paradigm which can be applied for modelling sequence detectors at word level for robot instructions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Panchev, C., Wermter, S.: Spike-timing-dependent synaptic plasticity from single spikes to spike trains. Neurocomputing (2004) (to appear)
Panchev, C., Wermter, S., Chen, H.: Spike-timing dependent competitive learning of integrate-and-fire neurons with active dendrites. In: Dorronsoro, J.R. (ed.) ICANN 2002. LNCS, vol. 2415, pp. 896–901. Springer, Heidelberg (2002)
Panchev, C., Wermter, S.: Temporal sequence detection with spiking neurons: towards recognizing robot language instructions. Connection Science 17(1) (2005)
Elman, J.L., Bates, E.A., Johnson, M.H., Karmiloff-Smith, A., Parisi, D., Plunkett, K.: Rethinking Innateness. MIT Press, Cambridge (1996)
Bates, E.: Plasticity, localization and language development. In: Broman, S., Fletcher, J. (eds.) The changing nervous system: Neurobehavioral consequences of early brain disorders. Oxford University Press, New York (1999)
Goodglass, H.: Understanding aphasia. Technical report, University of California. Academic Press, San Diego (1993)
Bates, E., Vicari, S., Trauner, D.: Neural mediation of language development: Perspectives from lesion studies of infants and children. In: Tager-Flusberg, H. (ed.) Neurodevelopmental disorders, pp. 533–581. MIT Press, Cambridge (1999)
Pulvermuller, F.: The Neuroscience of Language: On Brain Circuits of Words and Serial Order. Cambridge University Press, Cambridge (2002)
Bates, E., Thal, D., Trauner, D., Fenson, J., Aram, D., Eisele, J., Nass, R.: From first words to grammar in children with focal brain injury. Special issue on Origins of Communication Disorders, Developmental Neuropsychology, 275–343 (1997)
Stiles, J., Bates, E., Thal, D., Trauner, D., Reilly, J.: Linguistic, cognitive and affective development in children with pre- and perinatal focal brain injury: A ten-year overview from the san diego longitudinal project. In: Rovee-Collier, C., Hayne, H. (eds.) Advances in infancy research, pp. 131–163. Ablex, Norwood (1998)
Lenneberg, E.H.: Biological foundations of language. Wiley, New York (1962)
Stromswold, K.: The cognitive and neural bases of language acquisition. In: Gazzaniga, M.S. (ed.) The Cognitive Neuroscience. MIT Press, Cambridge (1995)
Jackson, J.H.: On affections of speech from disease of the brain. Brain 1, 304–330 (1878)
Broca, P.: Remarques sur le siége de la faculté du langage articulé, suivies d’une observation d’aphemie (perte de la parole). Bulletin de la Société d’Anatomie 36, 330–357 (1861)
Wernike, C.: Der Aphasische Symtomenkomplex. Eine Psychologische Studie auf Anatomischer Basis. M. Cohn und Weigart, Breslau (1874)
Caplan, D.: Neurolinguisitcs and linguistic aphasiology: An introduction. Cambridge University Press, Cambridge (1987)
Neininger, B., Pulvermüller, F.: The right hemisphere’s role in action word processing: a double case study. Neurocase 7, 303–320 (2001)
Pulvermüller, F., Mohr, B.: The concept of transcortical cell assemblies: a key to the understanding of cortical lateralization and interhemispheric interaction. Neuroscience and Biobehavioral Reviews 20, 557–566 (1996)
Ojemann, G.: Cortical organisation of language. Journal of Neuroscience 11(8), 20–39 (1991)
Price, C., Green, D., Studnitz, R.: A functional imaging study of translation and language switching. Brain 122, 2221–2235 (1999)
Mazoyer, B.M., Tzourio, N., Frak, V., Syrota, A., Murayama, N., Levrier, O.: The cortical representation of speech. Journal of Cognitive Neuroscience 5, 467–479 (1993)
Perani, D., Dehaene, S., Grassi, F., Cohen, L., Cappa, S., Dupoux, E.: Brain processing of native and foreign languages. Neuroreport 7, 2439–2444 (1996)
Dehaene, S., Dupoux, E., Mehler, J., Cohen, L., Paulesu, E., Perani, D.: Anatomical variability in the cortical representation of first and second languages. Neuroreport 8, 3809–3815 (1997)
Bavelier, D., Corina, D., Jezzard, P., Clark, V., Karni, A., Lalwani, A.: Hemispheric specialization for english and asl: left invariance-right variability. Neuroreport 9, 1537–1542 (1998)
Kim, K.H., Relkin, N.R., Lee, K.M., Hirsch, J.: Distinct cortical areas associated with native and second languages. Nature 388, 171–174 (1997)
Posner, M.I., DiGirolamo, G.J.: Flexible neural circuitry in word processing. Behavioral and Brain Sciences 22, 299–300 (1999)
Freud, S.: Zur Auffassung der Aphasien. Franz Deuticke, Wien (1891); English tanslation: On Aphasia: A Critical Study. Translated by Stengel, E. International Universities Press, New York (1953)
Pulvermüller, F.: A brain perspective on language mechanisms: from discrete neuronal ensembles to serial order. Progress in Neurobiology 67(2), 85–111 (2002)
Stein, B.E., Meredith, M.A.: Merging of the senses. MIT Press, Cambridge (1993)
Meredith, M.A.: On the neuronal basis for multisensory convergence: A brief overview. Cognitive Brain Research 14, 31–40 (2002)
Shimojo, S., Shams, L.: Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology 11, 505–509 (2001)
von Melchner, L., Pallas, S.L., Sur, M.: Visual behaviour mediated by retinal projections directed to the auditory pathway. Nature 404, 871–876 (2000)
Blake, R., Grossman, E.: Neural synergy between seeing and hearing, draft (2000)
Lappe, M.: Information transfer between sensory and motor networks. In: Moss, F., Gielen, S. (eds.) Handbook of bilogical physics, vol. 4. Elsevier Science, Amsterdam (2000)
Fuster, J.M., Bodner, M., Kroger, J.K.: Cross-modal and cross-temporal association in neurons of frontal cortex. Nature 405, 347–351 (2000)
Zhou, Y.D., Fuster, J.M.: Visio-tactile cross-modal associations in cortical somatosensory cells. Proceedings of National Academy of Sciences U.S.A. 97, 9777–9782 (2000)
Rizzolatti, G., Luppino, G., Matelli, M.: The organization of the cortical motor system: new concepts. Electroencephalography and Clinical Neurophysiology 106(4), 283–296 (1998)
Barbas, H.: Connections underlying the synthesis of cognition, memory, and emotion in primate prefrontal cortices. Brain Research Bulletin 52(5), 319–330 (2000)
Lewis, J.W., van Essen, D.C.: Corticocortical connections of visual, sensimotor, and multimodal processing areas in the parietal lobe of the macaque monkey. Journal of Computational Neurology 428, 112–137 (2000)
Pandya, D.N., Yeterian, E.H.: Architecture and connections of cortical association areas. In: Peters, A., Jones, E. (eds.) Cerebral Cortex, Association and Auditory Cortices, vol. 4, pp. 3–61. Plenum Press, London (1985)
Deacon, T.W.: Cortical connections of the inferior arcuate sulcus cortex in the macaque brain. Brain Research 573, 8–26 (1992)
Pulvermüller, F.: Brain reflections of words and their meaning. Trends in Cognitive Sciences 5(12), 517–524 (2001)
Landauer, T.K., Dumais, S.T.: A solution to plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review 104, 211–254 (1997)
Pulvermüller, F.: Words in the brain’s language. Behavioral and Brain Sciences 22, 253–336 (1999)
Pulvermüller, F.: Words in the brain’s language. Behavioral and Brain Sciences 22, 253–336 (1999)
Brent, M.R., Cartwright, T.A.: Distributional regularity and phonotactic constraints are useful for segmentation. Cognition 61(1-2), 93–125 (1996)
Harris, Z.S.: From phonemes to morphemes. Language 28, 1–30 (1955)
Redlich, A.N.: Redundancy reduction as a strategy for unsupervised learning. Neural Computation 3, 289–304 (1993)
Shaffran, J.R., Aslin, R.N., Newport, E.L.: Statistical learning by 8-month-old infants. Science 274(5294), 1926–1934 (1996)
Shannon, C.E., Weaver, W.: The Mathematical Theory of Communication. University of Illinois Press, Urbana (1963)
Locke, J.L.: Babbling and early speech: continuity and individual differences. First Language 9, 191–206 (1989)
Locke, J.L.: The child’s path to spoken language. Harvard University Press, Cambridge (1993)
Marslen-Wilson, W., Tyler, L.K.: The temporal structure of spoken language understanding. Cognition 8, 1–71 (1980)
Rugg, M.D.: Further study of electrophysiological correlates of lexical decision. Brain and Language 19, 142–194 (1983)
Price, C.J., Wise, R.J.S., Frackowiak, R.S.J.: Demonstrating the implicit processing of visually presented words and pseudowords. Cerebral Cortex 6, 62–70 (1996)
Korpilahti, P., Krause, C.M., Holopained, I., Lang, A.H.: Early and late mismatch negativity elicited by words and speech-like stimuli in children. Brain and Language 76, 332–371 (2001)
Pulvermüller, F.T.K., Shtyrov, Y., Simola, J., Tiitinen, H., Alku, P., Alho, K., Martinkauppi, S., Ilmoniemi, R.J., Näätänen, R.: Memory traces for words as revealed by the mismatch negativity. Neuroimage 14, 607–623 (2001)
Shtyrov, Y., Pulvermüller, F.: Neurophysiological evidence for memory traces for words in the human brain. Neuroreport 13, 521–525 (2002)
Aboitiz, F., GarcÃa, V.R.: The evolutionary origin of the language areas in the human brain. a neuroanatomical percpective. Brain Research Reviews 25, 381–396 (1997)
Rizzolatti, G., Arbib, M.: Language within our grasp. Trends in Neurosciences 21, 188–194 (1998)
Arbib, M.A.: The mirror neuron hypothesis for language ready brain. In: Angelo, C., Domenico, P. (eds.) Computational approaches to the evolution of language and communication, pp. 229–254. Springer, Berlin (2001)
Arbib, M.A.: The mirror system, imitation, and the evolution of language. In: Christopher, N., Kerstin, D. (eds.) Imitation in animals and artifacts, pp. 229–280. MIT Press, Cambridge (2002)
Arbib, M., Bota, M.: Language evolution: neural homologies and neuroinformatics. Neural Networks 16(9), 1237–1260 (2003)
Arbib, M.: Rana computatrix to human language: towards a computational neuroethology of language evolution. Philosophical Transactions: Mathematical, Physical and Engineering Sciences 361(1811), 2345–2379 (2003)
Arbib, M.: From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences (2005)
Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G.: Action recognition in the premotor cortex. Brain 119, 593–609 (1996)
Gallese, V., Keysers, C.: Mirror neurons: a sensorimotor representation system. Behavioral Brain Sciences 24(5), 983–984 (2001)
Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Paulesu, E., Perani, D., Fazio, F.: Localization of grasp representations in human by pet: 1. observation versus execution. Exp. Brain Res. 111, 246–252 (1996)
Rizzolatti, G., Fadiga, L., Matelli, M., Bettinardi, V., Paulesu, E., Perani, D., Fazio, F.: Parietal cortex: from sight to action. Experimental Brain Research 11(2), 246–252 (1996)
Gallese, V., Goldman, A.: Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science 2, 493–501 (1998)
Rizzolatti, G., Fogassi, L., Gallese, V.: Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Review 2, 661–670 (2001)
Gallese, V.: Echo mirror neurons: Recognizing action by sound. In: Human Frontiers Science Program Workshop Mirror System: Humans, Monkeys and Models, University of Southern California, Los Angelis, CA (2001)
Kohler, E., Keysers, C., Umilta, M., Fogassi, L., Gallese, V., Rizzolatti, G.: Hearing sounds, understanding actions: action representation in mirror neurons. Science 297, 846–848 (2002)
Keysers, C., Kohler, E., Umiltà , M., Fogassi, L., Nanetti, L., Gallese, V.: Audio-visual mirror neurones and action recognition. Experimental Brain Research 153, 628–636 (2003)
Zukow-Goldring, P., Arbib, M., Oztop, E.: Language and the mirror system: A perception/action based approach to cognitive development (2002) (in preparation)
Burns, B., Sutton, C., Morrison, C., Cohen, P.: Information theory and representation in associative word learning. In: Prince, C., Berthouze, L., Kozima, H., Bullock, D., Stojanov, G., Balkenius, C. (eds.) Proceedings Third International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems, Boston, MA, USA, pp. 65–71 (2003)
Oates, T.: Grounding Knowledge in Sensors: Unsupervised Learning for Language and Planning. PhD thesis, University of Massacgusetts, Amherst (2001)
Siskind, J.M.: Learning word-to-meaning mappings. In: Broeder, P., Murre, J. (eds.) Models of Language Acquisition: Inductive and Deductive Approaches, pp. 121–153. Oxford University Press, Oxford (2000)
Steels, L.: The Talking Heads Experiment. Words and Meanings, vol. 1. Antwerpen (1999)
Steels, L., Kaplan, F.: Bootstrapping grounded word semantics. In: Briscoe, T. (ed.) Linguistic evolution through language acquisition: formal and computational models. Cambridge University Press, Cambridge (1999)
Steels, L., Kaplan, F.: AIBO’s first words. the social learning of language and meaning (2001) (preprint)
Arbib, M.A., Grethe, J.S. (eds.): Computing the Brain: A Guide to Neuroinformatics. Academic Press, San Diego (2001)
Shtyrov, Y., Hauk, O., Pulvermüller, F.: Distributed neuronal networks for encoding catergory-specific semantic information: The mismatch negative to action words. European Journal of Neuroscience 19, 1–10 (2004)
Wolfe, J., Cave, K.: The psychophysical evidence for a binding proble in human vision. Neuron. 24(1), 11–17 (1999)
Triesman, A.: Solutions to the binding problem: Progress through controversy and convergence. Neuron. 24(1), 105–110 (1999)
Treisman, A., Schmidt, H.: Illusory conjunctions in the perception of objects. Cognitive Psychology 14, 107–141 (1982)
von der Malsburg, C.: The correlation theory of brain function. Technical report, Max-Planck-Institute for Biophysical Chemistry (1981) Internal Report 81-2
Rosenblatt, F.: Principles of Neurodynamics: Perceptions and the Theory of Brain Mechanisms. Spartan Books, Washington (1961)
Roskies, A.: The binding problem. Neuron. 24(1), 7–9 (1999)
Ghose, G., Maunsell, J.: Specialized representations in the visual cortext: A role for binding. Neuron. 24(1), 79–85 (1999)
Riesenhuber, M., Poggio, T.: Are cortical models really bound by the binding problem. Neuron. 24(1), 87–93 (1999)
Barlow, H.B.: Single units and sensation: A neuron doctrine for perceptual psychology? Perception 1, 371–394 (1972)
Van Essen, D., Gallant, J.: Neural mechanisms of form and motion processing in the primate visual system. Neuron. 13, 1–10 (1994)
Kobatake, E., Tanaka, K.: Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortext. Journal of Neurophysiology 71, 856–857 (1994)
Gallant, J.L., Connor, C.E., Rakshit, S., Lewis, J., Van Essen, D.: Neural responses to polar, hyperbolic, and cartesian gratings in area v4 of the macaque monkey. Journal of Neurophysiology 76, 2718–2739 (1996)
Tallon-Baudry, C., Bertrand, O.: Oscillatory gamma activity in humans and its role in object representation. Trends in Cognitive Sciences 3, 151–162 (1999)
Roelfsema, P.R., Engel, A.K., König, P., Singer, W.: Visio-motor integration is associated with zero time-lag synchronization among cortical areas. Nature 385, 157–161 (1997)
Bressler, S.L., Coppola, R., Nakamura, R.: Episodic multiregional cortical coherence at multiple frequencies during visual task performance. Nature 366, 153–156 (1993)
Moran, J., Desimone, R.: Selective attention gates visual processing in the extrastriate cortex. Science 229, 782–784 (1985)
Reynolds, J., Desimone, R.: The role of neural mechanisms of attention in solving the binding problem. Neuron. 24(1), 19–29 (1999)
Fuster, J.M.: Memory in the Cerebral Cortex. MIT Press, Cambridge (1995)
Goldman-Rakic, P.S.: Architecture of the prefrontal cortex and the central executive. In: Annals of the New York Academy of Science, New York, vol. 769, pp. 71–83 (1995)
Wilson, F.A., Ó Scalaide, S.P., Goldman-Rakic, P.S.: Dissociation of object and spatial processing domains in primate prefrontal cortex. Science 260, 1955–1958 (1993)
Lisman, J., Idiart, M.A.P.: Storage of 7+/-2 short-term memories in oscillatory subcycles. Science 267, 1512–1515 (1995)
Idiart, M.A.P., Lisman, J.: Short-term memory as a single cell phenomenon. In: Bower, J.M. (ed.) The Neurobiology of Computation: Proceedings of the third annual computational and neural systems conference. Kluwer Academic Publishers, Dordrecht (1995)
Jensen, O., Lisman, J.E.: Novel lists of 7 | 2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory. Learning and Memory 3, 257–263 (1996)
Jensen, O., Lisman, J.E.: Theta/gamma networks with slow nmda channels learn sequences and encode episodic memory: role of nmda channels in recall. Learning and Memory (1996)
Jensen, O., Lisman, J.E.: Hippocampal sequence-encoding driven by a cortical multi-item working memroy buffer. Trends in Neurosciences 28(2), 67–72 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Panchev, C. (2005). A Spiking Neural Network Model of Multi-modal Language Processing of Robot Instructions. In: Wermter, S., Palm, G., Elshaw, M. (eds) Biomimetic Neural Learning for Intelligent Robots. Lecture Notes in Computer Science(), vol 3575. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11521082_11
Download citation
DOI: https://doi.org/10.1007/11521082_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-27440-7
Online ISBN: 978-3-540-31896-5
eBook Packages: Computer ScienceComputer Science (R0)