Vision and Action in the Language-Ready Brain: From Mirror Neurons to SemRep

  • Michael A. Arbib
  • JinYong Lee
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4729)


The general setting for our work is to locate language perception and production within the broader context of brain mechanisms for action and perception in general, modeling brain function in terms of the competition and cooperation of schemas. Particular emphasis is placed on mirror neurons – neurons active both for execution of a certain class of actions and for recognition of a (possibly broader) class of similar actions. We build on the early VISIONS model of schema-based computer analysis of static scenes to present SemRep, a graphical representation of dynamic visual scenes designed to support the generation of varied descriptions of episodes. Mechanisms for parsing and production of sentences are currently being implemented within Template Construction Grammar (TCG), a new form of construction grammar distinguished by its use of SemRep to express semantics.


action action recognition brain mechanisms competition and cooperation construction grammar dynamic visual scenes language perception language production mirror neurons visual perception scene descriptions schema theory SemRep vision 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arbib, M.A.: Rana computatrix to human language: towards a computational neuroethology of language evolution. Philos. Transact. A Math. Phys. Eng. Sci. 361, 2345–2379 (2003)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Arbib, M.A., Liaw, J.-S.: Sensorimotor Transformations in the Worlds of Frogs and Robots. Artificial Intelligence 72, 53–79 (1995)CrossRefGoogle Scholar
  3. 3.
    Arbib, M.A.: Perceptual structures and distributed motor control. In: Brooks, V.B. (ed.) Handbook of Physiology — The Nervous System II. Motor Control, pp. 1449–1480. American Physiological Society, Bethesda, MD (1981)Google Scholar
  4. 4.
    Draper, B.A., Collins, R.T., Brolio, J., Hanson, A.R., Riseman, E.M.: The schema system. International Journal of Computer Vision 2, 209–250 (1989)CrossRefGoogle Scholar
  5. 5.
    Erman, L.D., Hayes-Roth, F., Lesser, V.R., Reddy, D.R.: The HEARSAY-II speech understanding system: Integrating knowledge to resolve uncertainty. Computing Surveys 12, 213–253 (1980)CrossRefGoogle Scholar
  6. 6.
    Itti, L., Arbib, M.A.: Attention and the minimal subscene. In: Arbib, M.A. (ed.) Action to language via the mirror neuron system, pp. 289–346. Cambridge University Press, Cambridge (2006)Google Scholar
  7. 7.
    Fagg, A.H., Arbib, M.A.: Modeling parietal-premotor interactions in primate control of grasping. Neural Netw. 11, 1277–1303 (1998)CrossRefGoogle Scholar
  8. 8.
    Gibson, J.J.: The ecological approach to visual perception. Houghton Mifflin, Boston (1979)Google Scholar
  9. 9.
    Bonaiuto, J., Rosta, E., Arbib, M.: Extending the mirror neuron system model, I: Audible actions and invisible grasps. Biol. Cybern. 96, 9–38 (2007)zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Oztop, E., Arbib, M.A.: Schema design and implementation of the grasp-related mirror neuron system. Biol. Cybern. 87, 116–140 (2002)zbMATHCrossRefGoogle Scholar
  11. 11.
    Gershkoff-Stowe, L., Goldin-Meadow, S.: Is there a natural order for expressing semantic relations? Cognitive Psychology 45, 375–412 (2002)CrossRefGoogle Scholar
  12. 12.
    Arbib, M.A., Rizzolatti, G.: Neural expectations: a possible evolutionary path from manual skills to language. Communication and Cognition 29, 393–424 (1997)Google Scholar
  13. 13.
    Arbib, M.A.: From Monkey-like Action Recognition to Human Language: An Evolutionary Framework for Neurolinguistics (with commentaries and author’s response). Behavioral and Brain Sciences 28, 105–167 (2005)CrossRefGoogle Scholar
  14. 14.
    Arbib, M.A.: Broca’s Area in System Perspective: Language in the Context of Action-Oriented Perception. In: Grodzinsky, Y., Amunts, K. (eds.) Broca’s Region, pp. 153–168. Oxford University Press, Oxford (2006)Google Scholar
  15. 15.
    Saussure, F.: Cours de linguistique générale. In: Bally, C., Sechehaye, A., Riedlinger, A. (eds.) Payot Lausanne and Paris (1916) (English translation by W. Baskin, Course in General Linguistics, Fontana/Collins, Glasgow (1977))Google Scholar
  16. 16.
    Gallese, V., Goldman, A.: Mirror neurons and the simulation theory of mind-reading. Trends Cognit. Sci. 2, 493–501 (1998)CrossRefGoogle Scholar
  17. 17.
    Chomsky, N.: Lectures on Government and Binding. Foris, Dordrecht (1981)Google Scholar
  18. 18.
    Croft, W., Cruse, D.A.: Cognitive Linguistics. Cambridge University Press, Cambridge (2005)Google Scholar
  19. 19.
    Goldberg, A.E.: Constructions: A new theoretical approach to language. Trends in Cognitive Science 7, 219–224 (2003)CrossRefGoogle Scholar
  20. 20.
    Fillmore, C.J., Kay, P., O’Connor, M.K.: Regularity and idiomaticity in grammatical constructions: the case of let alone. Language & Cognitive Processes 64, 501–538 (1988)Google Scholar
  21. 21.
    De Beule, J., Steels, L.: Hierarchy in Fluid Construction Grammar. In: Furbach, U. (ed.) KI 2005. LNCS (LNAI), vol. 3698, pp. 1–15. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  22. 22.
    Roy, D.: Semiotic schemas: A framework for grounding language in action and perception. Artificial Intelligence 167, 170–205 (2005)CrossRefGoogle Scholar
  23. 23.
    Steels, L.: Evolving grounded communication for robots. Trends in Cognitive Sciences 7, 308–312 (2003)CrossRefGoogle Scholar
  24. 24.
    Cangelosi, A., Riga, T.: An Embodied Model for Sensorimotor Grounding and Grounding Transfer: Experiments With Epigenetic Robots. Cognitive Science 30, 673–689 (2006)CrossRefGoogle Scholar
  25. 25.
    Dominey, P.F., Hoen, M.: Structure mapping and semantic integration in a construction-based neurolinguistic model of sentence processing. Cortex 42, 476–479 (2006)Google Scholar
  26. 26.
    Feldman, J., Narayanan, S.: Embodied meaning in a neural theory of language. Brain Lang. 89, 385–392 (2004)CrossRefGoogle Scholar
  27. 27.
    Steels, L., Loetzsch, M.: Perspective Alignment in Spatial Language. In: Coventry, K.R., Tenbrink, T., Bateman, J.A. (eds.) Spatial Language and Dialogue, Oxford University Press, Oxford (2007)Google Scholar
  28. 28.
    Kosko, B.: Bidirectional associative memories. IEEE Transactions on Systems, Man and Cybernetics 18, 49–60 (1988)CrossRefMathSciNetGoogle Scholar
  29. 29.
    Navalpakkam, V., Itti, I.: Modeling the influence of task on attention. Vision Research 45, 205–231 (2005)CrossRefGoogle Scholar
  30. 30.
    Guazzelli, A., Corbacho, F.J., Bota, M., Arbib, M.A.: Affordances, Motivation, and the World Graph Theory. Adaptive Behavior 6, 435–471 (1998)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Michael A. Arbib
    • 1
    • 2
    • 3
  • JinYong Lee
    • 1
  1. 1.Computer Science 
  2. 2.Neuroscience 
  3. 3.USC Brain Project, University of Southern California, Los Angeles, CA 90089-2520USA

Personalised recommendations