Actions and Imagined Actions in Cognitive Robots

  • Vishwanathan MohanEmail author
  • Pietro Morasso
  • Giorgio Metta
  • Stathis Kasderidis
Part of the Springer Series in Cognitive and Neural Systems book series (SSCNS)


Natural/Artificial systems that are capable of utilizing thoughts at the service of their actions are gifted with the profound opportunity to mentally manipulate the causal structure of their physical interactions with the environment. A cognitive robot can in this way virtually reason about how an unstructured world should “change,” such that it becomes a little bit more conducive towards realization of its internal goals. In this article, we describe the various internal models for real/mental action generation developed in the GNOSYS Cognitive architecture and demonstrate how their coupled interactions can endow the GNOSYS robot with a preliminary ability to virtually manipulate neural activity in its mental space in order to initiate flexible goal-directed behavior in its physical space. Making things more interesting (and computationally challenging) is the fact that the environment in which the robot seeks to achieve its goals consists of specially crafted “stick and ball” versions of real experimental scenarios from animal reasoning (like tool use in chimps, novel tool construction in Caledonian crows, the classic trap tube paradigm, and their possible combinations). We specifically focus on the progressive creation of the following internal models in the behavioral repertoire of the robot: (a) a passive motion paradigm based forward inverse model for mental simulation/real execution of goal-directed arm (and arm + tool) movements; (b) a spatial mental map of the playground; and (c) an internal model representing the causality of pushing objects and further learning to push intelligently in order to avoid randomly placed traps in the trapping groove. After presenting the computational architecture for the internal models, we demonstrate how the robot can use them to mentally compose a sequence of “Push–Move–Reach” in order to Grasp (an otherwise unreachable) ball in its playground.


Internal Model Artificial Agent Reward Structure Lateral Connection Goal Object 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was partly supported by the EU FP6 project GNOSYS and EU FP7 projects iTalk (Grant No: 214668) and HUMOR (Grant No: 231724).


  1. Abbot, L. and Sejnowski, T.J. (1999). Neural codes and distributed representations. Cambridge, MA: MIT.Google Scholar
  2. Amari, S. (1977). Dynamics of patterns formation in lateral-inhibition type neural fields. Biological Cybernetics, 27, 77–87.CrossRefPubMedGoogle Scholar
  3. Atkeson, C.G., Hale, J.G., Pollick, F. (2000). Using humanoid robots to study human behavior. IEEE Intelligent Systems, 15, 46–56.Google Scholar
  4. Baltzakis, H. (2004). A hybrid framework for mobile robot navigation: modelling with switching state space networks. PhD Thesis, University of Crete.Google Scholar
  5. Behrmann, M. (2000). The mind’s eye mapped onto the brain’s matter. Current Directions in Psychological Science. April 2000 9, 50–54, doi:10.1111/1467-8721.00059.CrossRefGoogle Scholar
  6. Blair, H.T., Cho, J., Sharp, P.E. (1998). Role of the lateral mammillary nucleus in the rat head direction circuit: a combined single unit recording and lesion study. Neuron, 21, 1387–1397.CrossRefPubMedGoogle Scholar
  7. Brooks, R.A. (1986). A robust layered control system for a mobile robot. IEEE Journal of Robotics and Automation, 2(1), 14–23.Google Scholar
  8. Brooks, R.A. (1997). The Cog Project. T. Matsui (ed.), Special Issue (Mini) on Humanoid, Journal of the Robotics Society of Japan, vol. 15, No. 7.Google Scholar
  9. Clark, A. (1997). Being there: putting brain, body and world together again. Cambridge, MA: MIT.Google Scholar
  10. Cohen, M.S. (1996). Changes in cortical activity during mental rotation. A mapping study using functional MRI. Brain 119, 89–100.Google Scholar
  11. Damasio, A.R. (2000). The feeling of what happens: body, emotion and the making of consciousness. New York: Vintage.Google Scholar
  12. Edelman, G.M. (2006). Second nature: brain science and human knowledge. New Haven, London: Yale University Press.Google Scholar
  13. Edelman, G.M. and Tononi, G. (2001). A universe of consciousness: how matter becomes imagination. New York: Basic Books.Google Scholar
  14. Emery, N.J. and Clayton, N.S. (2004). The mentality of crows: convergent evolution of intelligence in corvids and apes. Science, 306, 1903–1907.CrossRefPubMedGoogle Scholar
  15. French, R.M. (2006). The dynamics of the computational modelling of analogy-making, CRC handbook of dynamic systems modelling. Fishwick, P. (ed.), Boca Raton, FL: CRC, LLC.Google Scholar
  16. Fritzke, B. (1995). A growing neural gas network learns topologies. In Tesauro, G., Touretzky, D., Leen, T. (eds.), Advances in neural information processing systems, 7 (pp. 625–632). Cambridge, MA: MIT.Google Scholar
  17. Fuster, J.M. (2003). Cortex and mind: unifying cognition. Oxford: Oxford University Press.Google Scholar
  18. Geffner, H. (1992). Default reasoning: causal and conditional theories. MIT Press.Google Scholar
  19. Georgeff, M.P. (1999). The belief-desire-intention model of agency. In Müller, J.P., Smith, M.P., Rao, A.S. (eds.), Intelligent agents, V LNAI. 1555, pp. 1–10. Berlin: Springer.Google Scholar
  20. Georges-Francois, P., Rolls, E.T., Robertson, R.G. (1999). Spatial view cells in the primate hippocampus: allocentric view not head direction or eye position or place. Cerebral Cortex, 9(3), 197–212.CrossRefPubMedGoogle Scholar
  21. Gnadt, W. and Grossberg, S. (2008). SOVEREIGN: an autonomous neural system for incrementally learning planned action sequences to navigate towards a rewarded goal. Neural Networks, 21, 699–758.CrossRefPubMedGoogle Scholar
  22. GNOSYS project documentation:
  23. Grush, R. (1995). Emulation and cognition, doctoral dissertation, University of California, San Diego.Google Scholar
  24. Grush, R. (2004). The emulation theory of representation: motor control, imagery, and perception. Behavioral and Brain Sciences, 27, 377–396.PubMedGoogle Scholar
  25. Hesslow, G. (2002). Conscious thought as a simulation of behavior and perception. Trends in Cognitive Sciences, 6(6), 242–247.CrossRefPubMedGoogle Scholar
  26. Hesslow, G. and Jirenhed, D.A. (2007). The inner world of a simple robot. Journal of Consciousness Studies, 14, 85–96.Google Scholar
  27. Hirose, M. and Ogawa, K. (2007). Honda humanoid robots development. Philos Transact A Math Phys Eng Sci, 365, 11–19.CrossRefPubMedGoogle Scholar
  28. Hofstadter, D.R. (1984). The Copycat project: an experiment in nondeterminism and creative reasoning in intelligent systems. San Fransisco, CA: Morgan Kaufmann.Google Scholar
  29. Holland, O. and Goodman, R. (2003). Robots with internal models: a route to machine consciousness? Journal of Consciousness Studies, Special Issue on Machine Consciousness, 10(4), 77–109.Google Scholar
  30. Imamizu, N. (2000). Human cereballar activity reflecting an acquired internal model of a new tool. Nature, 403, 192–196.CrossRefPubMedGoogle Scholar
  31. Klein, I., Paradis, A.L., Poline, J.B., Kosslyn, S.M., Le Bihan, D. (2000) Transient activity in the human calcarine cortex during visual-mental imagery: an event-related fMRI study. Journal of Cognitive Neuroscience, 12 Suppl 2, 15–23.CrossRefPubMedGoogle Scholar
  32. Kohonen, T. (1995). Self-organizing maps. Berlin: Springer.Google Scholar
  33. Kokinov, B.N. and Petrov, A. (2001). Integration of memory and reasoning in analogy-making: the AMBR model, the analogical mind: perspectives from cognitive science, Cambridge, MA: MIT.Google Scholar
  34. Kosslyn, S.M. et al. (1993). Visual mental imagery activates topographically organized visual cortex: pet investigations. Journal of Cognitive Neuroscience, 5, 263–287.CrossRefGoogle Scholar
  35. Metzinger, T. and Gallese, V. (2003). Motor ontology: the representational reality of goals, actions and selves, Philosophical Psychology, 16, 365–388.CrossRefGoogle Scholar
  36. Mohan, V. and Morasso, P. (2007a). Towards reasoning and coordinating action in the mental space. International Journal of Neural Systems, 17(4), 1–13.CrossRefGoogle Scholar
  37. Mohan, V. and Morasso, P. (2007b). Neural network of a cognitive crow: an interacting map based architecture. Proceedings of IEEE international conference on self organizing and self adaptive systems, MIT Boston, MA, USA.Google Scholar
  38. Mohan, V., Morasso, P., Metta, G., Sandini, G. (2009). A biomimetic, force-field based computational model for motion planning and bimanual coordination in humanoid robots. Autonomous Robots, 27(3), 291–301.CrossRefGoogle Scholar
  39. Morasso, P. (1981). Spatial control of arm movements. Experimental Brain Research, 42, 223–227.CrossRefGoogle Scholar
  40. Morasso, P. (2006). Consciousness as the emergent property of the interaction between brain body and environment: the crucial role of haptic perception, Artificial Consciousness, Exeter, UK: Imprint Academic.Google Scholar
  41. Mussa Ivaldi, F.A, Morasso, P., Zaccaria, R. (1988). Kinematic networks. A distributed model for representing and regularizing motor redundancy. Biological Cybernetics, 60, 1–16.Google Scholar
  42. Natale, L., Orabona, F., Metta, G., Sandini, G. (2007). Sensorimotor coordination in a “baby” robot: learning about objects through grasping. Prog Brain Res, 164, 403–424.CrossRefPubMedGoogle Scholar
  43. Newell, A. and Simon, H. (1976). Computer science as empirical enquiry: symbols and search, Communications of ACM, 19, 113–126.CrossRefGoogle Scholar
  44. Nishiwaki, K., Kuffner, J., Kagami, S., Inaba, M., Inoue, H. (2007). The experimental humanoid robot H7: a research platform for autonomous behaviour. Philos Transact A Math Phys Eng Sci, 365, 79–107.CrossRefPubMedGoogle Scholar
  45. O’Craven, K.M. and Kanwisher, N. (2000). Mental imagery of faces and places activates corresponding stimulus-specific brain regions. Journal of Cognitive Neuroscience, 12, 1013–1023.CrossRefPubMedGoogle Scholar
  46. O’Keefe, J. and Dostrovsky, J. (1971). The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Research 34, 171–175.Google Scholar
  47. Op de Beeck, H., Haushofer, J., Kanwisher, N. (2008). Interpreting fMRI data: maps, modules, and dimensions. Nature Reviews Neuroscience.Google Scholar
  48. Oztop, E. Wolpert, D., Kawato, M. (2004). Mental state inference using visual control parameters. Cognitive Brain Research 158, 480–503.Google Scholar
  49. Parsons, L.M., Sergent, J., Hodges, D.A., Fox, P.T. (2005). Cerebrally-lateralized mental representations of hand shape and movement. Journal of Neuroscience, 18, 6539–6548.Google Scholar
  50. Pearl, J. (1988). Probabilistic analogies. AI Memo No. 755. Cambridge, MA: Massachusetts Institute of Technology.Google Scholar
  51. Pearl, J (1998). Graphs, causality, and structural equation models, UCLA Cognitive Systems Laboratory, Technical Report R-253.Google Scholar
  52. Rizzolatti, G., Fogassi, L., Gallese, V. (2001). Neurophysiological mechanisms underlying the understanding and imitation of action. Nature Reviews Neuroscience, 2, 661–670.CrossRefPubMedGoogle Scholar
  53. Shadmehr, R. (1999). Evidence for a forward dynamic model: human adaptive motor cotnrol. News in Physiological Sciences, 11, 3–9.Google Scholar
  54. Shanahan, M.P. (2005). Perception as abduction: turning sensor data into meaningful representation. Cognitive Science, 29, 109–140.CrossRefGoogle Scholar
  55. Sun, R. (2000). Symbol grounding: a new look at an old idea. Philosophical Psychology, 13(2), 149–172.CrossRefGoogle Scholar
  56. Sun, R. (2007). The importance of cognitive architectures: An analysis based on CLARION. Journal of Experimental and Theoretical Artificial Intelligence, 19(2), 159–193.CrossRefGoogle Scholar
  57. Sutton, R. and Barto, A. (1998). Reinforcement learning. Cambridge, MA: MIT.Google Scholar
  58. Tani, J., Yokoya, R., Ogata, T., Komatani, K., Okuno, H.G. (2007). Experience-based imitation using RNNPB Advanced Robotics, 21(12), 1351–1367.Google Scholar
  59. Taylor, J.G. (2000). Attentional movement: the control basis for consciousness. Neuroscience Abstracts 26 (Part 2), 839(3), 2231.Google Scholar
  60. Toussaint, M. (2004). Learning a world model and planning with a self-organizing dynamic neural system. Advances in neural information processing systems 16 (NIPS 2003), pp. 929–936, Cambridge: MIT.Google Scholar
  61. Toussaint, M. (2006). A sensorimotor map: modulating lateral connections for anticipation and planning. Neural Computation, 18, 1132–1155.CrossRefPubMedGoogle Scholar
  62. Varela, F.J., Maturana, H.R., Uribe, R. (1974). Autopoiesis: the organization of living systems, its characterization and a model. Biosystems, 5, 187–196.CrossRefGoogle Scholar
  63. Visalberghi, E. (1993). Capuchin monkeys: a window into tool use activities by apes and humans. In Gibson, K. and Ingold, T. (eds.), Tool, Language and Cognition in Human Evolution (pp. 138–150). Cambridge: Cambridge University Press.Google Scholar
  64. Visalberghi, E. and Limongelli, L. (1996). Action and understanding: tool use revisited through the mind of capuchin monkeys. In Russon, A., Bard, K., Parker, S. (eds.), Reaching into thought. The minds of the great apes pp. 57–79. Cambridge: Cambridge University Press.Google Scholar
  65. Visalberghi, E. and Tomasello, M. (1997). Primate causal understanding in the physical and in the social domains. Behavioral Processes, 42, 189–203.CrossRefGoogle Scholar
  66. Wolpert, D.M., Ghahrmani, Z., Jordanm, M.I. (1994). An internal model for integration. Science, 269, 1880–1882.CrossRefGoogle Scholar
  67. Weir, A.A.S., Chappell, J., Kacelnik, A. (2002). Shaping of hooks in New Caledonian Crows. Science, 297, 981–983.CrossRefPubMedGoogle Scholar
  68. Yuille, A., Carter, N., Tenenbaum, J.B. (2006). Probabilistic models of cognition: conceptual foundations, Trends in Cognitive Sciences, 10(7), 287–291.CrossRefPubMedGoogle Scholar
  69. Zak, M. (1988). Terminal attractors for addressable memory in neural networks. Physics Letters. A, 133, 218–222.CrossRefGoogle Scholar
  70. Zatorre, R.J., Chen, J.L., Penhune, V.B. (2007). When the brain plays music. Auditory-motor interactions in music perception and production. Nature Reviews Neuroscience, 8, 547–558.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Vishwanathan Mohan
    • 1
    Email author
  • Pietro Morasso
  • Giorgio Metta
  • Stathis Kasderidis
  1. 1.Cognitive Humanoids Lab, Robotics Brain and Cognitive Sciences DepartmentItalian Institute of TechnologyGenoaItaly

Personalised recommendations