Evolutionary Intelligence

, Volume 7, Issue 3, pp 169–182 | Cite as

An evolutionary cognitive architecture made of a bag of networks

  • Alexander W. ChurchillEmail author
  • Chrisantha Fernando
Special Issue


A cognitive architecture is presented for modelling some properties of sensorimotor learning in infants, namely the ability to accumulate adaptations and skills over multiple tasks in a manner which allows recombination and re-use of task specific competences. The control architecture invented consists of a population of compartments (units of neuroevolution) each containing networks capable of controlling a robot with many degrees of freedom. The nodes of the network undergo internal mutations, and the networks undergo stochastic structural modifications, constrained by a mutational and recombinational grammar. The nodes used consist of dynamical systems such as dynamic movement primitives, continuous time recurrent neural networks and high-level supervised and unsupervised learning algorithms. Edges in the network represent the passing of information from a sending node to a receiving node. The networks in a compartment operate in parallel and encode a space of possible subsumption-like architectures that are used to successfully evolve a variety of behaviours for a NAO H25 humanoid robot.


Cognitive architecture Darwinian neurodynamics Open-ended evolution Robotics 



The work is funded by the FQEB Templeton grant “Bayes, Darwin and Hebb”, and the FP-7 FET OPEN Grant INSIGHT.


  1. 1.
    Baldassarre G, Mirolli M (2013) Intrinsically motivated learning in natural and artificial systems. Springer, New YorkCrossRefGoogle Scholar
  2. 2.
    Baranes A, Oudeyer PY (2013) Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot Auton Syst 61(1):49–73CrossRefGoogle Scholar
  3. 3.
    Bellas F, Duro RJ, Faiña A, Souto D (2010) Multilevel darwinist brain (mdb): artificial evolution in a cognitive architecture for real robots. IEEE Trans Auton Mental Dev 2(4):340–354CrossRefGoogle Scholar
  4. 4.
    Brooks RA (1990) Elephants don’t play chess. Robot Auton Syst 6(1):3–15CrossRefGoogle Scholar
  5. 5.
    Bull L, Kovacs T (2005) Foundations of learning classifier systems, vol 183. Springer, New YorkzbMATHGoogle Scholar
  6. 6.
    Butz MV, Herbort O (2008) Context-dependent predictions and cognitive arm control with xcsf. In: Proceedings of the 10th annual conference on Genetic and evolutionary computation, ACM, New York, pp 1357–1364Google Scholar
  7. 7.
    Calabretta R, Nolfi S, Parisi D, Wagner GP (2000) Duplication of modules facilitates the evolution of functional specialization. Artif Life 6(1):69–84CrossRefGoogle Scholar
  8. 8.
    Chklovskii D, Mel B, Svoboda K (2004) Cortical rewiring and information storage. Nature 431:782–788CrossRefGoogle Scholar
  9. 9.
    Cliff D, Ross S (1994) Adding temporary memory to zcs. Adapt Behav 3(2):101–150CrossRefGoogle Scholar
  10. 10.
    Clune J, Mouret JB, Lipson H (2013) The evolutionary origins of modularity. Proc R Soc B Biol Sci 280(1755):20122,863CrossRefGoogle Scholar
  11. 11.
    Crapse TB, Sommer MA (2008) Corollary discharge across the animalkingdom. Nat Rev Neurosci 9(8):587–600 [Crapse, Trinity B Sommer, Marc A R01-EY017592/EY/NEI NIH HHS/United States Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov’t Review England Nature reviews]. Neuroscience Nat Rev Neurosci. 2008 Aug; 9(8):587–600Google Scholar
  12. 12.
    Dearden A, Demiris Y (2005) Learning forward models for robots. In: IJCAI, vol 5, p 1440Google Scholar
  13. 13.
    Der R, Martius G (2012) The playful machine. Cognitive systems monographs. Springer, New YorkGoogle Scholar
  14. 14.
    Dorigo M (1998) Robot shaping: an experiment in behavior engineering. MIT press, CambridgeGoogle Scholar
  15. 15.
    Durr P, Mattiussi C, Floreano D (2010) Genetic representation and evolvability of modular neural controllers. IEEE Comput Intell Mag 5(3):10–19CrossRefGoogle Scholar
  16. 16.
    Eaton M (2013) An approach to the synthesis of humanoid robot dance using non-interactive evolutionary techniques. In: IEEE international conference on systems, man, and cybernetics (SMC), pp 3305–3309. IEEEGoogle Scholar
  17. 17.
    Eaton M, Davitt TJ (2007) Evolutionary control of bipedal locomotion in a high degree-of-freedom humanoid robot: first steps. Artif Life Robot 11(1):112–115CrossRefGoogle Scholar
  18. 18.
    Farchy A, Barrett S, MacAlpine P, Stone P (2013) Humanoid robots learning to walk faster: from the real world to simulation and back. In: Proceedings of the 2013 international conference on autonomous agents and multi-agent systems, pp 39–46. International foundation for autonomous agents and multiagent systemsGoogle Scholar
  19. 19.
    Fernando C, Zachar I, Szathmry E (2010) Linguistic constructions as neuronal replicators. In: Steels L (ed) Fluid construction grammar. John Bengamin, OxfordGoogle Scholar
  20. 20.
    Fodor J, Pylyshyn Z (1988) Connectionism and cognitive architecture: a critical analysis. Cognition 28:3–71CrossRefGoogle Scholar
  21. 21.
    Gomez F, Miikkulainen R (1997) Incremental evolution of complex general behavior. Adapt Behav 5(3–4):317–342CrossRefGoogle Scholar
  22. 22.
    Gordon G, Ahissar E (2012) A curious emergence of reaching, lecture notes in computer science, vo. 7429, chap 1. Springer, Berlin, pp 1–12Google Scholar
  23. 23.
    Gordon G, Ahissar E (2012) Hierarchical curiosity loops and active sensing. Neural Netw 32:119–129 [Gordon, Goren Ahissar, Ehud Neural Netw. 2012 Aug; 32:119–129. Epub 2012 Feb 14]Google Scholar
  24. 24.
    Haruno M, Wolpert D, Kawato M (2001) Mosaic model for sensorimotor learning and control. Neural Comput 13:2201–2220CrossRefzbMATHGoogle Scholar
  25. 25.
    Harvey I (2011) The microbial genetic algorithm. In: Advances in artificial life. Darwin meets von Neumann. Springer, New York, pp 126–133Google Scholar
  26. 26.
    Harvey I, Husbands P, Cliff D (1994) Seeing the light: artificial evolution, real vision. School of Cognitive and Computing Sciences, University of Sussex, FalmerGoogle Scholar
  27. 27.
    Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12(3):353–380CrossRefGoogle Scholar
  28. 28.
    Husbands P, Harvey I, Cliff D (1993) An evolutionary approach to situated ai. In: Proceedings of the 9th Bi-annual conference of the society for the study of artificial intelligence and the simulation of behaviour (AISB 93), pp 61–70Google Scholar
  29. 29.
    Izhikevich EM (2006) Polychronization: computation with spikes. Neural Comput 18(2):245–282 [Izhikevich, Eugene M. Neural Comput.18(2), 245–82 (2006)]Google Scholar
  30. 30.
    Jordan MI, Rumelhart DE (1992) Forward models: supervised learning with a distal teacher. Cogn Sci 16(3):307–354CrossRefGoogle Scholar
  31. 31.
    Koos S, Cully A, Mouret JB (2013) Fast damage recovery in robotics with the t-resilience algorithm. Int J Robot Res 32(14):1700–1723. doi: 10.1177/0278364913499192 CrossRefGoogle Scholar
  32. 32.
    Koza JR (1992) Evolution of subsumption using genetic programming. In: Proceedings of the 1st European conference on artificial life, pp 110–119Google Scholar
  33. 33.
    Koza JR (1999) Genetic programming III: darwinian invention and problem solving. Morgan Kaufmann, San Francisco. 99010099 [John R. Koza et al. ill.; 25 cm. Includes bibliographical references (p. [1081]–1114)Google Scholar
  34. 34.
    Lalazar H, Vaadia E (2008) Neural basis of sensorimotor learning: modifying internal models. Curr Opin Neurobiol 18((6)):573–581CrossRefGoogle Scholar
  35. 35.
    Lehman J, Stanley KO (2011) Abandoning objectives: evolution through the search for novelty alone. Evol Comput 19(2):189–223CrossRefGoogle Scholar
  36. 36.
    Lewis FL, Vrabie D, Syrmos VL (2012) Optimal control, 3rd edn. Wiley, HobokenGoogle Scholar
  37. 37.
    Li H, Liao X, Carin L (2009) Multi-task reinforcement learning in partially observable stochastic environments. J Mach Learn Res 10(1131–1186):1577109MathSciNetGoogle Scholar
  38. 38.
    Mckay RI, Hoai NX, Whigham PA, Shan Y, ONeill M (2010) Grammar-based genetic programming: a survey. Genet Program Evol Mach 11(3–4):365–396CrossRefGoogle Scholar
  39. 39.
    Miller JF, Khan GM (2011) Where is the brain inside the brain? Memet Comput 3(3):217–228CrossRefGoogle Scholar
  40. 40.
    Miller JF, Thomson P (2000) Cartesian genetic programming. In: Genetic programming. Springer, New York, pp 121–132Google Scholar
  41. 41.
    Mohamed Z, Kitani M, Kaneko Si, Capi G (2014) Humanoid robot arm performance optimization using multi objective evolutionary algorithm. Int J Control Autom Syst 12(4):870–877CrossRefGoogle Scholar
  42. 42.
    Moriguchi H, Lipson H (2011) Learning symbolic forward models for robotic motion planning and control. In: Proceedings of European conference of artificial life (ECAL 2011), pp 558–564Google Scholar
  43. 43.
    Mouret JB, Doncieux S (2009) Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity. In: IEEE congress on evolutionary computation, 2009. CEC’09, pp 1161–1168. IEEEGoogle Scholar
  44. 44.
    Oudeyer PY, Kaplan F, Hafner VV (2007) Intrinsic motivation systems for autonomous mental development. IEEE Trans Evol Comput 11(2):265–286CrossRefGoogle Scholar
  45. 45.
    Pape L, Oddo CM, Controzzi M, Cipriani C, Frster A, Carrozza MC, Schmidhuber J(2012) Learning tactile skills through curious exploration. Frontiers in neurorobotics 6 Google Scholar
  46. 46.
    Poli R (1996) Parallel distributed genetic programming. CiteseerGoogle Scholar
  47. 47.
    Rozenberg G (1997) Handbook of graph grammars and computing by graph transformation. World Scientific, Singapore. 96037597 edited by Grzegorz Rozenberg. ill.; 23 cm. Includes bibliographical references and indexes. v. 1. FoundationsGoogle Scholar
  48. 48.
    Savastano P, Nolfi S (2013) A robotic model of reaching and grasping developmentGoogle Scholar
  49. 49.
    Schaal S (2006) Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In: Adaptive motion of animals and machines. Springer, New York, pp 261–280Google Scholar
  50. 50.
    Schaal S, Peters J, Nakanishi J, Ijspeert A (2005) Learning movement primitives. In: Robotics research. Springer, New York, pp 561–572Google Scholar
  51. 51.
    Schmidhuber J (2009) Driven by compression progress: a simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In: Anticipatory behavior in adaptive learning systems. Springer, New York, pp 48–76Google Scholar
  52. 52.
    Schrum J, Miikkulainen R (2014) Evolving multimodal behavior with modular neural networks in ms. pac-man. In: Proceedings of the genetic and evolutionary computation conference (GECCO 2014). Vancouver, BC, Canada, pp 325–332. Best Paper: Digital Entertainment and Arts
  53. 53.
    Shadmehr R (2004) Generalization as a behavioral window to the neural mechanisms of learning internal models. Hum Mov Sci 23:543–568CrossRefGoogle Scholar
  54. 54.
    Sporns O, Edelman GM (1993) Solving bernstein’s problem: a proposal for the development of coordinated movement by selection. Child Dev 64:960–981CrossRefGoogle Scholar
  55. 55.
    Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212CrossRefGoogle Scholar
  56. 56.
    Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127CrossRefGoogle Scholar
  57. 57.
    Steels L, De Beule J (2006) Unify and merge in fluid construction grammarGoogle Scholar
  58. 58.
    Stolle M, Precup D (2002) Learning options in reinforcement learning, lecture notes in computer science, vol 2371, chap 16. Springer, Berlin, pp 212–223Google Scholar
  59. 59.
    Studley M, Bull L (2005) X-tcs: accuracy-based learning classifier system robotics. In: The 2005 IEEE congress on evolutionary computation, vol 3, pp 2099–2106. IEEEGoogle Scholar
  60. 60.
    Sturm J, Plagemann C, Burgard W (2008) Unsupervised body scheme learning through self-perception. In: IEEE international conference on robotics and automation, 2008. ICRA 2008, pp 3328–3333. IEEEGoogle Scholar
  61. 61.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Adaptive computation and machine learning. MIT Press, Cambridge, MA. [97026416 Richard S. Sutton and Andrew G. Barto. ill.; 24 cm. Includes bibliographical references (p. [291]–312) and index]Google Scholar
  62. 62.
    Teller A, Veloso M (1995) Program evolution for data mining. Int J Exp Syst Res Appl 8:213–236CrossRefGoogle Scholar
  63. 63.
    Teller A, Veloso M (1996) Neural programming and an internal reinforcement policy. In: Late breaking papers at the genetic programming 1996 conference, pp 186–192. Citeseer, New JerseyGoogle Scholar
  64. 64.
    Thelen E (1995) Motor development: a new synthesis. Am Psychol 50(2):79CrossRefGoogle Scholar
  65. 65.
    Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5(11):1226–1235. doi: 10.1038/nn963 CrossRefGoogle Scholar
  66. 66.
    Togelius J (2004) Evolution of a subsumption architecture neurocontroller. J Intell Fuzzy Syst 15(1):15–20Google Scholar
  67. 67.
    Urzelai J, Floreano D, Dorigo M, Colombetti M (1998) Incremental robot shaping. Connect Sci 10(3–4):341–360CrossRefGoogle Scholar
  68. 68.
    Wilson SW (2000) Get real! xcs with continuous-valued inputs. In: Learning classifier systems. Springer, New York, pp 209–219Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Electronic Engineering and Computer ScienceQueen Mary, University of LondonLondonUK

Personalised recommendations