Abstract
A cognitive architecture is presented for modelling some properties of sensorimotor learning in infants, namely the ability to accumulate adaptations and skills over multiple tasks in a manner which allows recombination and re-use of task specific competences. The control architecture invented consists of a population of compartments (units of neuroevolution) each containing networks capable of controlling a robot with many degrees of freedom. The nodes of the network undergo internal mutations, and the networks undergo stochastic structural modifications, constrained by a mutational and recombinational grammar. The nodes used consist of dynamical systems such as dynamic movement primitives, continuous time recurrent neural networks and high-level supervised and unsupervised learning algorithms. Edges in the network represent the passing of information from a sending node to a receiving node. The networks in a compartment operate in parallel and encode a space of possible subsumption-like architectures that are used to successfully evolve a variety of behaviours for a NAO H25 humanoid robot.
Similar content being viewed by others
Notes
Supplementary material is available at http://www.alexchurchill.com/papers/evin2014.
References
Baldassarre G, Mirolli M (2013) Intrinsically motivated learning in natural and artificial systems. Springer, New York
Baranes A, Oudeyer PY (2013) Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot Auton Syst 61(1):49–73
Bellas F, Duro RJ, Faiña A, Souto D (2010) Multilevel darwinist brain (mdb): artificial evolution in a cognitive architecture for real robots. IEEE Trans Auton Mental Dev 2(4):340–354
Brooks RA (1990) Elephants don’t play chess. Robot Auton Syst 6(1):3–15
Bull L, Kovacs T (2005) Foundations of learning classifier systems, vol 183. Springer, New York
Butz MV, Herbort O (2008) Context-dependent predictions and cognitive arm control with xcsf. In: Proceedings of the 10th annual conference on Genetic and evolutionary computation, ACM, New York, pp 1357–1364
Calabretta R, Nolfi S, Parisi D, Wagner GP (2000) Duplication of modules facilitates the evolution of functional specialization. Artif Life 6(1):69–84
Chklovskii D, Mel B, Svoboda K (2004) Cortical rewiring and information storage. Nature 431:782–788
Cliff D, Ross S (1994) Adding temporary memory to zcs. Adapt Behav 3(2):101–150
Clune J, Mouret JB, Lipson H (2013) The evolutionary origins of modularity. Proc R Soc B Biol Sci 280(1755):20122,863
Crapse TB, Sommer MA (2008) Corollary discharge across the animalkingdom. Nat Rev Neurosci 9(8):587–600 [Crapse, Trinity B Sommer, Marc A R01-EY017592/EY/NEI NIH HHS/United States Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov’t Review England Nature reviews]. Neuroscience Nat Rev Neurosci. 2008 Aug; 9(8):587–600
Dearden A, Demiris Y (2005) Learning forward models for robots. In: IJCAI, vol 5, p 1440
Der R, Martius G (2012) The playful machine. Cognitive systems monographs. Springer, New York
Dorigo M (1998) Robot shaping: an experiment in behavior engineering. MIT press, Cambridge
Durr P, Mattiussi C, Floreano D (2010) Genetic representation and evolvability of modular neural controllers. IEEE Comput Intell Mag 5(3):10–19
Eaton M (2013) An approach to the synthesis of humanoid robot dance using non-interactive evolutionary techniques. In: IEEE international conference on systems, man, and cybernetics (SMC), pp 3305–3309. IEEE
Eaton M, Davitt TJ (2007) Evolutionary control of bipedal locomotion in a high degree-of-freedom humanoid robot: first steps. Artif Life Robot 11(1):112–115
Farchy A, Barrett S, MacAlpine P, Stone P (2013) Humanoid robots learning to walk faster: from the real world to simulation and back. In: Proceedings of the 2013 international conference on autonomous agents and multi-agent systems, pp 39–46. International foundation for autonomous agents and multiagent systems
Fernando C, Zachar I, Szathmry E (2010) Linguistic constructions as neuronal replicators. In: Steels L (ed) Fluid construction grammar. John Bengamin, Oxford
Fodor J, Pylyshyn Z (1988) Connectionism and cognitive architecture: a critical analysis. Cognition 28:3–71
Gomez F, Miikkulainen R (1997) Incremental evolution of complex general behavior. Adapt Behav 5(3–4):317–342
Gordon G, Ahissar E (2012) A curious emergence of reaching, lecture notes in computer science, vo. 7429, chap 1. Springer, Berlin, pp 1–12
Gordon G, Ahissar E (2012) Hierarchical curiosity loops and active sensing. Neural Netw 32:119–129 [Gordon, Goren Ahissar, Ehud Neural Netw. 2012 Aug; 32:119–129. Epub 2012 Feb 14]
Haruno M, Wolpert D, Kawato M (2001) Mosaic model for sensorimotor learning and control. Neural Comput 13:2201–2220
Harvey I (2011) The microbial genetic algorithm. In: Advances in artificial life. Darwin meets von Neumann. Springer, New York, pp 126–133
Harvey I, Husbands P, Cliff D (1994) Seeing the light: artificial evolution, real vision. School of Cognitive and Computing Sciences, University of Sussex, Falmer
Hurst J, Bull L (2006) A neural learning classifier system with self-adaptive constructivism for mobile robot control. Artif Life 12(3):353–380
Husbands P, Harvey I, Cliff D (1993) An evolutionary approach to situated ai. In: Proceedings of the 9th Bi-annual conference of the society for the study of artificial intelligence and the simulation of behaviour (AISB 93), pp 61–70
Izhikevich EM (2006) Polychronization: computation with spikes. Neural Comput 18(2):245–282 [Izhikevich, Eugene M. Neural Comput.18(2), 245–82 (2006)]
Jordan MI, Rumelhart DE (1992) Forward models: supervised learning with a distal teacher. Cogn Sci 16(3):307–354
Koos S, Cully A, Mouret JB (2013) Fast damage recovery in robotics with the t-resilience algorithm. Int J Robot Res 32(14):1700–1723. doi:10.1177/0278364913499192
Koza JR (1992) Evolution of subsumption using genetic programming. In: Proceedings of the 1st European conference on artificial life, pp 110–119
Koza JR (1999) Genetic programming III: darwinian invention and problem solving. Morgan Kaufmann, San Francisco. 99010099 [John R. Koza et al. ill.; 25 cm. Includes bibliographical references (p. [1081]–1114)
Lalazar H, Vaadia E (2008) Neural basis of sensorimotor learning: modifying internal models. Curr Opin Neurobiol 18((6)):573–581
Lehman J, Stanley KO (2011) Abandoning objectives: evolution through the search for novelty alone. Evol Comput 19(2):189–223
Lewis FL, Vrabie D, Syrmos VL (2012) Optimal control, 3rd edn. Wiley, Hoboken
Li H, Liao X, Carin L (2009) Multi-task reinforcement learning in partially observable stochastic environments. J Mach Learn Res 10(1131–1186):1577109
Mckay RI, Hoai NX, Whigham PA, Shan Y, ONeill M (2010) Grammar-based genetic programming: a survey. Genet Program Evol Mach 11(3–4):365–396
Miller JF, Khan GM (2011) Where is the brain inside the brain? Memet Comput 3(3):217–228
Miller JF, Thomson P (2000) Cartesian genetic programming. In: Genetic programming. Springer, New York, pp 121–132
Mohamed Z, Kitani M, Kaneko Si, Capi G (2014) Humanoid robot arm performance optimization using multi objective evolutionary algorithm. Int J Control Autom Syst 12(4):870–877
Moriguchi H, Lipson H (2011) Learning symbolic forward models for robotic motion planning and control. In: Proceedings of European conference of artificial life (ECAL 2011), pp 558–564
Mouret JB, Doncieux S (2009) Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity. In: IEEE congress on evolutionary computation, 2009. CEC’09, pp 1161–1168. IEEE
Oudeyer PY, Kaplan F, Hafner VV (2007) Intrinsic motivation systems for autonomous mental development. IEEE Trans Evol Comput 11(2):265–286
Pape L, Oddo CM, Controzzi M, Cipriani C, Frster A, Carrozza MC, Schmidhuber J(2012) Learning tactile skills through curious exploration. Frontiers in neurorobotics 6
Poli R (1996) Parallel distributed genetic programming. Citeseer
Rozenberg G (1997) Handbook of graph grammars and computing by graph transformation. World Scientific, Singapore. 96037597 edited by Grzegorz Rozenberg. ill.; 23 cm. Includes bibliographical references and indexes. v. 1. Foundations
Savastano P, Nolfi S (2013) A robotic model of reaching and grasping development
Schaal S (2006) Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In: Adaptive motion of animals and machines. Springer, New York, pp 261–280
Schaal S, Peters J, Nakanishi J, Ijspeert A (2005) Learning movement primitives. In: Robotics research. Springer, New York, pp 561–572
Schmidhuber J (2009) Driven by compression progress: a simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In: Anticipatory behavior in adaptive learning systems. Springer, New York, pp 48–76
Schrum J, Miikkulainen R (2014) Evolving multimodal behavior with modular neural networks in ms. pac-man. In: Proceedings of the genetic and evolutionary computation conference (GECCO 2014). Vancouver, BC, Canada, pp 325–332. http://nn.cs.utexas.edu/?schrum:gecco2014. Best Paper: Digital Entertainment and Arts
Shadmehr R (2004) Generalization as a behavioral window to the neural mechanisms of learning internal models. Hum Mov Sci 23:543–568
Sporns O, Edelman GM (1993) Solving bernstein’s problem: a proposal for the development of coordinated movement by selection. Child Dev 64:960–981
Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212
Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127
Steels L, De Beule J (2006) Unify and merge in fluid construction grammar
Stolle M, Precup D (2002) Learning options in reinforcement learning, lecture notes in computer science, vol 2371, chap 16. Springer, Berlin, pp 212–223
Studley M, Bull L (2005) X-tcs: accuracy-based learning classifier system robotics. In: The 2005 IEEE congress on evolutionary computation, vol 3, pp 2099–2106. IEEE
Sturm J, Plagemann C, Burgard W (2008) Unsupervised body scheme learning through self-perception. In: IEEE international conference on robotics and automation, 2008. ICRA 2008, pp 3328–3333. IEEE
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Adaptive computation and machine learning. MIT Press, Cambridge, MA. [97026416 Richard S. Sutton and Andrew G. Barto. ill.; 24 cm. Includes bibliographical references (p. [291]–312) and index]
Teller A, Veloso M (1995) Program evolution for data mining. Int J Exp Syst Res Appl 8:213–236
Teller A, Veloso M (1996) Neural programming and an internal reinforcement policy. In: Late breaking papers at the genetic programming 1996 conference, pp 186–192. Citeseer, New Jersey
Thelen E (1995) Motor development: a new synthesis. Am Psychol 50(2):79
Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5(11):1226–1235. doi:10.1038/nn963
Togelius J (2004) Evolution of a subsumption architecture neurocontroller. J Intell Fuzzy Syst 15(1):15–20
Urzelai J, Floreano D, Dorigo M, Colombetti M (1998) Incremental robot shaping. Connect Sci 10(3–4):341–360
Wilson SW (2000) Get real! xcs with continuous-valued inputs. In: Learning classifier systems. Springer, New York, pp 209–219
Acknowledgments
The work is funded by the FQEB Templeton grant “Bayes, Darwin and Hebb”, and the FP-7 FET OPEN Grant INSIGHT.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Churchill, A.W., Fernando, C. An evolutionary cognitive architecture made of a bag of networks. Evol. Intel. 7, 169–182 (2014). https://doi.org/10.1007/s12065-014-0121-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12065-014-0121-7