Supervised and Evolutionary Learning of Echo State Networks

  • Fei Jiang
  • Hugues Berry
  • Marc Schoenauer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5199)


A possible alternative to topology fine-tuning for Neural Network (NN) optimization is to use Echo State Networks (ESNs), recurrent NNs built upon a large reservoir of sparsely randomly connected neurons. The promises of ESNs have been fulfilled for supervised learning tasks, but unsupervised ones, e.g. control problems, require more flexible optimization methods – such as Evolutionary Algorithms. This paper proposes to apply CMA-ES, the state-of-the-art method in evolutionary continuous parameter optimization, to the evolutionary learning of ESN parameters. First, a standard supervised learning problem is used to validate the approach and compare it to the standard one. But the flexibility of Evolutionary optimization allows us to optimize not only the outgoing weights but also, or alternatively, other ESN parameters, sometimes leading to improved results. The classical double pole balancing control problem is then used to demonstrate the feasibility of evolutionary (i.e. reinforcement) learning of ESNs. We show that the evolutionary ESN obtain results that are comparable with those of the best topology-learning methods.


Neural networks Evolutionary algorithms Control 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Auger, A., Hansen, N.: Performance evaluation of an advanced local search evolutionary algorithm. In: Proc. CEC 2005 (2005)Google Scholar
  2. 2.
    Banzhaf, W.: Artificial Regulatory Networks and Genetic Programming. In: Riolo, R., Worzel, B. (eds.) Genetic Programming Theory and Practice, ch. 4, pp. 43–62. Kluwer Academic Publishers, Dordrecht (2003)CrossRefGoogle Scholar
  3. 3.
    Devert, A., Bredeche, N., Schoenauer, M.: Robust multi-cellular developmental design. In: GECCO 2007: Proceedings of the 9th annual conference on Genetic and evolutionary computation, pp. 982–989. ACM Press, New York (2007)Google Scholar
  4. 4.
    Dürr, P., Mattiussi, C., Floreano, D.: Neuroevolution with Analog Genetic Encoding. In: Runarsson, T.P., Beyer, H.-G., Burke, E.K., Merelo-Guervós, J.J., Whitley, L.D., Yao, X. (eds.) PPSN 2006. LNCS, vol. 4193, pp. 671–680. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  5. 5.
    Gomez, F.J., Miikkulainen, R.: Solving non-markovian control tasks with neuro-evolution. In: IJCAI, pp. 1356–1361 (1999)Google Scholar
  6. 6.
    Gruau, F., Whitley, D., Pyeatt, L.: A comparison between cellular encoding and direct encoding for genetic neural networks. In: Koza, J.R., et al. (eds.) Proc. GP 1996, pp. 28–31. MIT Press, Cambridge (1996)Google Scholar
  7. 7.
    Hansen, N., Kern, S.: Evaluating the CMA evolution strategy on multimodal test functions. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 282–291. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Hansen, N., Ostermeier, A.: Adapting arbitrary normal mutation distributions in evolutionstrategies: the covariance matrix adaptation. In: Proc. CEC 1996, pp. 312–317. IEEE Press, Los Alamitos (1996)Google Scholar
  9. 9.
    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 9(2), 159–195 (2001)CrossRefGoogle Scholar
  10. 10.
    Igel, C.: Neuroevolution for reinforcement learning using evolution strategies. In: Proc. CEC 2003, pp. 2588–2595. IEEE Press, Los Alamitos (2003)Google Scholar
  11. 11.
    Ishu, K., van der Zant, T., Becanovic, V., Ploger, P.: Identification of motion with echo state network. In: Proc. OCEANS 2004. MTTS/IEEE TECHNO-OCEAN 2004, vol. 3, pp. 1205–1210 (2004)Google Scholar
  12. 12.
    Jaeger, H.: The Echo State Approach to Analysing and Training Recurrent Neural Networks. Technical Report GMD Report 148, German National Research Center for Information Technology (2001)Google Scholar
  13. 13.
    Jaeger, H.: Tutorial on training recurrent neural networks. Technical report, GMD Report 159, Fraunhofer Institute AIS (2002)Google Scholar
  14. 14.
    Jaeger, H., Haas, H., Principe, J.C. (eds.): NIPS 2006 Workshop on Echo State Networks and Liquid State Machines (2006)Google Scholar
  15. 15.
    Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Comput. 19(3), 757–779 (2007)CrossRefzbMATHGoogle Scholar
  16. 16.
    Stanley, K.: Compositional Pattern Producing Networks: A Novel Abstraction of Development. Genetic Programming and Evolvable Machines 8(2), 131–162 (2007)CrossRefGoogle Scholar
  17. 17.
    Stanley, K.O., Miikkulainen, R.: Efficient reinforcement learning through evolving neural network topologies. In: Langdon, W.B., et al. (eds.) Proc. GECCO 2002, pp. 569–577. Morgan Kaufmann, San Francisco (2002)Google Scholar
  18. 18.
    Xu, D., Lan, J., Principe, J.: Direct adaptive control: an echo state network and genetic algorithm approach. In: Proc. IEEE International Joint Conference on Neural Networks IJCNN 2005, 31 July–4 August 2005, vol. 3, pp. 1483–1486 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Fei Jiang
    • 1
    • 2
  • Hugues Berry
    • 1
  • Marc Schoenauer
    • 2
  1. 1.Alchemy, INRIA SaclayOrsay CedexFrance
  2. 2.TAO, INRIA Saclay & LRI (UMR CNRS 8623), Bât 490Université Paris-SudOrsay CedexFrance

Personalised recommendations