Advertisement

Analysis of the Dynamics of the Echo State Network Model Using Recurrence Plot

  • Emmanuel Sam
  • Sebastian Basterrech
  • Pavel Kromer
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 874)

Abstract

At the beginning of the 2000s, a specific type of Recurrent Neural Networks (RNNs) was developed with the name Echo State Network (ESN). The model has become popular during the last 15 years in the area of temporal learning. The model has a RNN (named reservoir) that projects an input sequence in a feature map. The reservoir has two main parameters that impact the accuracy of the model: the reservoir size (number of neurons in the RNN) and the spectral radius of the hidden-hidden recurrent weight matrix. In this article, we analyze the impact of these parameters using the Recurrence Plot technique, which is a useful tool for visualizing chaotic systems. Experiments carried out with three well-known dynamical systems show the relevance of the spectral radius in the reservoir projections.

Keywords

Recurrent Neural Network Echo State Network Recurrence Plot Chaotic systems Time-series problems 

Notes

Acknowledgment

This work was supported by the Czech Science Foundation under the grant no. GJ16-25694Y, and by the projects SP2018/126 and SP2018/130 of the Student Grant System, VSB-Technical University of Ostrava, and it has been supported by the Czech Science Foundation (GAČR) under research project No. 18-18858S.

References

  1. 1.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  2. 2.
    Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. German National Research Center for Information Technology, Technical report, 148 (2001)Google Scholar
  3. 3.
    Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: a new framework for a neural computation based on perturbations. Neural Comput. 14, 2531–2560 (2002)CrossRefGoogle Scholar
  4. 4.
    Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 287–289 (2007)CrossRefGoogle Scholar
  5. 5.
    Lukos̆evic̆ius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Revi. 3, 127–149 (2009)CrossRefGoogle Scholar
  6. 6.
    Basterrech, S., Rubino, G.: Echo state queueing networks: a combination of reservoir computing and random neural networks. Probab. Eng. Inf. Sci. 31, 457–476 (2017).  https://doi.org/10.1017/S0269964817000110MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Basterrech, S.: Empirical analysis of the necessary and sufficient conditions of the echo state property. In: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, 14-19 May 2017, pp. 888–896 (2017)Google Scholar
  8. 8.
    Yildiza, I.B., Jaeger, H., Kiebela, S.J.: Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012)CrossRefGoogle Scholar
  9. 9.
    Manjunath, G., Jaeger, H.: Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Lukoševičius, M.: A Practical Guide to Applying Echo State Networks. In: Montavon, G., Orr, G., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol. 7700, pp. 659–686. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-35289-8_36CrossRefGoogle Scholar
  11. 11.
    Marwan, N., Romano, M.C., Thiel, M., Kurths, J.: Recurrence plots for the analysis of complex systems. Phys. Reports 438, 237–329 (2007)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Bianchi, F.M., Livi, L., Alippi, C.: Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Trans. Neural Netw. Learn. Syst. 29(2), 427–439 (2016)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)CrossRefGoogle Scholar
  14. 14.
    Schrauwen, B., Wardermann, M., Verstraeten, D., Steil, J.J., Stroobandt, D.: Improving reservoirs using intrinsic plasticity. Neurocomputing 71, 1159–1171 (2007)CrossRefGoogle Scholar
  15. 15.
    Steil, J.J.: Backpropagation-Decorrelation: online recurrent learning with O(N) complexity. In: Proceedings of IJCNN 04, vol. 1 (2004)Google Scholar
  16. 16.
    Xue, Y., Yang, L., Haykin, S.: Decoupled echo state networks with lateral inhibition. Neural Netw. 20(3), 365–376 (2007)CrossRefGoogle Scholar
  17. 17.
    Jaeger, H., Lukos̆evic̆ius, M., Popovici, D., Siewert, U.: Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw. 20(3), 335–352 (2007)CrossRefGoogle Scholar
  18. 18.
    Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Netw. 19, 757–779 (2007)zbMATHGoogle Scholar
  19. 19.
    Basterrech, S., Rubino, G.: Echo state queueing network: a new reservoir computing learning tool. In: 10th IEEE Consumer Communications and Networking Conference, CCNC 2013, Las Vegas, NV, USA, 11-14 January 2013, pp. 118–123 (2013). http://dx.doi.org/10.1109/CCNC.2013.6488435
  20. 20.
    Rodan, A., Tin̆o, P.: Minimum complexity echo state network. IEEE Trans. Neural Netw. 22, 131–144 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Emmanuel Sam
    • 1
  • Sebastian Basterrech
    • 2
  • Pavel Kromer
    • 3
  1. 1.Nduom School of Business and TechnologyElminaGhana
  2. 2.Department of Computer Science, Faculty of Electrical EngineeringCzech Technical UniversityPragueCzech Republic
  3. 3.Faculty of Electrical Engineering and Computer ScienceVŠB-Technical University of OstravaOstravaCzech Republic

Personalised recommendations