Abstract
At the beginning of the 2000s, a specific type of Recurrent Neural Networks (RNNs) was developed with the name Echo State Network (ESN). The model has become popular during the last 15 years in the area of temporal learning. The model has a RNN (named reservoir) that projects an input sequence in a feature map. The reservoir has two main parameters that impact the accuracy of the model: the reservoir size (number of neurons in the RNN) and the spectral radius of the hidden-hidden recurrent weight matrix. In this article, we analyze the impact of these parameters using the Recurrence Plot technique, which is a useful tool for visualizing chaotic systems. Experiments carried out with three well-known dynamical systems show the relevance of the spectral radius in the reservoir projections.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. German National Research Center for Information Technology, Technical report, 148 (2001)
Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: a new framework for a neural computation based on perturbations. Neural Comput. 14, 2531–2560 (2002)
Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 287–289 (2007)
Lukos̆evic̆ius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Revi. 3, 127–149 (2009)
Basterrech, S., Rubino, G.: Echo state queueing networks: a combination of reservoir computing and random neural networks. Probab. Eng. Inf. Sci. 31, 457–476 (2017). https://doi.org/10.1017/S0269964817000110
Basterrech, S.: Empirical analysis of the necessary and sufficient conditions of the echo state property. In: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, 14-19 May 2017, pp. 888–896 (2017)
Yildiza, I.B., Jaeger, H., Kiebela, S.J.: Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012)
Manjunath, G., Jaeger, H.: Echo state property linked to an input: exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013)
Lukoševičius, M.: A Practical Guide to Applying Echo State Networks. In: Montavon, G., Orr, G., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science, vol. 7700, pp. 659–686. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_36
Marwan, N., Romano, M.C., Thiel, M., Kurths, J.: Recurrence plots for the analysis of complex systems. Phys. Reports 438, 237–329 (2007)
Bianchi, F.M., Livi, L., Alippi, C.: Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Trans. Neural Netw. Learn. Syst. 29(2), 427–439 (2016)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)
Schrauwen, B., Wardermann, M., Verstraeten, D., Steil, J.J., Stroobandt, D.: Improving reservoirs using intrinsic plasticity. Neurocomputing 71, 1159–1171 (2007)
Steil, J.J.: Backpropagation-Decorrelation: online recurrent learning with O(N) complexity. In: Proceedings of IJCNN 04, vol. 1 (2004)
Xue, Y., Yang, L., Haykin, S.: Decoupled echo state networks with lateral inhibition. Neural Netw. 20(3), 365–376 (2007)
Jaeger, H., Lukos̆evic̆ius, M., Popovici, D., Siewert, U.: Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw. 20(3), 335–352 (2007)
Schmidhuber, J., Wierstra, D., Gagliolo, M., Gomez, F.: Training recurrent networks by evolino. Neural Netw. 19, 757–779 (2007)
Basterrech, S., Rubino, G.: Echo state queueing network: a new reservoir computing learning tool. In: 10th IEEE Consumer Communications and Networking Conference, CCNC 2013, Las Vegas, NV, USA, 11-14 January 2013, pp. 118–123 (2013). http://dx.doi.org/10.1109/CCNC.2013.6488435
Rodan, A., Tin̆o, P.: Minimum complexity echo state network. IEEE Trans. Neural Netw. 22, 131–144 (2011)
Acknowledgment
This work was supported by the Czech Science Foundation under the grant no. GJ16-25694Y, and by the projects SP2018/126 and SP2018/130 of the Student Grant System, VSB-Technical University of Ostrava, and it has been supported by the Czech Science Foundation (GAČR) under research project No. 18-18858S.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Sam, E., Basterrech, S., Kromer, P. (2019). Analysis of the Dynamics of the Echo State Network Model Using Recurrence Plot. In: Abraham, A., Kovalev, S., Tarassov, V., Snasel, V., Sukhanov, A. (eds) Proceedings of the Third International Scientific Conference “Intelligent Information Technologies for Industry” (IITI’18). IITI'18 2018. Advances in Intelligent Systems and Computing, vol 874. Springer, Cham. https://doi.org/10.1007/978-3-030-01818-4_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-01818-4_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01817-7
Online ISBN: 978-3-030-01818-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)