Abstract
Reservoir Computing (RC) offers a computationally efficient and well performing technique for using the temporal processing power of Recurrent Neural Networks (RNNs), while avoiding the traditional long training times and stability problems. The method is both simple and elegant: a random RNN (called the reservoir) is constructed using only a few global parameters to tune the dynamics into a desirable regime, and the dynamic response of the reservoir is used to train a simple linear regression function called the readout function - the reservoir itself remains untrained. This technique has shown some experimentally very convincing results on a variety of tasks, but a thorough understanding of the importance of the dynamics for the performance is still lacking. This contribution aims to extend this understanding, by presenting a more sophisticated extension on the traditional way of characterizing the reservoir dynamics, by using the dynamic profile of the Jacobian of the reservoir instead of static, a priori measures such as the standard spectral radius. We show that this measure gives a more accurate description of the reservoir dynamics, and can serve as predictor for the performance. Additionally, due to the theoretical background from dynamical systems theory, this measure offers some insight into the underlying mechanisms of RC.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Buehner, M., Young, P.: A tighter bound for the echo state property. IEEE Transactions on Neural Networks 17(3), 820–824 (2006)
Csaki, F.: State-space methods for control systems. Akademiai Kiado (1977)
Ganguli, S., Huh, D., Sompolinsky, H.: Memory traces in dynamical systems. Proceedings of the National Academy of Sciences 105(48), 18970 (2008)
Jaeger, H.: Short term memory in echo state networks. Technical Report GMD Report 152, German National Research Center for Information Technology (2001)
Jaeger, H.: Adaptive nonlinear system identification with echo state networks. In: Advances in Neural Information Processing Systems, pp. 593–600 (2003)
Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication. Science 308, 78–80 (2004)
Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation 14(11), 2531–2560 (2002)
Merlet, J.P.: Jacobian, manipulability, condition number, and accuracy of parallel robots. Journal of Mechanical Design 128, 199 (2006)
Ozturk, M.C., Xu, D., Principe, J.C.: Analysis and design of echo state networks. Neural Computation 19, 111–138 (2006)
Penrose, R.: A generalized inverse for matrices, vol. 51(1955), pp. 406–413 (1955)
Shawe-Taylor, J., Cristianini, N.: Kernel methods for pattern analysis. Cambridge University Press, Cambridge (2004)
Siewert, U., Wustlich, W.: Echo-state networks with band-pass neurons: Towards generic time-scale-independent reservoir structures. Preliminary Report (October 2007)
Vandoorne, K., Dierckx, W., Schrauwen, B., Verstraeten, D., Baets, R., Bienstman, P., Campenhout, J.V.: Toward optical signal processing using photonic reservoir computing. Optics Express 16(15), 11182–11192 (2008)
Verstraeten, D., Schrauwen, B., Stroobandt, D.: Reservoir-based techniques for speech recognition. In: Proceedings of the World Conference on Computational Intelligence, pp. 1050–1053 (2006)
Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt., D.: A unifying comparison of reservoir computing methods. Neural Networks 20, 391–403 (2007)
Verstraeten, D., Souza, S.X.-d., Schrauwen, B., Suykens, J., Stroobandt, D., Vandewalle, J.: Pattern classification with CNNs as reservoirs. In: Proceedings of the International Symposium on Nonlinear Theory and its Applications (NOLTA), vol. 9 (2008)
White, O.L., Lee, D.D., Sompolinsky, H.: Short-term memory in orthogonal neural networks. Neural Comput. Phys. Rev. Lett. 92, 148102 (2002)
Wyffels, F., Schrauwen, B., Verstraeten, D., Stroobandt, D.: Band-pass reservoir computing. In: Hou, Z., Zhang, N. (eds.) Proceedings of the International Joint Conference on Neural Networks, Hong Kong, June, pp. 3203–3208 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Verstraeten, D., Schrauwen, B. (2009). On the Quantification of Dynamics in Reservoir Computing. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds) Artificial Neural Networks – ICANN 2009. ICANN 2009. Lecture Notes in Computer Science, vol 5768. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04274-4_101
Download citation
DOI: https://doi.org/10.1007/978-3-642-04274-4_101
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04273-7
Online ISBN: 978-3-642-04274-4
eBook Packages: Computer ScienceComputer Science (R0)