Abstract
Training a neural network (NN) through reinforcement learning (RL) has been focused on recently, and a recurrent NN (RNN) is used in learning tasks that require memory. Meanwhile, to cover the shortcomings in learning an RNN, the reservoir network (RN) has been often employed mainly in supervised learning. The RN is a special RNN and has attracted much attention owing to its rich dynamic representations. An approach involving the use of a multi-layer readout (MLR), which comprises a multi-layer NN, was studied for acquiring complex representations using the RN. This study demonstrates that an RN with MLR can learn a “memory task” through RL with back propagation. In addition, non-linear representations required to clear the task are not observed in the RN but are constructed by learning in the MLR. The results suggest that the MLR can make up for the limited computational ability in an RN.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Shibata, K.: Functions that emerge through end-to-end reinforcement learning (2017). arXiv:1703.02239
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with deep reinforcement learning (2013). arXiv:1312.5602
Shibata, K., Utsunomiya, H.: Discovery of pattern meaning from delayed rewards by reinforcement learning with a recurrent neural network. In: Proceedings of IJCNN, pp. 1445–1452 (2011)
Shibata, K., Goto, K.: Emergence of flexible prediction-based discrete decision making and continuous motion generation through Actor-Q-Learning. In: Proceedings of ICDL-Epirob, ID 15 (2013)
Jaeger, H.: The “echo state approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report 148.34, p. 13 (2001)
Maass, W., Natschlger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11), 2531–2560 (2002)
Salmen, M., Ploger, P.G.: Echo state networks used for motor control. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, ICRA 2005. IEEE (2005)
Sussillo, D., Abbott, L.F.: Generating coherent patterns of activity from chaotic neural networks. Neuron Article 63(4), 544–557(2009)
Hoerzer, G.M., Legenstein, R., Maass, W.: Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cerebral Cortex 24(3), 677–690 (2014)
Lukusevic̆ius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)
Bush, K., Anderson, C.: Modeling reward functions for incomplete state representations via echo state networks. In: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. IJCNN ’05, vol. 5. IEEE (2005)
Babinec, S̆., Pospchal, J.: Merging echo state and feedforward neural networks for time series forecasting. In: Artificial Neural Networks ICANN 2006, pp. 367–375 (2006)
Goto, Y., Shibata, K.: Emergence of higher exploration in reinforcement learning using a chaotic neural network. In: Proceedings of International Conference on Neural Information Processing (ICONIP) 2016, LNCS 9947, pp. 40–48 (2016)
Matsuki, T., Shibata, K.: Reward-based learning of a memory-required task based on the internal dynamics of a chaotic neural network. In: Proceedings of International Conference on Neural Information Processing (ICONIP) 2016, LNCS 9947, pp. 376–383 (2016)
Acknowledgements
This work was supported by JSPS KAKENHI Grant Number 15K00360.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Matsuki, T., Shibata, K. (2019). Reinforcement Learning of a Memory Task Using an Echo State Network with Multi-layer Readout. In: Kim, JH., et al. Robot Intelligence Technology and Applications 5. RiTA 2017. Advances in Intelligent Systems and Computing, vol 751. Springer, Cham. https://doi.org/10.1007/978-3-319-78452-6_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-78452-6_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78451-9
Online ISBN: 978-3-319-78452-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)