Abstract
We focused on the autonomous control of a three-dimensional snake-like robot that moves on rubble. To realize an autonomous controller, we employed reinforcement learning. However, applying reinforcement learning in a conventional framework to a robot with many degrees of freedom and moving in a complex environment is difficult. There are three problems: state explosion, lack of reproducibility, and lack of generality. To solve these problems, we previously proposed abstracting the state-action space by utilizing the universal properties of the body and environment. The effectiveness of the proposed framework was demonstrated experimentally. Unfortunately, analysis of the obtained policy was lacking. In the present study, we analyzed the obtained policy (i.e., Q-values of Q-learning) to determine the mechanism for abstraction of the state-action space and confirmed that the three problems were solved.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
R.S. Sutton, Reinforcement Learning: An Introduction (The MIT Press, Massachusetts, 1988)
L.P. Kealbling, M.L. Littman, Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)
C.J.H. Watkins, P. Dayan, Technical note Q-learning. Mach. Learn. 8, 279–292 (1992)
M. Asada, Y. Katoh, M. Ogino, K. Hosoda, A humanoid approaches to the goal: reinforcement learning based on rhythmic walking parameters, in Proceedings of the International Symposium of RoboCup2003 (2003, CD-ROM)
H. Kimura, T. Yamashita, S. Kobayashi, Reinforcement learning of walking behavior for a four-legged robot, in Proceedings of 40th IEEE Conference on Decision and Control (2001), pp. 411–416
K. Doya, H. Kimura, M. Kawato, Neural mechanism of learning and control. IEEE Control Syst. Mag. 21(4), 42–44 (2001)
C. Anderson, Z. Hong, Reinforcement learning with modular neural networks for control, in Proceedings of NNACIP’94: IEEE International Workshop on Neural Networks Applied to Control and Image Processing (1994)
W. Sun, X. Wang, Y. Cheng, Reinforcement learning method for continuous state space based on dynamic neural network, in Proceedings of the 7th World Congress on Intelligent Control and Automation (2008), pp. 750–754
D. Gu, H. Hu, Reinforcement learning of fuzzy logic controller for quadruped walking robots. Presented at Proceedings of 15th IFAC World Congress, Barcelona, Spain, 21–26 July 2002
H.R. Berenji, Fuzzy learning for generalization of reinforcement learning, in Proceedings of the Fifth IEEE International Conference on Fuzzy Systems, vol. 3 (1996), pp. 2208–2214
W.M. Hinojosa, S. Nefti, U. Kaymak, Systems control with generalized probabilistic fuzzy-reinforcement learning. IEEE Trans. Fuzzy Syst. 19(1), 51–64 (2011)
A. Likas, Reinforcement learning using the stochastic fuzzy min-max neural network. Neural Process. Lett. 13, 213–220 (2001)
T. Inamura, M. Inada, H. Inoue, Integration model of learning mechanism and dialogue strategy based on stochastic experience representation using Bayesian network, in Proceedings of the 2000 IEEE International Workshop on Robot and Human Interactive Communication (2000), pp. 247–252
S. Ushio, M. Svinin, K. Ueda, S. Hosoe, An evolutionary approach to decentralized reinforcement learning for walking robots, in Proceedings of the 6th International Symposium on Artificial Life and Robotics (2001), pp. 176–179
K. Ito, F. Matsuno, Reinforcement learning for redundant robot: solution of state explosion problem in real world, in Proceedings of ROBIO’05 Workshop on Biomimetic Robotics and Biomimetic Control (2005), pp. 36–41
R. Pfeifer, Understand Intell, new edn. (The MIT Press, Massachusetts, 2001)
J.J. Gibson, The Ecological Approach to Visual Perception (Lawrence Erlbaum Associates, Hillsdale, NJ, 1987)
K. Ito, A. Takayama, T. Kobayashi, Hardware design of autonomous snake-like robot for reinforcement learning based on environment: discussion of versatility on different tasks, in Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (2009), pp. 2622–2627
K. Ito, Y. Fukumori, A. Takayama, Autonomous control of real snake-like robot using reinforcement learning: abstraction of state-action space using properties of real world, in Proceedings of the International Conference on Intelligent Sensors, Sensor Networks and Information Processing (2007), pp. 389–394
K. Ito, S. Kuroe, T. Kobayashi, Abstraction of state-action space utilizing properties of the body and the environment: application to a three-dimensional snake-like robot that operates on rubble, in Proceedings of IEEE International Conference on Intelligent Systems (2012), pp. 114–120
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Ito, K., Kuroe, S., Kobayashi, T. (2016). Abstraction of State-Action Space Utilizing Properties of the Body and Environment. In: Hadjiski, M., Kasabov, N., Filev, D., Jotsov, V. (eds) Novel Applications of Intelligent Systems. Studies in Computational Intelligence, vol 586. Springer, Cham. https://doi.org/10.1007/978-3-319-14194-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-14194-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-14193-0
Online ISBN: 978-3-319-14194-7
eBook Packages: EngineeringEngineering (R0)