Abstract
State space construction is one of the most fundamental issues for reinforcement learning methods to be applied to real robot tasks because they need a well-defined state space so that they can converge correctly. Especially in multi-agent environments, the problem becomes more difficult since visual information observed by a learning robot seems irrelevant to its self motion due to actions by other agents of which policies are unknown. This paper proposes a method which estimates the relationship between the learner’s behaviors and the other agents’ ones in the environment through interactions (observation and action) using the method of system identification to construct a state space in such an environment. In order to determine the state vectors of each agent, Akaike’s Information Criterion is applied to the result of the system identification. Next, reinforcement learning based on the estimated state vectors is utilized to obtain the optimal behavior. The proposed method is applied to soccer playing physical agents, which learn to cope with a rolling ball and another moving agent. The computer simulations and the real experiments are shown and a discussion is given.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
H. Akaike. A new look on the statistical model identification. IEEE Trans. AC-19, pages 716–723, 1974.
M. Asada, S. Noda, and K. Hosoda. Action-based sensor space categorization for robot learning. In Proc. of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, 1996.
M. Asada, S. Noda, S. Tawaratsumida, and K. Hosoda. Vision-based reinforcement learning for purposive behavior acquisition. In Proc. of IEEE International Conference on Robotics and Automation, pages 146–153, 1995.
J. H. Connel and S. Mahadevan. Robot Learning. Kluwer Academic Publishers, 1993.
M. Inaba. Remote-brained robotics: Interfacing AI with real world behaviors. In Preprints of ISRR’93, Pitsuburg, 1993.
H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, E. Osawa, and H. Matsubara. Robocup a challenge problem for ai. AI Magazine, 18(l):73–85, 1997.
W. E. Larimore. Canonical variate analysis in identification, filtering, and adaptive control. In Proc. 29th IEEE Conference on Decision and Control, pages 596–604, Honolulu, Hawaii, December 1990.
L.-J. Lin and T. M. Mitchell. Reinforcement learning with hidden states. In Proc. of the 2nd International Conference on Simulation of Adaptive Behavior: From Animals to Animats 2., pages 271–280, 1992.
M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proc. of the 11th International Conference on Machine Learning, pages 157–163, 1994.
A. W. Moore and C. G. Atkeson. The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces. Machine Learning, 21:199–233, 1995.
T. W. Sandholm and R. H. Crites. On multiagent Q-learning in a semi-competitive domain. In Workshop Notes of Adaptation and Learning in Multiagent Systems Workshop, IJCAI-95, 1995.
P. Stone and M. Veloso. Using machine learning in the soccer server. In Proc. of IROS-96 Workshop on Robocup, 1996.
E. Uchibe, M. Asada, and K. Hosoda. Behavior coordination for a mobile robot using modular reinforcement learning. In Proc. of the 1996 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1329–1336, 1996.
E. Uchibe, M. Asada, and K. Hosoda. Environmental complexity control for vision-based learning mobile robot. In Proc. of IEEE International Conference on Robotics and Automation, 1998 (to appear).
P. Van Overschee and B. De Moor. A unifying theorem for three subspace system identification algorithms. Automatica, 31(12):1853–1864, 1995.
C. J. C. H. Watkins and P. Dayan. Technical note: Q-learning. Machine Learning, pages 279–292, 1992.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Uchibe, E., Asada, M., Hosoda, K. (1998). Vision Based State Space Construction for Learning Mobile Robots in Multi Agent Environments. In: Birk, A., Demiris, J. (eds) Learning Robots. EWLR 1997. Lecture Notes in Computer Science(), vol 1545. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-49240-2_5
Download citation
DOI: https://doi.org/10.1007/3-540-49240-2_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-65480-3
Online ISBN: 978-3-540-49240-5
eBook Packages: Springer Book Archive