Concurrent Modular Q-Learning with Local Rewards on Linked Multi-Component Robotic Systems

  • Borja Fernandez-Gauna
  • Jose Manuel Lopez-Guede
  • Manuel Graña
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6686)


Applying conventional Q-Learning to Multi-Component Robotic Systems (MCRS) increasing the number of components produces an exponential growth of state storage requirements. Modular approaches limit the state size growth to be polynomial on the number of components, allowing more manageable state representation and manipulation. In this article, we advance on previous works on a modular Q-learning approach to learn the distributed control of a Linked MCRS. We have chosen a paradigmatic application of this kind of systems using only local rewards: a set of robots carrying a hose from some initial configuration to a desired goal. The hose dynamics are simplified to be a distance constraint on the robots positions.


Constraint Module Negative Reward Goal Module Local Reward Veto System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Busoniu, L., Babuska, R., Schutter, B.D.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 38(2), 156–172 (2008)CrossRefGoogle Scholar
  2. 2.
    Duro, R.J., Graña, M., de Lope, J.: On the potential contributions of hybrid intelligent approaches to multicomponen robotic system development. Information Sciences 180(14), 2635–2648 (2010)CrossRefGoogle Scholar
  3. 3.
    Echegoyen, Z., Villaverde, I., Moreno, R., Graña, M., d’Anjou, A.: Linked multi-component mobile robots: modeling, simulation and control. Robotics and Autonomous Systems 58(12), 1292–1305 (2010)CrossRefGoogle Scholar
  4. 4.
    Fernandez-Gauna, B., Graña, M., Lopez-Guede, J.M.: Towards concurrent q-learning on linked multi-component robotic systems. In: HAIS 2011. LNCS (LNAI), Springer, Heidelberg (2011) (in press)Google Scholar
  5. 5.
    Fernandez-Gauna, B., Lopez-Guede, J.M., Zulueta, E., Graña, M.: Learning hose transport control with Q-learning. Neural Network World 20(7), 913–923 (2010)Google Scholar
  6. 6.
    Fernandez-Gauna, B., Lopez-Guede, J.M., Zulueta, E.: Linked multicomponent robotic systems: Basic assessment of linking element dynamical effect. In: Corchado, E., Graña, M., Savio, A. (eds.) Hybrid Artificial Intelligence Systems, Part I, vol. 6076, pp. 73–79. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Fernandez-Gauna, B., Lopez-Guede, J.M., Zulueta, E., Echegoyen, Z., Graña, M.: Basic results and experiments on robotic multi-agent system for hose deployment and transportation. International Journal of Artificial Intelligence 6(S11), 183–202 (2011)Google Scholar
  8. 8.
    Graña, M., Torrealdea, F.J.: Hierarchically structured systems. European Journal of Operational Research 25, 20–26 (1986)CrossRefGoogle Scholar
  9. 9.
    Maravall, D., de Lope, J., Martin, J.A.: Hybridizing evolutionary computation and reinforcement learning for the design of almost universal controllers for autonomous robots. Neurocomputing 72(4-6), 887–894 (2009)CrossRefGoogle Scholar
  10. 10.
    Panait, L., Luke, S.: Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-Agent Systems 11(3), 387–434 (2005)CrossRefGoogle Scholar
  11. 11.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  12. 12.
    Whitehead, S., Karlsson, J., Tenenberg, J.: Learning multiple goal behavior via task decomposition and dynamic policy merging. In: Robot Learning, pp. 45–78. Kluwer Academic Publisher, Dordrecht (1993)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Borja Fernandez-Gauna
    • 1
  • Jose Manuel Lopez-Guede
    • 1
  • Manuel Graña
    • 1
  1. 1.University of the Basque Country(UPV/EHU)Spain

Personalised recommendations