Distributed Real-Time Scheduling by Using Multi-agent Reinforcement Learning

  • Koji Iwamura
  • Nobuhiro Sugimura


Autonomous Distributed Manufacturing Systems (ADMS) have been proposed to realise flexible control structures of manufacturing systems. In the previous researches, a real-time scheduling method based on utility values has been proposed and applied to the ADMS. Multi-agent reinforcement learning is newly proposed and implemented to the job agents and resource agents, in order to improve their coordination processes. The status, the action and the reward are defined for the individual job agents and the resource agents to evaluate the suitable utility values based on the status of the ADMS. Some case studies of the real-time scheduling have been carried out to verify the effectiveness of the proposed methods.


Objective Function Schedule Problem Reinforcement Learning Machine Operation Machine Accuracy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Moriwaki, & T., Sugimura, N. (1992). Object-oriented modelling of autonomous distributed manufacturing system and its application to real-time scheduling. Proceedings of the ICOOMS ’92 (pp. 207–212).Google Scholar
  2. 2.
    Kadar, B., Monostori, L., & Szelke, E. (1998). An object-oriented framework for developing distributed manufacturing architectures. Journal of Intelligent Manufacturing, 9, 173–179.CrossRefGoogle Scholar
  3. 3.
    Ueda, K. (1992). An approach to bionic manufacturing systems based on DNA-type information. Proceedings of the ICOOMS ‘92, (pp. 303–308).Google Scholar
  4. 4.
    Ueda, K., Hatono, I., Fujii, N., & Vaario, J. (2000). Reinforcement learning approach to biological manufacturing systems. Annals of the CIRP, 49, 343–346.CrossRefGoogle Scholar
  5. 5.
    Hendrik, B., Jo, W., Paul, V., Luc, B., & Patrick, P. (1998). Reference architecture for holonic manufacturing systems: PROSA. Computers in Industry, 37, 255–274.CrossRefGoogle Scholar
  6. 6.
    Sugimura, N., Tanimizu, Y., & Iwamura, K. (2004). A Study on real-time scheduling for holonic manufacturing system. CIRP Journal of Manufacturing Systems, 33(5), 467–475.Google Scholar
  7. 7.
    Iwamura, K., Okubo, N., Tanimizu, Y., & Sugimura, N. (2006). Real-time scheduling for holonic manufacturing systems based on estimation of future status. International Journal of Production Research, 44(18–19), 3657–3675.MATHCrossRefGoogle Scholar
  8. 8.
    Iwamura, K., Nakano, A., Tanimizu, Y., & Sugimura, N. (2007). A study on real-time scheduling for holonic manufacturing systems -Simulation for estimation of future status by individual holons-. In M. Vladimir, V. Valeriy, & W. C. Armando (Eds.), LNAI 4659 HoloMAS 2007 (pp. 205–214). Heidelberg: Springer.Google Scholar
  9. 9.
    Sutton, R., & Barto, A. (1998). Reinforcement learning: an introduction. Cambridge: The MIT Press.Google Scholar
  10. 10.
    Paternina-Arboleda, C., & Das, T. (2005). A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem. Simulation Modelling Practice and Theory, 13, 389–406.CrossRefGoogle Scholar
  11. 11.
    Kaelbling, L., Littman, M., & Moore, A. (1996). Reinforcement learning: a survey. Journal of Artificial Intelligence Research, 4, 237–285.Google Scholar
  12. 12.
    Sutton, R. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9–44.Google Scholar
  13. 13.
    Wang, Y., & Usher, J. (2005). Application of reinforcement learning for agent-based production scheduling. Engineering Applications of Artificial Intelligence, 18, 73–82.CrossRefGoogle Scholar
  14. 14.
    Fujii, N., Takasu, R., Kobayashi, M., Ueda, K. (2005). Reinforcement learning based product dispatching scheduling in a semiconductor manufacturing system. Proceedings of the 38th CIRP International seminar on manufacturing systems, CD-ROMGoogle Scholar
  15. 15.
    Aydin, E., & Oztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.CrossRefGoogle Scholar
  16. 16.
    Aissani, N., Beldjilali, B., & Trentesaux, D. (2009). Dynamic scheduling of maintenance tasks in the petroleum industry: A reinforcement approach. Engineering Applications of Artificial Intelligence, 22, 1089–1103.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.Graduate School of EngineeringOsaka Prefecture UniversityOsakaJapan

Personalised recommendations