Skip to main content
SpringerLink
Log in

Distributed MPC Using Reinforcement Learning Based Negotiation: Application to Large Scale Systems

  • Chapter
  • First Online:
Book cover Distributed Model Predictive Control Made Easy

Part of the book series: Intelligent Systems, Control and Automation: Science and Engineering ((ISCA,volume 69))

Abstract

This chapter describes a methodology to deal with the interaction (negotiation) between MPC controllers in a distributed MPC architecture. This approach combines ideas from Distributed Artificial Intelligence (DAI) and Reinforcement Learning (RL) in order to provide a controller interaction based on negotiation, cooperation and learning techniques. The aim of this methodology is to provide a general structure to perform optimal control in networked distributed environments, where multiple dependencies between subsystems are found. Those dependencies or connections often correspond to control variables. In that case, the distributed control has to be consistent in each subsystem. One of the main new concepts of this architecture is the negotiator agent. Negotiator agents interact with MPC agents to reach an agreement on the optimal value of the shared control variables. The optimal value of those shared control variables has to accomplish a common goal, probably incompatible with the specific goals of each partition that share the variable. Two cases of study are discussed, a small water distribution network and the Barcelona water network. The results suggest that this approach is a promising strategy when centralized control is not a reasonable choice.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. D. Barcelli, Decomposizione ottima e controllo predittivo distribuito della rete idrica di Barcellona (Università di Siena, Facoltà di Ingegneria Informatica, Master’s thesis, 2008)

    Google Scholar 

  2. M. Brdys, B. Ulanicki, Operational Control of Water Systems: Structures, Algorithms and Applications (Prentice Hall International, Hemel Hempstead, Hertfordshire, 1994)

    Google Scholar 

  3. V. Fambrini, C. Ocampo-Martinez, Modelling and decentralized Model Predictive Control of drinking water networks. Technical Report IRI-TR-04-09, Institut de Robòtica i Informàtica Industrial (CSIC-UPC), April 2009

    Google Scholar 

  4. H. El Fawal, D. Georges, and G. Bornard, Optimal control of complex irrigation systems via descomposition-coordination and the use of augmented Lagrangian, in IEEE International Conference Systems, Man and Cybernetics, San Diego, 1998, pp. 3874–3879

    Google Scholar 

  5. M. Gómez, J. Rodellar, F. Vea, J. Mantecón, and J. Cardona, Decentralized adaptive control for water distribution, in IEEE International on Systems, Man and Cybernetics, San Diego, 1998, pp. 1411–1416

    Google Scholar 

  6. T. Jaakkola, M.I. Jordan, S.P. Singh, Q-learning. Mach. Learn. 8, 1185–1201 (1994)

    Google Scholar 

  7. V. Javalera, B. Morcego, V. Puig, Distributed MPC for large scale systems using agent-based reinforcement learning, in IFAC Symposium Large Scale Systems, Lille 2010

    Google Scholar 

  8. R.R. Negenborn, B. De Schutter, J. Hellendoorn, Multi-agent model predictive control for transportation networks: Serial vs. parallel schemes. Eng. Appl. Artif. Intell. 21(3), 353–366 (April 2008)

    Article  Google Scholar 

  9. C. Ocampo-Martinez, D. Barcelli, V. Puig, and A. Bemporad, Hierarchical and decentralised model predictive control of drinking water networks: Application to the barcelona case study. IET Control Theory & Applications, Conditionally accepted, 2011

    Google Scholar 

  10. C. Ocampo-Martinez, S. Bovo, V. Puig, Partitioning approach oriented to the decentralised predictive control of large-scale systems. J. Process Control 21(5), 775–786 (2011)

    Article  Google Scholar 

  11. J. Quevedo, V. Puig, G. Cembrano, J. Blanch, Validation and reconstruction of flow meter data in the Barcelona water distribution network. Control Eng. Pract. 11(6), 640–651 (June 2010)

    Article  Google Scholar 

  12. J.B. Rawlings, B. Stewart, Coordinating multiple optimization-based controllers: New opportunities and challenges. J. Process Control 18(9), 839–845 (2008)

    Article  Google Scholar 

  13. D.D. Šiljak, Decentralized Control of Complex Systems (Academic Press, New York, 1991)

    Google Scholar 

  14. J.N. Tsitsiklis, Asynchronous stochastic approximation and Q-learning. Mach. Learn. 16, 185–202 (1994)

    MATH  Google Scholar 

  15. A.N. Venkat, J.B. Rawlings, S.J. Wrigth, Stability and optimality of distributed model predictive control, in IEEE Conference on Decision and Control and European Control Conference, Seville, 2005

    Google Scholar 

  16. C.I.C.H. Watkins, Learning from Delayed Rewards. Doctoral Dissertation (University of Cambridge, Cambridge, 1989)

    Google Scholar 

  17. C.I.C.H. Watkins, P. Dayan, Q-learning. Mach. Learn. 8, 279–292 (1992)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Advanced Control Systems Group, Terrassa, Spain

    B. Morcego, V. Javalera, V. Puig & R. Vito

Authors
  1. B. Morcego
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. Javalera
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. V. Puig
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Vito
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B. Morcego .

Editor information

Editors and Affiliations

  1. Systems Engineering and Automation Dept, University of Seville, Seville, Spain

    José M. Maestre

  2. Dept of Marine & Transport Technology, Delft University of Technology, Delft, The Netherlands

    Rudy R. Negenborn

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Morcego, B., Javalera, V., Puig, V., Vito, R. (2014). Distributed MPC Using Reinforcement Learning Based Negotiation: Application to Large Scale Systems. In: Maestre, J., Negenborn, R. (eds) Distributed Model Predictive Control Made Easy. Intelligent Systems, Control and Automation: Science and Engineering, vol 69. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-7006-5_32

Download citation

  • DOI: https://doi.org/10.1007/978-94-007-7006-5_32

  • Published:

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-007-7005-8

  • Online ISBN: 978-94-007-7006-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation