Abstract
Traffic congestion is a challenging problem faced in everyday life. It has multiple negative effects on average speed, overall total travel time, and fuel consumption; in addition, it is a primary cause of accidents and air pollution. Hence, comes the need for an intelligent reliable traffic control system. The objective of this paper is to optimize the overall traffic congestion in freeways via multiple ramps metering controls. An optimal freeway operation can be reached if we always keep the freeway density within a small margin of the critical ratio for maximum traffic flow. In this paper, we propose a multi-agent reinforcement learning control system for ramp metering. Our system introduces a new microscopic framework at the network level based on collaborative Markov Decision Process modeling and an associated cooperative Q-learning algorithm. The technique incorporates payoff propagation (max-plus algorithm) under the coordination graph framework, particularly suited for optimal control purposes. The proposed system provides three control designs: fully independent, fully distributed, and centralized; suited for different network architectures. Our framework was extensively tested in order to assess the proposed model of the joint payoff, as well as the global payoff. We conducted experiments with heavy traffic flow under the renowned VISSIM traffic simulator so as to evaluate the proposed framework. The experimental results show that we achieved a significant decrease in the total travel time and an increase in the average speed -when compared with the base case- while maintaining an optimal traffic flow.
An erratum to this chapter is available at http://dx.doi.org/10.1007/978-3-319-08422-0_131
An erratum to this chapter can be found at http://dx.doi.org/10.1007/978-3-319-08422-0_131
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Yu, X., Xu, W., Alam, F., Potgieter, J., Fang, C.: Genetic fuzzy logic approach to local ramp metering control using microscopic traffic simulation. In: Mechatronics and Machine Vision in Practice (M2VIP), 2012 19th International Conference. (2012) 290–297
Ghods, A., Kian, A., Tabibi, M.: Adaptive freeway ramp metering and variable speed limit control: a genetic-fuzzy approach. Intelligent Transportation Systems Magazine, IEEE 1(1) (2009) 27–36
Liang, X., Li, J., Luo, N.: Single neuron based freeway traffic density control via ramp metering. In: Information Engineering and Computer Science (ICIECS), 2010 2nd International Conference on. (2010) 1–4
Li, J., Liang, X.: Freeway ramp control based on single neuron. In: Intelligent Computing and Intelligent Systems, 2009. ICIS 2009. IEEE International Conference on. Volume 2. (2009) 122–125
Feng, C., Yuanhua, J., Jian, L., Huixin, Y., Zhonghai, N.: Design of fuzzy neural network control method for ramp metering. In: Measuring Technology and Mechatronics Automation (ICMTMA), 2011 Third International Conference on. Volume 1. (2011) 966–969
Veljanovska, K., Gacovski, Z., Deskovski, S.: Intelligent system for freeway ramp metering control. In: Intelligent Systems (IS), 2012 6th IEEE International Conference, IEEE (2012) 279–282
Davarynejad, M., Hegyi, A., Vrancken, J., van den Berg, J.: Motorway ramp-metering control with queuing consideration using q-learning. In: Intelligent Transportation Systems (ITSC), 2011 14th International IEEE Conference on, IEEE (2011) 1652–1658
Ji, X., He, Z.: An optimal control method for expressways entering ramps metering based on q-learning. In: Intelligent Computation Technology and Automation, 2009. ICICTA’09. Second International Conference on. Volume 1., IEEE (2009) 739–741
Vlassis, N., Elhorst, R., Kok, J.R.: Anytime algorithms for multiagent decision making using coordination graphs. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on. Volume 1., IEEE (2004) 953–957
Guestrin, C.E.: Planning under uncertainty in complex structured environments. PhD thesis, Stanford University (2003)
Pearl, J.: Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan Kaufmann (1988)
Ernst, D., Glavic, M., Capitanescu, F., Wehenkel, L.: Reinforcement learning versus model predictive control: a comparison on a power system problem. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on 39(2) (2009) 517–529
Watkins, C.J., Dayan, P.: Q-learning. Machine learning 8(3–4) (1992) 279–292
Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. arXiv preprint cs/9605103 (1996)
Ahmed Fares, W.G.: Freeway ramp-metering control based on reinforcement learning. In: Control and Automation, 2014. ICCA 2014. 11th IEEE International Conference on. (in press)
Papageorgiou, M., Hadj-Salem, H., Blosseville, J.M.: Alinea: A local feedback control law for on-ramp metering. Transportation Research Record (1320) (1991)
Wainwright, M., Jaakkola, T., Willsky, A.: Tree consistency and bounds on the performance of the max-product algorithm and its generalizations. Statistics and Computing 14(2) (2004) 143–166
Maciejewski, M.: A comparison of microscopic traffic flow simulation systems for an urban area. Problemy Transportu 5 (2010) 27–38
Prendinger, H., Gajananan, K., Bayoumy Zaki, A., Fares, A., Molenaar, R., Urbano, D., van Lint, H., Gomaa, W.: Tokyo virtual living lab: Designing smart cities based on the 3d internet. Internet Computing, IEEE 17(6) (Nov 2013) 30–38
Acknowledgments
This research is supported by the Ministry of Higher Education (MoHE) of Egypt through PhD fellowships. Our sincere thanks to E-JUST University for guidance and support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Fares, A., Gomaa, W. (2015). Multi-Agent Reinforcement Learning Control for Ramp Metering. In: Selvaraj, H., Zydek, D., Chmaj, G. (eds) Progress in Systems Engineering. Advances in Intelligent Systems and Computing, vol 366. Springer, Cham. https://doi.org/10.1007/978-3-319-08422-0_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-08422-0_25
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-08421-3
Online ISBN: 978-3-319-08422-0
eBook Packages: EngineeringEngineering (R0)