Markov Decision Processes in Practice pp 371-386 | Cite as

# Dynamic Control of Traffic Lights

## Abstract

Traffic lights are put in place to dynamically change priority between traffic participants. Commonly, the duration of green intervals and the grouping, and ordering in which traffic flows are served are pre-fixed. In this chapter, the problem of minimizing vehicle delay at isolated intersections is formulated as a Markov Decision Process (MDP). Solving the MDP is hampered by a large multi-dimensional state space that contains information on the traffic lights and on the queue lengths. For a single intersection, an approximate solution is provided that is based on policy iteration (PI) and decomposition of the state space. The approach starts with a Markov chain analysis of a pre-timed control policy, called Fixed Cycle (FC). The computation of relative states values for FC can be done fast, since, under FC, the multi-dimensional state space can be decomposed into sub-spaces per traffic flow. The policy obtained by executing a single iteration of Policy Iteration (PI) using relative values is called RV1. RV1 is compared for two intersections by simulation with FC, a few dynamic (vehicle actuated) policies, and an optimal MDP policy (if tractable). RV1, approximately solves the MDP, and compared to FC, it shows less delay of vehicles, shorter queues, and is robust to changes in traffic volumes. The approach shows very short computation times, which allows the application to networks of intersections, and the inclusion of estimated arrival times of vehicles approaching the intersection.

## Notes

### Acknowledgements

This work has been funded by grants from the Spanish state (TIN2015-66680-C2-2-R) and, Junta de Andalucía (P11-TIC-7176), in part financed by the European Regional Development Fund (ERDF).

## References

- 1.L. Baskar, B. De Schutter, J. Hellendoorn, Z. Papp, Traffic control and intelligent vehicle highway systems: a survey. IET Intell. Transp. Syst.
**5**(1), 38–52 (2011)CrossRefGoogle Scholar - 2.S. Bhulai, Dynamic routing policies for multiskill call centers. Probab. Eng. Inf. Sci.
**23**(01), 101–119 (2009)CrossRefGoogle Scholar - 3.R. Haijema, Solving large structured Markov decision problems for perishable inventory management and traffic control. Ph.D. thesis, Univeristy of Amsterdam, Tinbergen Institute, Amsterdam School of Economics, 2008Google Scholar
- 4.R. Haijema, E.M. Hendrix, Traffic responsive control of intersections with predicted arrival times: a Markovian approach. Comput. Aided Civ. Infrastruct. Eng.
**29**(2), 123–139 (2014)CrossRefGoogle Scholar - 5.R. Haijema, J. van der Wal, An MDP decomposition approach for traffic control at isolated signalized intersections. Probab. Eng. Inf. Sci.
**22**(4), 587–602 (2008)CrossRefGoogle Scholar - 6.J.F. Herrera, E.M. Hendrix, L.G. Casado, R. Haijema, Data parallelism in traffic control tables with arrival information, in
*Euro-Par 2014: Parallel Processing Workshops*(Springer, Berlin, 2014), pp. 60–70Google Scholar - 7.H.X. Liu, A. Danczyk, Optimal sensor locations for freeway bottleneck identification. Comput. Aided Civ. Infrastruct. Eng.
**24**(8), 535–550 (2009)CrossRefGoogle Scholar - 8.J.Y.K. Luk, Two traffic-responsive area traffic control methods: SCAT and SCOOT. Traffic Eng. Control
**25**, 14–22 (1984)Google Scholar - 9.M. Papageorgiou, C. Diakaki, V. Dinopoulou, A. Kotsialos, Y. Wang, Review of road traffic control strategies, in
*Proceedings of the IEEE*, vol. 91 (IEEE, New York, 2003), pp. 2043–2067Google Scholar - 10.W.B. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley Series in Probability and Statistics (Wiley, New York, 2007)Google Scholar
- 11.M.L. Puterman,
*Markov Decision Processes: Discrete Stochastic Dynamic Programming*(Wiley, New York, 2014)Google Scholar - 12.A. Stathopoulos, L. Dimitriou, T. Tsekeris, Fuzzy modeling approach for combined forecasting of urban traffic flow. Comput. Aided Civ. Infrastruct. Eng.
**23**(7), 521–535 (2008)CrossRefGoogle Scholar - 13.J. Wijngaard, Decomposition for dynamic programming in production and inventory control. Eng. Process. Econ.
**4**, 385–388 (1979)CrossRefGoogle Scholar - 14.D. Zhao, Y. Dai, Z. Zhang, Computational intelligence in urban traffic signal control: a survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev.
**42**(4), 485–494 (2012)CrossRefGoogle Scholar