Learning of Coordinated Behavior

  • Dirk Helbing
Part of the Understanding Complex Systems book series (UCS)


In many social dilemmas, individuals tend to generate a situation with low payoffs instead of a system optimum (”tragedy of the commons”). Is the routing of traffic a similar problem? In order to address this question, we present experimental results on humans playing a route choice game in a computer laboratory, which allow one to study decision behavior in repeated games beyond the Prisoner’s Dilemma. We will focus on whether individuals manage to find a cooperative and fair solution compatible with the system-optimal road usage. We find that individuals tend towards a user equilibrium with equal travel times in the beginning. However, after many iterations, they often establish a coherent oscillatory behavior, as taking turns performs better than applying pure or mixed strategies. The resulting behavior is fair and compatible with system-optimal road usage. In spite of the complex dynamics leading to coordinated oscillations, we have identified mathematical relationships quantifying the observed transition process. Our main experimental discoveries for 2- and 4-person games can be explained with a novel reinforcement learning model for an arbitrary number of persons, which is based on past experience and trial-and-error behavior. Gains in the average payoff seem to be an important driving force for the innovation of time-dependent response patterns, i.e. the evolution of more complex strategies. Our findings are relevant for decision support systems and routing in traffic or data networks.


Test Person Route Choice User Equilibrium Congestion Game Minority Game 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



D.H. is grateful for the warm hospitality of the Santa Fe Institute, where the Social Scaling Working Group Meeting in August 2003 inspired many ideas of this paper. The results shall be presented during the workshop on “Collectives Formation and Specialization in Biological and Social Systems” in Santa Fe (April 20–22, 2005).


  1. 1.
    W.B. Arthur, Inductive reasoning and bounded rationality. Am. Econ. Rev. 84, 406–411 (1994)Google Scholar
  2. 2.
    R. Axelrod, D. Dion, The further evolution of cooperation. Science 242, 1385–1390 (1988)Google Scholar
  3. 3.
    R. Axelrod, W.D. Hamilton, The evolution of cooperation. Science 211, 1390–1396 (1981)Google Scholar
  4. 4.
    M. Beckmann, C.B., McGuire, C.B. Winsten, Studies in the Economics of Transportation (Yale University Press, New Haven, 1956)Google Scholar
  5. 5.
    K.G. Binmore, Evolutionary stability in repeated games played by finite automata. J. Econ. Theor. 57, 278–305 (1992)Google Scholar
  6. 6.
    K. Binmore, Fun and Games: A Text on Game Theory (Heath, Lexington, MA, 1992), pp. 373–377Google Scholar
  7. 7.
    U. Bohnsack, Uni DuE: Studie SURVIVE gibt Einblicke in das Wesen des Autofahrers, Press release by Informationsdienst Wissenschaft (January 21, 2005)Google Scholar
  8. 8.
    P.W. Bonsall, T. Perry, Using an interactive route-choice simulator to investigate driver’s compliance with route guidance information. Transpn. Res. Rec. 1306, 59–68 (1991)Google Scholar
  9. 9.
    G. Bottazzi, G. Devetag, Coordination and self-organization in minority games: Experimental evidence, Working Paper 2002/09, Sant’Anna School of Advances Studies, May 2002Google Scholar
  10. 10.
    D. Braess, Über ein Paradoxon der Verkehrsplanung [A paradox of traffic assignment problems]. Unternehmensforschung 12, 258–268 (1968). For about 100 related references see
  11. 11.
    L. Browning, A.M. Colman, Evolution of coordinated alternating reciprocity in repeated dyadic games. J. Theor. Biol. 229, 549–557 (2004)Google Scholar
  12. 12.
    C.F. Camerer, Behavioral Game Theory: Experiments on Strategic Interaction (Princeton University Press, Princeton, 2003)Google Scholar
  13. 13.
    C.F. Camerer, T.-H. Ho, J.-K. Chong, Sophisticated experience-weighted attraction learning and strategic teaching in repeated games. J. Econ. Theor. 104, 137–188 (2002)Google Scholar
  14. 14.
    N. Cetin, K. Nagel, B. Raney, A. Voellmy, Large scale multi-agent transportation simulations. Comput. Phys. Comm. 147(1–2), 559–564 (2002)Google Scholar
  15. 15.
    D. Challet, M. Marsili, Relevance of memory in minority games. Phys. Rev. E 62, 1862–1868 (2000)Google Scholar
  16. 16.
    D. Challet, Y.-C. Zhang, Emergence of cooperation and organization in an evolutionary game. Physica A 246, 407–418 (1997)Google Scholar
  17. 17.
    J.E. Cohen, P. Horowitz Paradoxial behaviour of mechanical and electrical networks. Nature 352, 699–701 (1991)Google Scholar
  18. 18.
    A.M. Colman, Game Theory and its Applications in the Social and Biological Sciences, 2nd edn. (Butterworth-Heinemann, Oxford, 1995)Google Scholar
  19. 19.
    A.M. Colman, Depth of strategic reasoning in games. Trends Cognit. Sci. 7(1), 2–4 (2003)Google Scholar
  20. 20.
    P.H. Crowley, Dangerous games and the emergence of social structure: evolving memory-based strategies for the generalized hawk-dove game. Behav. Ecol. 12, 753–760 (2001)Google Scholar
  21. 21.
    A. Eriksson, K. Lindgren Cooperation in an unpredictable environment, in Proceedings of Artificial Life VIII, ed. by R.K. Standish, M.A. Bedau, H.A. Abbass (MIT Press, Sidney, 2002), pp. 394–399 and poster available at
  22. 22.
    C.B. Garcia, W.I. Zangwill, Pathways to Solutions, Fixed Points, and Equilibria (Prentice Hall, New York, 1981)Google Scholar
  23. 23.
    N.S. Glance, B.A. Huberman, The outbreak of cooperation. J. Math. Sociol. 17(4), 281–302 (1993)Google Scholar
  24. 24.
    B.D. Greenshield, A study of traffic capacity, in Proceedings of the Highway Research Board, Vol. 14 (Highway Research Board, Washington, D.C., 1935), pp. 448–477Google Scholar
  25. 25.
    G. Hardin, The tragedy of the commons. Science 162, 1243–1248 (1968)Google Scholar
  26. 26.
    D. Helbing, Dynamic decision behavior and optimal guidance through information services: Models and experiments, in Human Behaviour and Traffic Networks, ed. by M. Schreckenberg, R. Selten (Springer, Berlin, 2004), pp. 47–95Google Scholar
  27. 27.
    D. Helbing, M. Schönhof, D. Kern, Volatile decision dynamics: Experiments, stochastic description, intermittency control, and traffic optimization. New J. Phys. 4, 33.1–33.16 (2002)Google Scholar
  28. 28.
    J. Hofbauer, K. Sigmund, The Theory of Evolution and Dynamical Systems (Cambridge University Press, Cambridge, 1988)Google Scholar
  29. 29.
    T. Hogg, B.A. Huberman, Controlling chaos in distributed systems. IEEE Trans. Syst. Man Cybern. 21(6), 1325–1333 (1991)Google Scholar
  30. 30.
    T.-Y. Hu, H.S. Mahmassani, Day-to-day evolution of network flows under real-time information and reactive signal control. Transport. Res. C 5(1), 51–69 (1997)Google Scholar
  31. 31.
    S. Iwanaga, A. Namatame, The complexity of collective decision. Nonlinear Dynam. Psychol. Life Sci. 6(2), 137–158 (2002)Google Scholar
  32. 32.
    J.H. Kagel, A.E. Roth (eds.), The Handbook of Experimental Economics (Princeton University, Princeton, NJ, 1995)Google Scholar
  33. 33.
    J.O. Kephart, T. Hogg, B.A. Huberman, Dynamics of computational ecosystems. Phys. Rev. A 40(1), 404–421 (1989)Google Scholar
  34. 34.
    F. Klügl, A.L.C. Bazzan, Route decision behaviour in a commuting scenario: Simple heuristics adaptation and effect of traffic forecast. JASSS 7(1) (2004)Google Scholar
  35. 35.
    Y.A. Korilis, A.A. Lazar, A. Orda Avoiding the Braess-paradox in non-cooperative networks. J. Appl. Probab. 36, 211–222 (1999)Google Scholar
  36. 36.
    P. Laureti, P. Ruch, J. Wakeling, Y.-C. Zhang, The interactive minority game: a Web-based investigation of human market interactions. Physica A 331, 651–659 (2004)Google Scholar
  37. 37.
    K. Lee, P.M. Hui, B.H. Wang, N.F. Johnson, Effects of announcing global information in a two-route traffic flow model. J. Phys. Soc. Japan 70, 3507–3510 (2001)Google Scholar
  38. 38.
    T.S. Lo, H.Y. Chan, P.M. Hui, N.F. Johnson, Theory of networked minority games based on strategy pattern dynamics. Phys. Rev. E 70, 056102 (2004)Google Scholar
  39. 39.
    T.S. Lo, P.M. Hui, N.F. Johnson, Theory of the evolutionary minority game. Phys. Rev. E 62, 4393–4396 (2000)Google Scholar
  40. 40.
    M.W. Macy, A. Flache, Learning dynamics in social dilemmas. Proc. Natl. Acad. Sci. USA 99(Suppl. 3), 7229–7236 (2002)Google Scholar
  41. 41.
    H.S. Mahmassani, R.C. Jou, Transferring insights into commuter behavior dynamics from laboratory experiments to field surveys. Transport. Res. A 34, 243–260 (2000)Google Scholar
  42. 42.
    R. Mansilla, Algorithmic complexity in the minority game. Phys. Rev. E 62, 4553–4557 (2000)Google Scholar
  43. 43.
    M. Marsili, R. Mulet, F. Ricci-Tersenghi, R. Zecchina, Learning to coordinate in a complex and nonstationary world. Phys. Rev. Lett. 87, 208701 (2001)Google Scholar
  44. 44.
    J.M. McNamara, Z. Barta, A.I. Houston, Variation in behaviour promotes cooperation in the prisoner’s dilemma game. Nature 428, 745–748 (2004)Google Scholar
  45. 45.
    F. Michor, M.A. Nowak, The good, the bad and the lonely. Nature 419, 677–679 (2002)Google Scholar
  46. 46.
    M. Milinski, D. Semmann, H.-J. Krambeck, Reputation helps solve the ‘tragedy of the commons’. Nature 415, 424–426 (2002)Google Scholar
  47. 47.
    D. Monderer, L.S. Shapley, Fictitious play property for games with identical interests. J. Econ. Theor. 1, 258–265 (1996)Google Scholar
  48. 48.
    D. Monderer, L.S. Shapley, Potential games. Game. Econ. Behav. 14, 124–143 (1996)Google Scholar
  49. 49.
    M.A. Nowak, A. Sasaki, C. Taylor, D. Fudenberg, Emergence of cooperation and evolutionary stability in finite populations. Nature 428, 646–650 (2004)Google Scholar
  50. 50.
    M. Novak, K. Sigmund, A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game. Nature 364, 56–58 (1993)Google Scholar
  51. 51.
    M.A. Nowak, K. Sigmund The alternating prisoner’s dilemma. J. theor. Biol. 168, 219–226 (1994)Google Scholar
  52. 52.
    M.A. Nowak, K. Sigmund, Evolution of indirect reciprocity by image scoring. Nature 393, 573–577 (1998)Google Scholar
  53. 53.
    A.C. Pigou, The Economics of Welfare (Macmillan, London, 1920)Google Scholar
  54. 54.
    M. Posch, Win-stay, lose-shift strategies for repeated games—Memory length, aspiration levels and noise. J. Theor. Biol. 198, 183–195 (1999)Google Scholar
  55. 55.
    D.C. Queller, Kinship is relative. Nature 430, 975–976 (2004)Google Scholar
  56. 56.
    A. Rapoport, Exploiter, leader, hero, and martyr: the four archtypes of the 2 ×2 game. Behav. Sci. 12, 81–84 (1967)Google Scholar
  57. 57.
    A. Rapoport, M. Guyer, A taxonomy of 2 ×2 games. Gen. Syst. 11, 203–214 (1966)Google Scholar
  58. 58.
    P.D.V.G. Reddy, et al., Design of an artificial simulator for analyzing route choice behavior in the presence of information system. J. Math. Comp. Mod. 22, 119–147 (1995)Google Scholar
  59. 59.
    R.L. Riolo, M.D. Cohen, R. Axelrod, Evolution of cooperation without reciprocity. Nature 414, 441–443 (2001)Google Scholar
  60. 60.
    R.W. Rosenthal, A class of games possessing pure-strategy Nash equilibria. Int. J. Game Theor. 2, 65–67 (1973)Google Scholar
  61. 61.
    T. Roughgarden, E. Tardos, How bad is selfish routing? J. ACM 49(2), 236–259 (2002)Google Scholar
  62. 62.
    T.C. Schelling, Micromotives and Macrobehavior (WW Norton and Co, New York, 1978), pp. 224–231 + 237Google Scholar
  63. 63.
    M. Schreckenberg, R. Selten (eds.), Human Behaviour and Traffic Networks (Springer, Berlin, 2004)Google Scholar
  64. 64.
    F. Schweitzer, L. Behera, H. Mühlenbein, Evolution of cooperation in a spatial prisoner’s dilemma. Adv. Complex Syst. 5(2/3), 269–299 (2002)Google Scholar
  65. 65.
    R. Selten, et al., Experimental investigation of day-to-day route-choice behaviour and network simulations of autobahn traffic in North Rhine-Westphalia, in Human Behaviour and Traffic Networks, ed. by M. Schreckenberg, R. Selten (Springer, Berlin, 2004), pp. 1–21Google Scholar
  66. 66.
    R. Selten, M. Schreckenberg, T. Pitz, T. Chmura, S. Kube, Experiments and simulations on day-to-day route choice-behaviour. See
  67. 67.
    D. Semmann, H.-J. Krambeck, M. Milinski, Volunteering leads to rock-paper-scissors dynamics in a public goods game. Nature 425, 390–393 (2003)Google Scholar
  68. 68.
    P. Spirakis, Algorithmic aspects of congestion games, Invited talk at the 11th Colloquium on Structural Information and Communication Complexity (Smolenice Castle, Slovakia, June 21-23, 2004)Google Scholar
  69. 69.
    G. Szabó, C. Hauert, Phase transitions and volunteering in spatial public goods games. Phys. Rev. Lett. 89, 118101 (2002)Google Scholar
  70. 70.
    J. Wahle, A.L.C. Bazzan, F. Klügl, M. Schreckenberg, Decision dynamics in a traffic scenario. Physica A 287, 669–681 (2000)Google Scholar
  71. 71.
    J.G. Wardrop, Some theoretical aspects of road traffic research, in Proceedings of the Institution of Civil Engineers II, Vol. 1 (1952), pp. 325–378Google Scholar
  72. 72.
    D.H. Wolpert, K. Tumer, Collective intelligence, data routing and Braess’ paradox. J. Artif. Intell. Res. 16, 359–387 (2002)Google Scholar
  73. 73.
    T. Yamashita, K. Izumi, K. Kurumatani, Effect of using route information sharing to reduce traffic congestion. Lect. Notes Comput. Sci. 3012, 86–104 (2004)Google Scholar
  74. 74.
    B. Yuan, K. Chen, Evolutionary dynamics and the phase structure of the minority game. Phys. Rev. E 69, 067106 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Dirk Helbing
    • 1
  1. 1.CLU E1ETH ZurichZurichSwitzerland

Personalised recommendations