Evolutionary Game Theory

  • Ross CressmanEmail author
  • Joe Apaloo
Reference work entry


Evolutionary game theory developed as a means to predict the expected distribution of individual behaviors in a biological system with a single species that evolves under natural selection. It has long since expanded beyond its biological roots and its initial emphasis on models based on symmetric games with a finite set of pure strategies where payoffs result from random one-time interactions between pairs of individuals (i.e., on matrix games). The theory has been extended in many directions (including nonrandom, multiplayer, or asymmetric interactions and games with continuous strategy (or trait) spaces) and has become increasingly important for analyzing human and/or social behavior as well. This chapter initially summarizes features of matrix games before showing how the theory changes when the two-player game has a continuum of traits or interactions become asymmetric. Its focus is on the connection between static game-theoretic solution concepts (e.g., ESS, CSS, NIS) and stable evolutionary outcomes for deterministic evolutionary game dynamics (e.g., the replicator equation, adaptive dynamics).


ESS CSS NIS Neighborhood superiority Evolutionary game dynamics Replicator equation Adaptive dynamics Darwinian dynamics 



The authors thank Abdel Halloway for his assistance with Fig. 10.4. This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 690817.


  1. Abrams PA, Matsuda H (1997) Fitness minimization and dynamic instability as a consequence of predator–prey coevolution. Evol Ecol 11:1–10CrossRefGoogle Scholar
  2. Abrams PA, Matsuda H, Harada Y (1993) Evolutionary unstable fitness maxima and stable fitness minima of continuous traits. Evol Ecol 7:465–487CrossRefGoogle Scholar
  3. Akin E (1982) Exponential families and game dynamics. Can J Math 34:374–405MathSciNetzbMATHCrossRefGoogle Scholar
  4. Apaloo J (1997) Revisiting strategic models of evolution: the concept of neighborhood invader strategies. Theor Popul Biol 52:71–77zbMATHCrossRefGoogle Scholar
  5. Apaloo J (2006) Revisiting matrix games: the concept of neighborhood invader strategies. Theor Popul Biol 69:235–242zbMATHCrossRefGoogle Scholar
  6. Apaloo J, Brown JS, Vincent TL (2009) Evolutionary game theory: ESS, convergence stability, and NIS. Evol Ecol Res 11:489–515Google Scholar
  7. Aubin J-P, Cellina A (1984) Differential inclusions. Springer, BerlinzbMATHCrossRefGoogle Scholar
  8. Barabás G, Meszéna G (2009) When the exception becomes the rule: the disappearance of limiting similarity in the Lotka–Volterra model. J Theor Biol 258:89–94MathSciNetCrossRefGoogle Scholar
  9. Barabás G, Pigolotti S, Gyllenberg M, Dieckmann U, Meszéna G (2012) Continuous coexistence or discrete species? A new review of an old question. Evol Ecol Res 14:523–554Google Scholar
  10. Barabás G, D’Andrea R, Ostling A (2013) Species packing in nonsmooth competition models. Theor Ecol 6:1–19CrossRefGoogle Scholar
  11. Binmore K (2007) Does game theory work: the bargaining challenge. MIT Press, Cambridge, MAzbMATHCrossRefGoogle Scholar
  12. Bishop DT, Cannings C (1976) Models of animal conflict. Adv Appl Probab 8:616–621zbMATHCrossRefGoogle Scholar
  13. Bomze IM (1991) Cross entropy minimization in uninvadable states of complex populations. J Math Biol 30:73–87MathSciNetzbMATHCrossRefGoogle Scholar
  14. Bomze IM, Pötscher BM (1989) Game theoretical foundations of evolutionary stability. Lecture notes in economics and mathematical systems, vol 324. Springer-Verlag, BerlinzbMATHGoogle Scholar
  15. Broom M, Rychtar J (2013) Game-theoretic models in biology. CRC Press, Boca RatonzbMATHCrossRefGoogle Scholar
  16. Brown GW (1951) Iterative solutions of games by fictitious play. In: Koopmans TC (ed) Activity analysis of production and allocation. Wiley, New York, pp 374–376Google Scholar
  17. Brown JS, Pavlovic NB (1992) Evolution in heterogeneous environments: effects of migration on habitat specialization. Theor Popul Biol 31:140–166CrossRefGoogle Scholar
  18. Brown JS, Vincent TL (1987) Predator-prey coevolution as an evolutionary game. In: Applications of control theory in ecology. Lecture notes in biomathematics, vol 73. Springer, Berlin, pp 83–101Google Scholar
  19. Brown JS, Vincent TL (1992) Organization of predator-prey communities as an evolutionary game. Evolution 46:1269–1283CrossRefGoogle Scholar
  20. Bulmer M (1974) Density-dependent selection and character displacement. Am Nat 108:45–58CrossRefGoogle Scholar
  21. Christiansen FB (1991) On conditions for evolutionary stability for a continuously varying character. Am Nat 138:37–50CrossRefGoogle Scholar
  22. Cohen Y, Vincent TL, Brown JS (1999) A g-function approach to fitness minima, fitness maxima, evolutionary stable strategies and adaptive landscapes. Evol Ecol Res 1:923–942Google Scholar
  23. Courteau J, Lessard S (2000) Optimal sex ratios in structured populations. J Theor Biol 207:159–175CrossRefGoogle Scholar
  24. Cressman R (1992) The stability concept of evolutionary games (a dynamic approach). Lecture notes in biomathematics, vol 94. Springer-Verlag, BerlinzbMATHCrossRefGoogle Scholar
  25. Cressman R (2003) Evolutionary dynamics and extensive form games. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  26. Cressman R (2009) Continuously stable strategies, neighborhood superiority and two-player games with continuous strategy spaces. Int J Game Theory 38:221–247MathSciNetzbMATHCrossRefGoogle Scholar
  27. Cressman R (2010) CSS, NIS and dynamic stability for two-species behavioral models with continuous trait spaces. J Theor Biol 262:80–89MathSciNetCrossRefGoogle Scholar
  28. Cressman R (2011) Beyond the symmetric normal form: extensive form games, asymmetric games and games with continuous strategy spaces. In: Sigmund K (ed) Evolutionary game dynamics. Proceedings of symposia in applied mathematics, vol 69. American Mathematical Society, Providence, pp 27–59CrossRefGoogle Scholar
  29. Cressman R, Hines WGS (1984) Correction to the appendix of ‘three characterizations of population strategy stability’. J Appl Probab 21:213–214MathSciNetCrossRefGoogle Scholar
  30. Cressman R, Hofbauer J (2005) Measure dynamics on a one-dimensional continuous strategy space: theoretical foundations for adaptive dynamics. Theor Popul Biol 67:47–59zbMATHCrossRefGoogle Scholar
  31. Cressman R, Krivan V (2006) Migration dynamics for the ideal free distribution. Am Nat 168:384–397CrossRefGoogle Scholar
  32. Cressman R, Krivan V (2013) Two-patch population podels with adaptive dispersal: the effects of varying dispersal speed. J Math Biol 67:329–358MathSciNetzbMATHCrossRefGoogle Scholar
  33. Cressman R, Tao Y (2014) The replicator equation and other game dynamics. Proc Natl Acad Sci USA 111:10781–10784MathSciNetzbMATHCrossRefGoogle Scholar
  34. Cressman R, Tran T (2015) The ideal free distribution and evolutionary stability in habitat selection games with linear fitness and Allee effect. In: Cojocaru MG et al (eds) Interdisciplinary topics in applied mathematics, modeling and computational sciences. Springer proceedings in mathematics and statistics, vol 117. Springer, New York, pp 457–464zbMATHCrossRefGoogle Scholar
  35. Cressman R, Garay J, Hofbauer J (2001) Evolutionary stability concepts for N-species frequency-dependent interactions. J Theor Biol 211:1–10CrossRefGoogle Scholar
  36. Cressman R, Halloway A, McNickle GG, Apaloo J, Brown J, Vincent TL (2016, preprint) Infinite niche packing in a Lotka-Volterra competition gameGoogle Scholar
  37. Cressman R, Krivan V, Garay J (2004) Ideal free distributions, evolutionary games, and population dynamics in mutiple-species environments. Am Nat 164:473–489Google Scholar
  38. Cressman R, Hofbauer J, Riedel F (2006) Stability of the replicator equation for a single-species with a multi-dimensional continuous trait space. J Theor Biol 239:273–288MathSciNetCrossRefGoogle Scholar
  39. D’Andrea R, Barabás G, Ostling A (2013) Revising the tolerance-fecundity trade-off; or, on the consequences of discontinuous resource use for limiting similarity, species diversity, and trait dispersion. Am Nat 181:E91–E101CrossRefGoogle Scholar
  40. Darwin C (1871) The descent of man and selection in relation to sex. John Murray, LondonCrossRefGoogle Scholar
  41. Dawkins R (1976) The selfish gene. Oxford University Press, OxfordGoogle Scholar
  42. Dercole F, Rinaldi S (2008) Analysis of evolutionary processes. The adaptive dynamics approach and its applications. Princeton University Press, PrincetonzbMATHCrossRefGoogle Scholar
  43. Dieckmann U, Law R (1996) The dynamical theory of coevolution: a derivation from stochastic ecological processes. J Math Biol 34:579–612MathSciNetzbMATHCrossRefGoogle Scholar
  44. Doebeli M, Dieckmann U (2000) Evolutionary branching and sympatric speciation caused by different types of ecological interactions. Am Nat 156:S77–S101CrossRefGoogle Scholar
  45. Eshel I (1983) Evolutionary and continuous stability. J Theor Biol 103:99–111MathSciNetCrossRefGoogle Scholar
  46. Eshel I, Motro U (1981) Kin selection and strong evolutionary stability of mutual help. Theor Popul Biol 19:420–433MathSciNetzbMATHCrossRefGoogle Scholar
  47. Fisher RA (1930) The genetical theory of natural selection. Clarendon Press, OxfordzbMATHCrossRefGoogle Scholar
  48. Fretwell DS, Lucas HL (1969) On territorial behavior and other factors influencing habitat distribution in birds. Acta Biotheor 19:16–32CrossRefGoogle Scholar
  49. Fudenberg D, Levine DK (1998) The theory of learning in games. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  50. Geritz SAH, Kisdi É, Meszéna G, Metz JAJ (1998) Evolutionarily singular strategies and the adaptive growth and branching of the evolutionary tree. Evol Ecol 12:35–57CrossRefGoogle Scholar
  51. Gilboa I, Matsui A (1991) Social stability and equilibrium. Econometrica 59:185–198MathSciNetzbMATHCrossRefGoogle Scholar
  52. Gintis H (2000) Game theory evolving. Princeton University Press, PrincetonzbMATHGoogle Scholar
  53. Gyllenberg M, Meszéna G (2005) On the impossibility of coexistence of infinitely many strategies. J Math Biol 50:133–160MathSciNetzbMATHCrossRefGoogle Scholar
  54. Haigh J (1975) Game theory and evolution. Adv Appl Probab 7:8–11zbMATHCrossRefGoogle Scholar
  55. Hamilton WD (1964) The genetical evolution of social behavior I and II. J Theor Biol 7:1–52CrossRefGoogle Scholar
  56. Hamilton WD (1967) Extraordinary sex ratios. Science 156:477–488CrossRefGoogle Scholar
  57. Hines WGS (1980) Three characterizations of population strategy stability. J Appl Probab 17: 333–340MathSciNetzbMATHCrossRefGoogle Scholar
  58. Hines WGS (1987) Evolutionarily stable strategies: a review of basic theory. Theor Popul Biol 31:195–272zbMATHCrossRefGoogle Scholar
  59. Hofbauer J (1995, preprint) Stability for the best response dynamicsGoogle Scholar
  60. Hofbauer J, Sigmund K (1988) The theory of evolution and dynamical systems. Cambridge University Press, CambridgezbMATHGoogle Scholar
  61. Hofbauer J, Sigmund K (1990) Adaptive dynamics and evolutionary stability. Appl Math Lett 3:75–79MathSciNetzbMATHCrossRefGoogle Scholar
  62. Hofbauer J, Sigmund K (1998) Evolutionary games and population dynamics. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  63. Hofbauer J, Sigmund K (2003) Evolutionary game dynamics. Bull Am Math Soc 40:479–519MathSciNetzbMATHCrossRefGoogle Scholar
  64. Hofbauer J, Schuster P, Sigmund K (1979) A note on evolutionarily stable strategies and game dynamics. J Theor Biol 81:609–612CrossRefGoogle Scholar
  65. Kisdi É, Meszéna G (1995) Life histories with lottery competition in a stochastic environment: ESSs which do not prevail. Theor Popul Biol 47:191–211zbMATHCrossRefGoogle Scholar
  66. Krivan V (2014) The Allee-type ideal free distribution. J Math Biol 69:1497–1513MathSciNetzbMATHCrossRefGoogle Scholar
  67. Krivan V, Cressman R, Schneider C (2008) The ideal free distribution: a review and synthesis of the game theoretic perspective. Theor Popul Biol 73:403–425zbMATHCrossRefGoogle Scholar
  68. Kuhn H (1953) Extensive games and the problem of information. In: Kuhn H, Tucker A (eds) Contributions to the thoery of games II. Annals of mathematics, vol 28. Princeton University Press, Princeton, pp 193–216Google Scholar
  69. Leimar O (2001) Evolutionary change and Darwinian demons. Selection 2:65–72CrossRefGoogle Scholar
  70. Leimar O (2009) Multidimensional convergence stability. Evol Ecol Res 11:191–208Google Scholar
  71. Lessard S (1990) Evolutionary stability: one concept, several meanings. Theor Popul Biol 37: 159–170MathSciNetzbMATHCrossRefGoogle Scholar
  72. Marrow P, Law R, Cannings C (1992) The coevolution of predator–prey interactions: ESSs and Red Queen dynamics. Proc R Soc Lond B 250:133–141CrossRefGoogle Scholar
  73. Matsui A (1992) Best response dynamics and socially stable strategies. J Econ Theory 57:343–362MathSciNetzbMATHCrossRefGoogle Scholar
  74. Maynard Smith J (1974) The theory of games and the evolution of animal conflicts. J Theor Biol 47:209–221MathSciNetCrossRefGoogle Scholar
  75. Maynard Smith J (1982) Evolution and the theory of games. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  76. Maynard Smith J, Price G(1973) The logic of animal conflicts. Nature 246:15–18zbMATHCrossRefGoogle Scholar
  77. McKelvey R, Apaloo J (1995) The structure and evolution of competition-organized ecological communities. Rocky Mt J Math 25:417–436MathSciNetzbMATHCrossRefGoogle Scholar
  78. Mesterton-Gibbons M (2000) An introduction to game-theoretic modelling, 2nd edn. American Mathematical Society, ProvidencezbMATHGoogle Scholar
  79. Meszéna G, Kisdi E, Dieckmann U, Geritz SAH, Metz JAJ (2001) Evolutionary optimisation models and matrix games in the unified perspective of adaptive dynamics. Selection 2:193–210CrossRefGoogle Scholar
  80. Meszéna G, Gyllenberg M, Pásztor L, Metz JAJ (2006) Competitive exclusion and limiting similarity: a unified theory. Theor Popul Biol 69:69–87zbMATHCrossRefGoogle Scholar
  81. Morris D (2002) Measure the Allee effect: positive density effect in small mammals. Ecology 83:14–20CrossRefGoogle Scholar
  82. Nagylaki T (1992) Introduction to theoretical population genetics. Biomathematics, vol 21. Springer-Verlag, BerlinGoogle Scholar
  83. Nash J (1950) Equilibrium points in n-person games. Proc Natl Acad Sci USA 36:48–49MathSciNetzbMATHCrossRefGoogle Scholar
  84. Nash J (1951) Non-cooperative games. Ann Math 54:286–295MathSciNetzbMATHCrossRefGoogle Scholar
  85. Nowak MA (2006) Evolutionary dynamics. Harvard University Press, Cambridge, MAzbMATHGoogle Scholar
  86. Oechssler J, Riedel F (2001) Evolutionary dynamics on infinite strategy spaces. Econ Theory 17:141–162MathSciNetzbMATHCrossRefGoogle Scholar
  87. Parvinen K, Meszéna G (2009) Disturbance-generated niche-segregation in a structured metapopulation model. Evol Ecol Res 11:651–666Google Scholar
  88. Pintor LM, Brown JS, Vincent TL (2011) Evolutionary game theory as a framework for studying biological invasions. Am Nat 177:410–423CrossRefGoogle Scholar
  89. Ripa J, Storling L, Lundberg P, Brown JS (2009) Niche co-evolution in consumer-resource communities. Evol Ecol Res 11:305–323Google Scholar
  90. Robinson J (1951) An iterative method of solving a game. Ann Math 54:296–301, 333–340MathSciNetGoogle Scholar
  91. Roughgarden J (1979) Theory of population genetics and evolutionary ecology: an introduction. Macmillan, New YorkGoogle Scholar
  92. Samuelson L (1997) Evolutionary games and equilibrium selection. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  93. Samuelson L, Zhang J (1992) Evolutionary stability in asymmetric games. J Econ Theory 57: 363–391MathSciNetzbMATHCrossRefGoogle Scholar
  94. Sandholm WH (2010) Population games and evolutionary dynamics. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  95. Sasaki A (1997) Clumped distribution by neighborhood competition. J Theor Biol 186: 415–430CrossRefGoogle Scholar
  96. Sasaki A, Ellner S (1995) The evolutionarily stable phenotype distribution in a random environment. Evolution 49:337–350CrossRefGoogle Scholar
  97. Schlag KH (1997) Why imitate, and if so, how? A boundedly rational approach to multi-armed bandits. J Econ Theory 78:133–156MathSciNetGoogle Scholar
  98. Schuster P, Sigmund K (1983) Replicator dynamics. J Theor Biol 100:533–538MathSciNetCrossRefGoogle Scholar
  99. Selten R (1978) The chain-store paradox. Theory Decis 9:127–159MathSciNetzbMATHCrossRefGoogle Scholar
  100. Selten R (1980) A note on evolutionarily stable strategies in asymmetrical animal contests. J Theor Biol 84:93–101MathSciNetCrossRefGoogle Scholar
  101. Selten R (1983) Evolutionary stability in extensive two-person games. Math Soc Sci 5:269–363MathSciNetzbMATHCrossRefGoogle Scholar
  102. Selten R (1988) Evolutionary stability in extensive two-person games: corrections and further development. Math Soc Sci 16:223–266MathSciNetzbMATHCrossRefGoogle Scholar
  103. Sigmund K (1993) Games of life. Oxford University Press, OxfordGoogle Scholar
  104. Sigmund K (ed) (2011) Evolutionary game dynamics. Proceedings of symposia in applied mathematics, vol 69. American Mathematical Society, ProvidenceGoogle Scholar
  105. Szabó P, Meszéna G (2006) Limiting similarity revisited. Oikos 112:612–619CrossRefGoogle Scholar
  106. Taylor PD (1989) Evolutionary stability in one-parameter models under weak selection. Theor Popul Biol 36:125–143MathSciNetzbMATHCrossRefGoogle Scholar
  107. Taylor PD, Jonker L (1978) Evolutionarily stable strategies and game dynamics. Math Biosci 40:145–156MathSciNetzbMATHCrossRefGoogle Scholar
  108. van Damme E (1991) Stability and perfection of Nash equilibria, 2nd edn. Springer-Verlag, BerlinzbMATHCrossRefGoogle Scholar
  109. Vega-Redondo F (1996) Evolution, games, and economic behavior. Oxford University Press, OxfordzbMATHCrossRefGoogle Scholar
  110. Vincent TL, Brown JS (2005) Evolutionary game theory, natural selection, and Darwinian dynamics. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  111. Vincent TL, Cohen Y, Brown JS (1993) Evolution via strategy dynamics. Theor Popul Biol 44: 149–176zbMATHCrossRefGoogle Scholar
  112. von Neumann J, Morgenstern O (1944) Theory of games and economic behavior. Princeton University Press, PrincetonzbMATHGoogle Scholar
  113. Weibull J (1995) Evolutionary game theory. MIT Press, Cambridge, MAzbMATHGoogle Scholar
  114. Xu Z (2016) Convergence of best-response dynamics in extensive-form games. J Econ Theory 162:21–54MathSciNetzbMATHCrossRefGoogle Scholar
  115. Young P (1998) Individual strategy and social structure. Princeton University Press, PrincetonGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of MathematicsWilfrid Laurier UniversityWaterlooCanada
  2. 2.Department of Mathematics, Statistics and Computer ScienceSt Francis Xavier UniversityAntigonishCanada

Personalised recommendations