Stochastic Games with Endogenous Transitions

  • Reinoud JoostenEmail author
  • Robin Meijboom
Part of the Indian Statistical Institute Series book series (INSIS)


We present and analyze a stochastic game in which transition probabilities between states are not fixed as in standard stochastic games, but depend on the history of the play, i.e., the players’ past action choices. For the limiting average reward criterion we determine the set of jointly convergent pure-strategy rewards which can be supported by equilibria involving threats. For expository purposes we analyze a stylized fishery game. Each period, two agents choose between catching with restraint or without. The resource is in either of two states, High or Low. Restraint is harmless to the fish, but it is a dominated action at each stage. The less restraint shown during the play, the higher the probabilities that the system moves to or stays in Low. The latter state may even become “absorbing temporarily,”’ i.e., transition probabilities to High temporarily become zero while transition probabilities to Low remain nonzero.


Stochastic games Limiting average rewards Endogenous transition probabilities Temporarily absorbing states Hysteresis 


  1. 1.
    Amir, R.: Stochastic games in economics and related fields: an overview. In: Neyman, A., Sorin, S. (eds.) Stochastic Games and Applications. NATO Advanced Study Institute, Series D, pp. 455–470. Kluwer, Dordrecht (2003)CrossRefGoogle Scholar
  2. 2.
    Anderson, T., Carstensen, J., Hernández-Garcia, E., Duarte, C.M.: Ecological thresholds and regime shifts: approaches to identification. Trends Ecol. Evol. 24, 49–57 (2008)CrossRefGoogle Scholar
  3. 3.
    Armstrong, M.J., Connolly, P., Nash, R.D.M., Pawson, M.G., Alesworth, E., Coulahan, P.J., Dickey-Collas, M., Milligan, S.P., O’Neill, M., Witthames, P.R., Woolner, L.: An application of the annual egg production method to estimate spawning biomass of cod (Gadus morhua L.), plaice (Pleuronectes platessa L.) and sole (Solea solea L.) in the Irish Sea. ICES J. Mar. Sci. 58, 183–203 (2001)CrossRefGoogle Scholar
  4. 4.
    Aumann, R.: Game engineering. In: Neogy, S.K., Bapat, R.B., Das, A.K., Parthasarathy, T. (eds.) Mathematical Programming and Game Theory for Decision Making, pp. 279–286. World Scientific, Singapore (2008)CrossRefGoogle Scholar
  5. 5.
    BenDor, T., Scheffran, J., Hannon, B.: Ecological and economic sustainability in fishery management: a multi-agent model for understanding competition and cooperation. Ecol. Econ. 68, 1061–1073 (2009)CrossRefGoogle Scholar
  6. 6.
    Bewley, T., Kohlberg, E.: The asymptotic theory of stochastic games. Math Oper Res. 1, 197–208 (1976)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Bewley, T., Kohlberg, E.: The asymptotic solution of a recursive equation occuring in stochastic games. Math. Oper. Res. 1, 321–336 (1976)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Billingsley, P.: Probability and Measure. Wiley, New York (1986)zbMATHGoogle Scholar
  9. 9.
    Blanchard, O., Summers, L.: Hysteresis and the European unemployment problem. In: Fisher, S. (ed.) NBER Macroecon. Annu., pp. 15–78. MIT Press, Cambridge (1986)Google Scholar
  10. 10.
    Brooks, S.E., Reynolds, J.D., Allison, A.E.: Sustained by snakes? seasonal livelihood strategies and resource conservation by Tonle Sap fishers in Cambodia. Hum. Ecol. 36, 835–851 (2008)CrossRefGoogle Scholar
  11. 11.
    Bulte, E.H.: Open access harvesting of wildlife:the poaching pit and conservation of endangered species. Agric. Econ. 28, 27–37 (2003)CrossRefGoogle Scholar
  12. 12.
    Carpenter, S.R., Ludwig, D., Brock, W.A.: Management of eutrophication for lakes subject to potentially irreversible change. Ecol. Appl. 9, 751–771 (1999)CrossRefGoogle Scholar
  13. 13.
    Courchamp, F., Angulo, E., Rivalan, P., Hall, R.J., Signoret, L., Meinard, Y.: Rarity value and species extinction: the anthropogenic Allee effect. PLoS Biol. 4, 2405–2410 (2006)CrossRefGoogle Scholar
  14. 14.
    Cross, J.G., Guyer, M.J.: Social Traps. University of Michigan Press, Ann Arbor (1980)Google Scholar
  15. 15.
    Ehtamo, H., Hämäläinen, R.P.: On affine incentives for dynamic decision problems. In: Başar, T. (ed.) Dynamic Games and Applications in Economics, pp. 47–63. Springer, Berlin (1986)CrossRefGoogle Scholar
  16. 16.
    Ehtamo, H., Hämäläinen, R.P.: Incentive strategies and equilibria for dynamic games with delayed information. JOTA 63, 355–369 (1989)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Ehtamo, H., Hämäläinen, R.P.: A cooperative incentive equilibrium for a resource management problem. J. Econ. Dyn. Control. 17, 659–678 (1993)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Ehtamo, H., Hämäläinen, R.P.: Credibility of linear equilibrium strategies in a discrete-time fishery management game. Group Decis. Negot. 4, 27–37 (1995)CrossRefGoogle Scholar
  19. 19.
    Filar, J., Raghavan, T.E.S.: A matrix game solution to a single-controller stochastic game. Math. Oper. Res. 9, 356–362 (1984)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Filar, J., Vrieze, O.J.: Competitive Markov Decision Processes. Springer, Berlin (1996)CrossRefGoogle Scholar
  21. 21.
    Flesch, J.: Stochastic games with the average reward. Ph.D. thesis, Maastricht University, ISBN 90-9012162-5 (1998)Google Scholar
  22. 22.
    Flesch, J., Schoenmakers, G., Vrieze, O.J.: Loss of skills in coordination games. Int. J. Game Theory 40, 769–789 (2011)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Forges, F.: An approach to communication equilibria. Econometrica 54, 1375–1385 (1986)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Hall, R.J., Milner-Gulland, E.J., Courchamp, F.: Endangering the endangered: the effects of perceived rarity on species exploitation. Conserv. Lett. 1, 75–81 (2008)CrossRefGoogle Scholar
  25. 25.
    Hämäläinen, R.P., Haurie, A., Kaitala, V.: Equilibria and threats in a fishery management game. Optim. Control. Appl. Methods 6, 315–333 (1985)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Hamburger, H.: N-person prisoner’s dilemma. J. Math. Psychol. 3, 27–48 (1973)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Hardin, G.: The tragedy of the commons. Science 162, 1243–1248 (1968)Google Scholar
  28. 28.
    Hart, S.: Nonzero-sum two-person repeated games with incomplete information. Math. Oper. Res. 10, 117–153 (1985)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Haurie, A., Krawczyk, J.B., Zaccour, G.: Games and Dynamic Games. World Scientific, Singapore (2012)CrossRefGoogle Scholar
  30. 30.
    Heckathorn, D.D.: The dynamics and dilemmas of collective action. Am. Sociol. Rev. 61, 250–277 (1996)CrossRefGoogle Scholar
  31. 31.
    Herings P.J.J., Predtetchinski, A.: Voting in collective stopping games, working paper Maastricht University (2012)Google Scholar
  32. 32.
    Hillis, J.F., Wheelan, J.: Fisherman’s time discounting rates and other factors to be taken into account in planning rehabilitation of depleted fisheries. In: Antona, M., et al. (eds.) Proceedings of the 6th Conference of the International Institute of Fisheries Economics Trade, pp. 657–670. IIFET-Secretariat, Paris (1994)Google Scholar
  33. 33.
    Holden, M.: The Common Fisheries Policy: Origin, Evaluation and Future. Fishing News Books, Blackwell (1994)Google Scholar
  34. 34.
    Hordijk, A., Vrieze, O.J., Wanrooij, L.: Semi-Markov strategies in stochastic games. Int. J. Game Theory 12, 81–89 (1983)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Joosten, R.: Dynamics, Equilibria, and Values. Ph.D. thesis, Faculty of Economics and Business Administration, Maastricht University (1996)Google Scholar
  36. 36.
    Joosten, R.: A note on repeated games with vanishing actions. Int. Game Theory Rev. 7, 107–115 (2005)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Joosten, R.: Small Fish Wars: a new class of dynamic fishery-management games. ICFAI J. Manag. Econ. 5, 17–30 (2007a)Google Scholar
  38. 38.
    Joosten, R.: Small Fish Wars and an authority. In: Prinz, A. (ed.) The Rules of the Game: Institutions, Law, and Economics, pp. 131–162. LIT, Berlin (2007)Google Scholar
  39. 39.
    Joosten, R.: Strategic advertisement with externalities: a new dynamic approach. In: Neogy, S.K., Das, A.K., Bapat, R.B. (eds.) Modeling, Computation and Optimization. ISI Platinum Jubilee Series, vol. 6, pp. 21–43. World Scientific Publishing Company, Singapore (2009)CrossRefGoogle Scholar
  40. 40.
    Joosten, R.: Long-run strategic advertisement and short-run Bertrand competition. Int. Game Theory Rev. 17, 1540014 (2015). Scholar
  41. 41.
    Joosten, R.: Strong and weak rarity value: resource games with complex price-scarcity relationships. Dyn. Games Appl. 16, 97–111 (2016)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Joosten, R., Brenner, T., Witt, U.: Games with frequency-dependent stage payoffs. Int. J. Game Theory 31, 609–620 (2003)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Joosten, R., Samuel, L.: On stochastic fishery games with endogenous stage payoffs and transition probabilities. In: Proceedings of 3rd Joint Chinese-Dutch Workshop on Game Theory and Applications and 7th China Meeting on Game Theory and Applications. CCIS-series. Springer, Berlin (2017)CrossRefGoogle Scholar
  44. 44.
    Joosten, R., Samuel, L.: On the computation of large sets of rewards in ETP-ESP-games with communicating states. Research memorandum, Twente University, The Netherlands (2017)Google Scholar
  45. 45.
    Joosten, R., Thuijsman, F., Peters, H.: Unlearning by not doing: repeated games with vanishing actions. Games Econ. Behav. 9, 1–7 (1993)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Kelly, C.J., Codling, E.A., Rogan, E.: The Irish Sea cod recovery plan: some lessons learned. ICES J. Mar. Sci. 63, 600–610 (2006)CrossRefGoogle Scholar
  47. 47.
    Komorita, S.S., Parks, C.D.: Social Dilemmas. Westview Press, Boulder (1996)Google Scholar
  48. 48.
    Krawczyk, J.B., Tołwinski, B.: A cooperative solution for the three nation problem of exploitation of the southern bluefin tuna. IMA J. Math. Appl. Med. Biol. 10, 135–147 (1993)CrossRefGoogle Scholar
  49. 49.
    Lenton, T.M., Livina, V.N., Dakos, V., Scheffer, M.: Climate bifurcation during the last deglaciation? Clim. Past 8, 1127–1139 (2012)CrossRefGoogle Scholar
  50. 50.
    Levhari, D., Mirman, L.: The great fish war: an example using a dynamic Cournot-Nash solution. Bell J. Econ. 11, 322–334 (1980)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Long, N.V.: A Survey of Dynamic Games in Economics. World Scientific, Singapore (2010)CrossRefGoogle Scholar
  52. 52.
    Mäler, K.-G., Xepapadeas, A., de Zeeuw, A.: The economics of shallow lakes. Environ. Resour. Econ. 26, 603–624 (2003)CrossRefGoogle Scholar
  53. 53.
    Marwell, G., Oliver, P.: The Critical Mass in Collective Action: A Micro-Social Theory. Cambridge University Press, Cambridge (1993)CrossRefGoogle Scholar
  54. 54.
    Messick, D.M., Brewer, M.B.: Solving social dilemmas: a review. Annu. Rev Pers. Soc. Psychol. 4, 11–43 (1983)Google Scholar
  55. 55.
    Messick, D.M., Wilke, H., Brewer, M.B., Kramer, P.M., Zemke, P.E., Lui, L.: Individual adaptation and structural change as solutions to social dilemmas. J Pers. Soc. Psychol. 44, 294–309 (1983)CrossRefGoogle Scholar
  56. 56.
    Mertens, J.F., Neyman, A.: Stochastic games. Int. J. Game Theory. 10, 53–66 (1981)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Oosthuizen, E., Daan, N.: Egg fecundity and maturity of North Sea cod, gadus morhua. Neth. J. Sea Res. 8, 378–397 (1974)CrossRefGoogle Scholar
  58. 58.
    Ostrom, E.: Governing the Commons. Cambridge University Press, Cambridge (1990)CrossRefGoogle Scholar
  59. 59.
    Ostrom, E., Gardner, R., Walker, J.: Rules, Games, and Common-Pool Resources. Michigan University Press, Ann Arbor (1994)CrossRefGoogle Scholar
  60. 60.
    Parthasarathy, T., Raghavan, T.E.S.: An orderfield property for stochastic games when one player controls the transition probabilities. J. Optim. Theory Appl. 33, 375–392 (1981)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Platt, J.: Social traps. Am. Psychol. 28, 641–651 (1973)CrossRefGoogle Scholar
  62. 62.
    Raghavan, T.E.S., Filar, J.: Algorithms for stochastic games, a survey. Z. Oper. Res. 33, 437–472 (1991)MathSciNetzbMATHGoogle Scholar
  63. 63.
    Rose, G.A., Bradbury, I.R., de Young, B., Fudge, S.B., Lawson, G.L., Mello, L.G.S., Robichaud, D., Sherwood, G., Snelgrove, P.V.R., Windle, M.J.S.: Rebuilding Atlantic Cod: Lessons from a Spawning Ground in Coastal Newfoundland. In: Kruse, G.H., et al. (eds.) 24th Lowell Wakefield Fisheries Symposium on Resiliency of gadid stocks to fishing and climate change, pp. 197–219 (2008)Google Scholar
  64. 64.
    Sanchirico, J.N., Smith, M.D., Lipton, D.W.: An empirical approach to ecosystem-based fishery management. Ecol. Econ. 64, 586–596 (2008)CrossRefGoogle Scholar
  65. 65.
    Scheffer, M.: The Ecology of Shallow Lakes. Chapman & Hall, London (1998)Google Scholar
  66. 66.
    Scheffer, M., Carpenter, S., Foley, J.A., Folke, C., Walker, B.: Catastrophic shifts in ecosystems. Nature 413, 591–596 (2001)CrossRefGoogle Scholar
  67. 67.
    Schoenmakers, G.M.: The profit of skills in repeated and stochastic games. Ph.D. thesis Maastricht University (2004)Google Scholar
  68. 68.
    Schoenmakers, G.M., Flesch, J., Thuijsman, F.: Coordination games with vanishing actions. Int. Game Theory Rev. 4, 119–126 (2002)MathSciNetCrossRefGoogle Scholar
  69. 69.
    Shapley, L.: Stochastic games. Proc. Natl. Acad. Sci. USA 39, 1095–1100 (1953)MathSciNetCrossRefGoogle Scholar
  70. 70.
    Steg, L.: Motives and behavior in social dilemmas relevant to the environment. In: Hendrickx, L., Jager, W., Steg, L. (eds.) Human Decision Making and Environmental Perception. Understanding and Assisting Human Decision Making in Real-Life Settings, pp. 83–102 (2003)Google Scholar
  71. 71.
    Thuijsman, F., Vrieze, O.J.: The power of threats in stochastic games. In: Bardi, M., et al. (eds.) Stochastic and Differential Games, Theory and Numerical Solutions, pp. 343–358. Birkhauser, Boston (1998)zbMATHGoogle Scholar
  72. 72.
    Tołwinski, B.: A concept of cooperative equilibrium for dynamic games. Automatica 18, 431–441 (1982)MathSciNetCrossRefGoogle Scholar
  73. 73.
    Tołwinski, B., Haurie, A., Leitmann, G.: Cooperative equilibria in differential games. JOTA 119, 182–202 (1986)MathSciNetzbMATHGoogle Scholar
  74. 74.
    Van Damme, E.E.C.: Stability and Perfection of Nash Equilibria. Springer, Berlin (1992)zbMATHGoogle Scholar
  75. 75.
    Vrieze, O.J.: Linear programming and undiscounted games in which one player controls transitions. OR Spektrum 3, 29–35 (1981)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.IEBIS, BMSUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations