Population Games and Discrete Optimal Transport

Abstract

We propose an evolutionary dynamics for population games with discrete strategy sets, inspired by optimal transport theory and mean field games. The proposed dynamics is the Smith dynamics with strategy graph structure, in which payoffs are modified by logarithmic terms. The dynamics can be described as a Fokker–Planck equation on a discrete strategy set. For potential games, the dynamics is a gradient flow system under a Riemannian metric from optimal transport theory. The stability of the dynamics is studied through optimal transport metric tensor, free energy and Fisher information.

Appendix

In this section, we briefly review the Best-reply dynamics and its connection with optimal transport theory. These serve the motivations of the dynamics considered in this paper. For more details see Degond et al. (2014), Villani (2008).

Best-reply dynamics and Fokker–Planck equations We first consider a game consisting N players $i\in \{1,\ldots ,N\}$. Each player i chooses a strategy $x_i$ from a same Borel strategy set S. For concreteness, we consider $S=\mathbb {T}^d$, which is a d dimensional torus. Suppose each player receives a payoff function $F_i\in C^{\infty }(S)$. For notational connivence, we denote $F_i(x_i,x_{-i})=F_i(x_1,\ldots , x_N)$, where we abuse the notation by

$$\begin{aligned} x_{-i}=\{x_1,\ldots , x_{i-1}, x_{i+1},\ldots , x_N\}\ . \end{aligned}$$

We model players’ decision-making processes in a game by stochastic process $x_{i}(t),~t\in [0,+\infty )$. Here, t is an artificial time variable, at which player i selects his/her decision based on the current strategies of all other players $x_{-i}(t)$. We note that all players make their decisions simultaneously and without knowing others’ decisions. Each player selects his or her strategy that increases the player’s payoff most rapidly. In other words, we model the game by the following stochastic differential equations (SDEs)

$$\begin{aligned} \hbox {d} x_i= \nabla _{x_i}F_i (x_i, x_{-i})\hbox {d}t + \sqrt{2\beta } dB_{t}^{i}\ , \end{aligned}$$

(16)

where the independent Brownian motion $(B_t^i)_{i=1}^N$ is added to model the uncertainty of each player and $\beta >0$ controls the magnitude of the noise.

Under the standard assumptions in population games, i.e., the game is autonomous and the players are symmetric, one can simply encode all the information of players into one probability density $\rho \in \mathcal {P}(S)$ by taking $N\rightarrow \infty$. In this limiting processes, each player’s cost function is rewritten as $F:S\times \mathcal {P}(S)\rightarrow \mathbb {R}$, and the limiting stochastic process forms the following mean field SDE

$$\begin{aligned} \hbox {d}X_t= \nabla _{X_t}F (X_t, \rho )\hbox {d}t + \sqrt{2\beta } dB_{t}^{i}\ , \end{aligned}$$

(17)

where $X_t$ has probability law $\rho (t,x)$.

In Degond et al. (2014), SDE (17) is called the Best-reply dynamics, and $X_t$ is the Best-reply decision process. Here, the transition density function $\rho (t,x)$ of the stochastic process X(t) satisfies the FPE

$$\begin{aligned} \frac{\partial \rho (t,x)}{\partial t}=-\nabla \cdot (\rho (t,x)F(x,\rho ))+\beta \Delta \rho (t,x)\ . \end{aligned}$$

(18)

The game is called a potential game if there exists a potential function $\mathcal {F}:\mathcal {P}(S)\rightarrow \mathbb {R}$, such that $\frac{\delta }{\delta \rho (x)}\mathcal {F}(\rho )=F(x,\rho )$. For potential games, the Best-reply SDE (17) becomes

$$\begin{aligned} \hbox {d}X_t=\nabla \frac{\delta }{\delta \rho (t, X_t)}\mathcal {F}(\rho ) \hbox {d}t+\sqrt{2\beta } dB_t\ , \end{aligned}$$

which is a perturbed gradient flow and whose transition equation (FPE) forms

$$\begin{aligned} \frac{\partial \rho (t,x)}{\partial t}=-\nabla \cdot (\rho (t,x)\nabla \frac{\delta }{\delta \rho (t,x)}\mathcal {F}(\rho ))+\beta \Delta \rho (t,x)\ . \end{aligned}$$

(19)

From the theory of optimal transport, Equation (19) can be interpreted as a gradient ascend flow of the free energy

$$\begin{aligned} \mathcal {\bar{F}}(\rho )=\mathcal {F}(\rho )-\beta \int _{S}\rho (x)\log \rho (x)\hbox {d}x\ . \end{aligned}$$

(20)

Optimal transport and density manifold We next review the geometry of optimal transport on the continuous strategy set S.

Consider the set $\mathcal {P}_2(S)$ of Borel measurable probability density functions on S with finite second moment. Given $\rho ^0, \rho ^1\in \mathcal {P}_2(S)$, the $L^2$-Wasserstein distance between $\rho ^0$ and $\rho ^1$ is denoted by $W:\mathcal {P}_2(S)\times \mathcal {P}_2(S)\rightarrow \mathbb {R}_+$. There are two equivalent ways of defining this distance.

The first definition is the following linear programming formulation:

$$\begin{aligned} W(\rho ^0, \rho ^1)^2=\inf _{\pi \in \Pi (\rho ^0, \rho ^1)}\int _{\Omega \times \Omega }d_\Omega (x,y)^2\pi (\hbox {d}x,\hbox {d}y)\ , \end{aligned}$$

(21)

where the infimum is taken over the set $\Pi$ of joint probability measures on $\Omega \times \Omega$ that have marginals $\rho ^0$, $\rho ^1$.

The second definition considers a probability path $\rho :[0,1]\rightarrow \mathcal {P}_2(S)$ connecting $\rho ^0$, $\rho ^1$. And the distance is defined by a variational problem known as the Benamou–Brenier formula:

$$\begin{aligned} W(\rho ^0, \rho ^1)^2=\inf _{\Phi }~\int _0^1\int _{\Omega } (\nabla \Phi (t,x), \nabla \Phi (t,x))\rho (t,x) \hbox {d}x \hbox {d}t\ , \end{aligned}$$

(22a)

where the infimum is taken over the set of Borel potential functions $[0,1]\times S \rightarrow \mathbb {R}$. Each potential function $\Phi$ determines a corresponding density path $\rho$ as the solution of the continuity equation

$$\begin{aligned} \frac{\partial \rho (t,x)}{\partial t}+\text {div} (\rho (t,x)\nabla \Phi (t,x))=0\ ,\quad \rho (0,x)=\rho ^0(x)\ ,\quad \rho (1,x)=\rho ^1(x)\ . \end{aligned}$$

(22b)

Here, $\text {div}$ and $\nabla$ are the divergence and gradient operators in $\Omega$. The continuity equation is known as the probability density transition equation according to the given vector field.

The equivalence between the static (21) and dynamical (22) formulations is well known. Moreover, the variational formulation (22) entails a similar Riemannian structure used in this paper. For simplicity, we only consider the set of smooth and strictly positive probability densities

$$\begin{aligned} \mathcal {P}_+(S)=\Big \{\rho \in C^{\infty }(\Omega ):\rho (x)>0\ ,~\int _{\Omega }\rho (x)\hbox {d}x=1\Big \} \subset \mathcal {P}_2(S)\ . \end{aligned}$$

Denote $\mathcal {F}(S):=C^{\infty }(S)$ the set of smooth real valued functions on S. The tangent space of $\mathcal {P}_+(S)$ is given by

$$\begin{aligned} T_\rho \mathcal {P}_+(S) = \Big \{\sigma \in \mathcal {F}(S):\int _{S}\sigma (x) \hbox {d}x=0 \Big \}\ . \end{aligned}$$

Given $\Phi \in \mathcal {F}(S)$ and $\rho \in \mathcal {P}_+(S)$, define

$$\begin{aligned} V_{\Phi }(x):=-\text {div} (\rho (x) \nabla \Phi (x))\ . \end{aligned}$$

Thus, $V_\Phi \in T_{\rho }\mathcal {P}_+(S)$. The elliptic operator $\nabla \cdot (\rho \nabla )$ identifies the function $\Phi$ on S modulo additive constants with the tangent vector $V_{\Phi }$ of the space of densities. This gives an isomorphism

$$\begin{aligned} \mathcal {F}(S)/\mathbb {R}\rightarrow T_{\rho }\mathcal {P}_+(S); \quad \Phi \mapsto V_\Phi \ . \end{aligned}$$

Define the Riemannian metric (inner product) on the tangent space of positive densities $g^W:{T_\rho }\mathcal {P}_+(S)\times {T_\rho }\mathcal {P}_+(S)\rightarrow \mathbb {R}$ by

$$\begin{aligned} g^W_\rho (V_{\Phi }, V_{\tilde{\Phi }})=\int _{S}(\nabla \Phi (x), \nabla \tilde{\Phi }(x))\rho (x) \hbox {d}x\ , \end{aligned}$$

where $\Phi (x)$, $\tilde{\Phi }(x)\in \mathcal {F}(S)/\mathbb {R}$. The inner product endows $\mathcal {P}_+(S)$ with an infinite-dimensional Riemannian metric tensor. In other words, the variational problem (22) is a geometric action energy in $(\mathcal {P}_+(S), g^W)$.

We are now ready to present the gradient operator of free energy w.r.t. $L^2$-Wasserstein metric tensor. Following

$$\begin{aligned} g^W(\text {grad}_W\mathcal {\bar{F}}(\rho ), V_{\Phi })=\int _{S}\frac{\delta }{\delta \rho (x)}\mathcal {\bar{F}}(\rho )V_{\Phi }\hbox {d}x\ \end{aligned}$$

and $\frac{\delta }{\delta \rho (x)}\mathcal {F}(\rho )=F(x,\rho )$, and noticing $\frac{\delta }{\delta \rho (x)}\int _{S}\rho (x)\log \rho (x)\hbox {d}x=\log \rho (x)+1$, we obtain

$$\begin{aligned} \text {grad}_W\mathcal {\bar{F}}(\rho )=-\nabla \cdot (\rho \nabla (F(x,\rho )-\beta \log \rho (x)))\ . \end{aligned}$$

From the fact that $\nabla \cdot (\rho \nabla \log \rho )=\nabla \cdot (\nabla \rho )=\Delta \rho$, we derive FPE (19) by the gradient flow of the free energy

$$\begin{aligned} \frac{\partial \rho }{\partial t}=\text {grad}_W\mathcal {F}(\rho )=-\nabla \cdot (\rho \nabla F(x,\rho ))+\beta \Delta \rho . \end{aligned}$$