Evolutionary game dynamics of cooperation in prisoner's dilemma with time delay

1.   Introduction

Evolutionary game theory studies populations where individuals can use different strategies, which are subsequently interpreted as different phenotypes ^[1,2,3]. Replicator dynamics are among the core dynamics used to describe the evolution of the frequencies of the strategies in EGT. It shows how populations allocate the different pure strategies that are related in a game over time.

Cooperation is of great significance to the sustainable development of the population in evolutionary game theory ^[4,5]. How to promote cooperation is the core issue of evolutionary game theory, because when a player chooses to defect, the average fitness is lower than when the player decides to cooperate, which results in a social dilemma. For the stag hunt game, Zhang et al. ^[6] studied the cooperative behavior and various stages in the coevolutionary network dynamics. They found that the status of cooperators can be changed by controlling the payoff parameter $r$ and reconnection probability $q$. Du and Wu ^[7] studied the evolutionary dynamics of cooperation based on the co-evolution of strategy and network structure. When the cooperators are active to a certain extent, the cooperation strategy will emerge and remain stable. In an $N$-player iterative prisoner's dilemma game, the random mobility can improve cooperation and was explored by Chiong and Kirley ^[8]. Chen et al. ^[9] proposed a mechanism to promote cooperation on the lattice. This mechanism allowed players to decide whether to keep or delete neighbors by comparing profits. It proved that under this mechanism, if the cost-benefit ratio is small or the temptation of betrayal is small, the level of cooperation would be improved. These studies examined the changes of cooperative strategy in games through different forms.

In addition, abundant studies have shown that imposing a penalty on defectors can promote cooperative behavior. For example, based on the $N$-player snowdrift game with peer punishment, Pi et al. ^[10] assumed that the non-cooperators in the well-mixed population have an individual disguise, and they found that the high cost of disguise and severe punishment inhibit the existence of non-cooperators. Zhu et al. ^[11] investigated peer punishment and pool punishment together in the spatial public goods game. The influence of penalty-type transfer on evolutionary dynamics was fully analyzed. They showed that peer punishment is more efficacious than collective punishment in promoting players' cooperation. In addition, in a finite population, Catalán et al. ^[12] assumed that the offspring of players inherit their parents' strategy and can mutate to another strategy. They studied the influence of mutation and selection in the Hawk-dove game with mixed strategies. In^[13], Nagatani et al. proposed a metapopulation model of rock paper scissors game under the influence of mutation. The change in mutation rate would cause the dynamic phase transition to occur in three stages: the stable coexistence of three species, the stable phase of two species, and a single-species phase. This study imposes a penalty when both players choose the defective strategy to encourage players to cooperate, and the paper discusses the impact of the mutation.

Time delay is widely used in biological systems and often leads to bifurcation. It is also essential to discuss time delay in evolutionary game theory. For example, in Wettergren (2021) ^[14], based on replicator dynamics, a snowdrift game model for $N$ players was constructed. It was found that delay leads to Hopf bifurcation. As the time delay increases, when it is larger than the critical delay, it presents oscillatory replicator dynamics rather than asymptotic stability. Alboszta and Miekisz ^[15] considered the replicator dynamics models with social and biological delays. When the social delay model is considered, the small delay leads to the asymptotic stability of dynamics. For biological time delay, the dynamics are asymptotically stable for any time delay. Miekisz and Wesołowski ^[16] discussed the joint influence of stochasticity and time delay. When an individual renewal strategy is randomly selected, the time delay has little influence on dynamics. Burridge et al. ^[17] showed that memory affects dynamics stability. When the game is stable, long memory is beneficial, but it is different when the game is unstable. Mittal et al. ^[18] found that mutation does not lead to oscillation of cooperative state, but the delayed information of population state may lead to oscillation. Hu and Qiu ^[19] studied stochastic delay and fixed delay, in which players in different communities have different delays. However, the impact of mutation time delay has not been fully discussed. We consider that it takes time for players to mutate to another strategy. We study the effects of double delays, i.e., payoff delay and mutation delay, on dynamics.

The rest of this paper is organized as follows. First, Section 2 considers increasing the penalty for the prisoner's dilemma without mutation. We obtain the stable equilibrium by the replicator equation and prove its stability. The co-existence of cooperators and defectors has emerged. In addition, the change of cooperator ratio under the influence of time delay is studied, and we obtain the critical time delay from stable equilibrium to oscillation. In Section 3, we study the prisoner's dilemma with penalty and mutation and discuss the effects of the time delay of payoff and the time delay of mutation on the game. Section 4 is the conclusion.

2.   The 2-player prisoner's dilemma with penalty

In this paper, we choose the payoff matrix of the prisoner's dilemma

where $b-c > 0$, and $a$, $b$, $c$ are positive constants. When both players cooperate, they get $b-c$. When one player cooperates, and the other player defects, the cooperative player bears a cost $c$. When a player who defects meets a player who cooperates, he immediately gets a payoff of $b$. When both players defect, they get nothing. At the same time, the classic prisoner's dilemma takes defection as the dominant strategy. To improve players' enthusiasm for cooperation, when two defection strategies meet, a penalty $a$ $(a > 0)$ is imposed, and the payoff matrix is given by

We see that when $a < c$, it is still the defection strategy that dominates, so assume that $a > c$ and see what happens.

In a well-mixed infinite population, $x(t)$ represents the proportion of people who choose to cooperate (that is, to execute strategy $C$) at time $t$, and $1-x(t)$ represents the proportion of people who choose to defect (that is, to execute strategy $D$) at time $t$. The replicator dynamics' equilibria correspond to the game's optional strategies, so we next study the replicator dynamics of the prisoner's dilemma.

2.1.   Model with penalty and no time delay

If time delay is not considered, the expected average payoff of participants who choose to cooperate is

The expected average payoff for the participants who defect is

According to the replicator equation $\dot{x }(t) = x\left (t\right)\left (1-x\left (t\right)\right)\left (\pi _{c}-\pi _{d}\right)$, and Eqs (2.3), (2.4), the dynamics for the 2-player prisoner's dilemma with penalty and no time delay is

2.2.   Result with penalty and no time delay

Equation (2.5) has three real equilibria ${x_{1}, x_{2}, x_{3}}$, given by $x_{1} = 0$, $x_{2} = 1$, $x_{3} = \frac{a-c}{a}$. The $x_{3}$ only makes sense if $a > c$ and $c > 0$, where $0 < x_{3} < 1$.

Lemma 2.1. When $\dot{x} = f\left (x\right)$, $f\left (x^{\ast }\right) = 0$. The internal equilibrium point $x^{\ast }$ is a stable equilibrium point if $f^{'}\left (x^{\ast }\right) < 0$.

Let $f\left (x\right) = x\left (t\right)\left (1-x\left (t\right)\right)\left [a-c-ax\left (t\right)\right]$, $f^{'}\left (0\right) = -c+a > 0$, and $f^{'}\left (1\right) = c > 0$. It follows from the stability test that the solutions at $x_{1} = 0$ and $x_{2} = 1$ are unstable, while $f^{'}\left (\frac{a-c}{a}\right) = -\frac{\left (a-c\right)c}{a} < 0$, so the solution at $x_{3}$ is stable. Thus, in a 2-player prisoner's dilemma with no delay and a penalty, the proportion of cooperators eventually tends to $x_{3} = \frac{a-c}{a}$ regardless of the initial values of cooperators and defectors. Thus, $x_{3}$ is called the stable equilibrium, denoted by

From this, we find that when the penalty for defection increases, the proportion of players choosing to cooperate increases, which means that players tend to cooperate to avoid the penalty for defection. Figure 1 shows the proportion of cooperators and defectors over time without the penalty for defection. We can see that no matter what the initial value is, the proportion of cooperators tends to $0$, and the proportion of defectors tends to $1$. That means the cooperators will disappear, and all that remains are the defectors. However, when the defection penalty is greater than the cheated payoff $(a > c)$, cooperators and defectors coexist regardless of the initial values, and the proportion of cooperators is stable at $x_{3}$, as shown in Figure 2.

According to ^{[20,21,22,23,24]}, we can find that Figure 2 is satisfied with the coexistence of the snowdrift game, which changes the intensity of social dilemma and ensures the existence of cooperation. When the benefits generated by the cooperation of the prisoner's dilemma are available to both players, and the cooperators share the costs, this leads to the so-called snowdrift game ^[25]. In addition, when sufficient penalty $(a > c)$ is added to the defectors in the prisoner's dilemma, the conditions of snowdrift game are also met, which breaks the dominance of strategy $D$ and ensures the coexistence of strategy $C$ and strategy $D$.

2.3.   Model with penalty and time delay

We next study the game with time delay, considering that the expected payoffs $\pi _{c}^{d}(t)$ and $\pi _{c}^{d}(t)$ depend on the payoffs of the players at the previous time $(t-\tau)$ (the superscript $d$ denotes payoffs with time delay). The expected average payoffs of the cooperator and the defector with time delay are, respectively,

The replicator equation of the 2-player prisoner's dilemma with time delay and penalty is

2.4.   Result with penalty and time delay

Next, we find out the critical delay of Hopf bifurcation through characteristic equation analysis. Let $\xi \left (t\right) = x\left (t\right)-x^{\ast }$, $\xi ^{d} \left (t\right) = x\left (t-\tau \right)-x^{\ast }$, and replace $\xi \left (t\right)$ and $\xi ^{d} \left (t\right)$ with $x\left (t\right)$ and $x\left (t-\tau \right)$. Take their linearization and keep only the linear term, and get

Let $\xi \left (t\right) = e^{\lambda t}$, and let $\xi ^{d} \left (t\right) = e^{\lambda\left (t-\tau \right)}$. The characteristic equation of (2.9) is

Let $\lambda = i\omega$ ($\omega > 0$), and we have

The real and the imaginary parts are separated, and we get

Squaring both sides of (2.15) and adding them together, we get

Substituting (2.16) into (2.15), we obtain

where $\sigma = \frac{c}{a}$.

According to Eq (2.17), the critical delay $\tau _{c}$ decreases as the defection penalty $a$ increases. When $\tau < \tau _{c}$, we have $x(t)\rightarrow x^{\ast }$, which means the proportion of cooperators is stable at the equilibrium. When $\tau > \tau _{c}$, the phenomenon of periodic oscillation occurs.

Figure 3 shows how the equilibrium $x_{3}$ changes as time increases for different initial values and different time delays. We find that when the time delay is slight ($\tau = 1$), it presents a stable dynamic performance with the time change. When the delay is significant ($\tau = 2$), the dynamics behave as an oscillation, and the player swings between the two strategies. Figure 4 shows the phase plane of system (2.9). The asymptotically stable dynamics become the behavior of a limit cycle with the increase of time delay. From Figure 5, we can see that for different initial values, the dynamics tend to a stable equilibrium when the delay is small, and a stable limit cycle is formed when the delay is large. In other words, the proportion of cooperators eventually stabilizes at the equilibrium value when the time delay is less than the critical time delay. The stable equilibrium point disappears, and the player swings between cooperative and defective strategies when the time delay exceeds the critical time delay.

Figure 6 shows the correlation between critical time delay $\tau _{c}$ and $\sigma (\sigma = \frac{c}{a})$. Since $a > 0$, $c > 0$, and $a > c$, $\sigma \in \left (0, 1\right)$. When the cheating penalty $c$ is infinitely close to the penalty $a$ when both sides defect, the delay $\tau _{c}$ becomes larger. When $c$ is small enough, the delay $\tau _{c}$ becomes larger. Figure 7 is the bifurcation diagram. We can find that the Hopf bifurcation occurs as the time delay increases. The black line indicates that the system is stable when the time delay $\tau$ is small, and Hopf bifurcation occurs at the critical time delay. When the time delay increases, the stable equilibrium disappears, generating a stable limit cycle.

3.   The 2-player prisoner's dilemma with penalty and mutation

Based on Section 2, consider the factors of mutation to observe how the proportion of cooperators changes. Here, we assume that players who choose cooperation or defection have the same probability of $u$ $(0 < u < 1)$ mutating to each other. For example, $ux(t)$ mutants of these cooperators choose defection, and similarly, $u(1-x(t))$ mutants of some defectors desire cooperation.

3.1.   Model with penalty, mutation and no time delay

In this case, we can obtain the replicator-mutator equation for the 2-player prisoner's dilemma with penalty and mutation:

3.2.   Result with penalty, mutation and no time delay

Theorem 3.1. The internal equilibrium $x_{1}$ of system (3.1) exists and is stable.

Proof. Let $g\left (x\right) = x\left (t\right)\left (1-x\left (t\right)\right)\left [a-c-ax\left (t\right)\right]+u\left (1-2x\left (t\right)\right)$, and we get $g\left (0\right) = u > 0$, $g\left (1\right) = -u < 0$. So, there is at least one point $x_{1}$ such that $g\left (x_{1}\right) = 0$, which means $x_{1}$ is the equilibrium of Eq (3.1). According to Lemma 2.1, in the following, we want to prove that $g'\left (x_{1}\right) < 0$. Discussing Figure 7 in four cases, we find that in either case, $x_{1}$ is between two extreme values ($x_{a}$ and $x_{b}$) and thus satisfies $g'\left (x_{1}\right) < 0$.

Remark 1. Because $g\left (x\right) = ax^{3}\left (t\right)-\left (2a-c\right)x^{2}\left (t\right)+\left (a-c-2u\right)x\left (t\right)+u$, let

According to ^[26,27], we only discuss cases $\Delta = B^{2}-4AC\leq 0$ and $A = B$. $\Delta = B^{2}-4AC > 0$, which generates imaginary roots, is not considered.

Figure 9 shows the temporal dynamics of system (3.1) for different initial values and mutation rates when $a = 5$ and $c = 2$. We find that when a mutation $u$ is added, the co-existence of cooperators and defectors still occurs. The proportion of cooperators increases with the increase of $u$, but the mutation rate $u$ has little effect on the proportion of cooperators. The mutation $u$ increases, and the percentage of cooperator decreases.

3.3.   Model with penalty, mutation and time delay

Next, we study how time delays affect games. There are two delays, a payoff delay $\tau_{1}$ (the same as (2.3)) and a mutation delay $\tau_{2}$, considering that player mutation depends on the proportion of cooperators at $t-\tau_{2}$. We can obtain the replicator-mutator equation for the 2-player prisoner's dilemma with time delays:

3.4.   Result with penalty, mutation and time delay

Case 1. $\tau_{1} = \tau_{2} = 0$. The system (3.3) is the same as system (3.1).

Case 2. $\tau_{1} > 0, \tau_{2} = 0$. Then, system (3.3) becomes

The characteristic equation of system (3.4) at $x_{1}$ is

Assuming $\lambda = i\omega \left (\omega > 0\right)$ is a pure imaginary solution of Eq (3.5), substituting $\lambda = i\omega \left (\omega > 0\right)$ into Eq (3.5), we can get

The real and the imaginary parts are separated, and we get

According to (3.7), we have

Then, we get

Combining (3.7) and (3.9), the critical time delay is defined as

Theorem 3.2. Suppose $\omega_{1} > 0$ holds.

1) If $\tau_{1}\in(0, \tau_{1c})$, the equilibrium $x_{1}$ is locally asymptotically stable for system (3.4).

2) If $\tau_{1}\in (\tau _{1c}, +\infty)$, the equilibrium $x_{1}$ is unstable for system (3.4).

3) If $\tau_{1} = \tau _{1c}$, Hopf bifurcation occurs in the system (3.4) at $x_{1}$.

When $\tau_{1} < \tau_{1c}$, the internal equilibrium $x_{1}$ is asymptotically stable. When $\tau_{1} > \tau_{1c}$, the internal equilibrium $x_{1}$ is unstable, showing oscillating dynamics, as shown in Figure 10. Figure 11 shows the phase-portrait of system (3.4). We find that system (3.4) has a Hopf bifurcation when $\tau_{1} = \tau_{1c}$ and $\tau_{2} = 0$ at equilibrium $x_{1}$. Figure 12 is the bifurcation diagram with $\tau_{2} = 0$. We can find that the bifurcation occurs as the time delay increases. The black line indicates that the system has an equilibrium point when the time delay $\tau_{1}$ is slight, and Hopf bifurcation occurs at the critical time delay $\tau_{1c}$. The stable equilibrium point disappears when the time delay increases, and then the system develops a stable limit cycle.

Case 3. $\tau_{1} = 0, \tau_{2} > 0$. Then, system (3.4) becomes

The characteristic equation of system (3.11) at $x_{1}$ is

Let $\lambda = i\omega \left (\omega > 0\right)$ be a root of Eq (3.12), and we get

According to (3.13), we obtain

Thus, we have

We find no explicit solution for satisfying $\omega _{1} > 0$ and $\omega _{2} > 0$, which means that in most cases, $\tau_{1}$ and $\tau_{2}$ can not work together, that is, $\tau_{2}$ does not affect $\tau_{1}$. Next, we consider the particular case when $\tau_{1} = \tau_{2} > 0$.

Case 4. $\tau_{1} = \tau_{2} = \tau_{3} > 0$. The system (3.4) becomes

The characteristic equation of system (3.16) at $x_{1}$ is

Let $\lambda = i\omega \left (\omega > 0\right)$ be a root of Eq (3.17), substitute $\lambda = i\omega \left (\omega > 0\right)$ into Eq (3.17), and we get

The real and the imaginary parts are separated from Eq (3.18), and we obtain

Thus, we get

From (3.20), we obtain

Theorem 3.3. Suppose $\omega_{3} > 0$ holds.

1) If $\tau_{3}\in(0, \tau_{3c})$, the equilibrium $x_{1}$ is locally asymptotically stable for system (3.16).

2) If $\tau_{3}\in (\tau _{3c}, +\infty)$, the equilibrium $x_{1}$ is unstable for system (3.16).

3) If $\tau_{3} = \tau _{3c}$, Hopf bifurcation occurs in system (3.16) at $x_{1}$.

When $\tau_{1} = \tau_{2} = \tau_{3}$, Figure 13 shows the temporal dynamics of delay model (3.16). When the time delay is slight ($\tau_{3} = 0.5$), it presents a stable dynamic performance with the time change. When the delay is significant ($\tau_{3} = 0.6$), the dynamics behave as oscillation, and the cooperators swing between the two strategies. Figure 14 shows the phase-portrait of system (3.16). The system (3.16) has a Hopf bifurcation when $\tau_{3} = \tau_{3c}$ at equilibrium $x_{1}$. Figure 15 is the bifurcation diagram. We find that the bifurcation occurs as the time delay increases. The black line indicates that the equilibrium is stable. The Hopf bifurcation occurs at the critical time delay $\tau_{3c}$. The stable equilibrium point disappears when the time delay increases, and then the system develops a limit cycle.

4.   Conclusions

The classic prisoner's dilemma is dominated by defection, with the percentage of cooperators approaching zero over time. In this paper, we mainly add defection penalty and mutation to the prisoner's dilemma, so that cooperators do not disappear with the change of time. The influence of time delay on the prisoner's dilemma with or without mutation is discussed.

In Section 2 we study the 2-player prisoner's dilemmas with a penalty. Without considering the time delay, when the given penalty $a$ is greater than the cheated penalty $c$, the cooperation strategy and defection strategy coexist. The larger $a$ is, the more significant the proportion of final cooperators. Thus, when the penalty for defection is high enough, players tend to cooperate. When time delay is considered, the equilibrium is the same as the replicator equation without time delay. The critical time delay when Hopf bifurcation occurs is obtained. Critical time delay is related to a cheated penalty $c$ and a defect penalty $a$, and the condition $a > c$ is satisfied. The critical delay $\tau _{c}$ decreases as the defection penalty $a$ increases. When $\tau > \tau _{c}$, periodic oscillation occurrs.

Section 3 considers the case where the mutation is not zero based on penalty and finds that the proportion of cooperators decreases when mutation increases. Then, the system with two delays is studied, in which $\tau_{1}$ is the payoff delay, and $\tau_{2}$ is the mutation delay. Only considering the payoff delay or mutation delay, and in the particular case where the payoff delay and mutation delay are equal, also leads to oscillation. Oscillation due to delay is a general system behavior^[28,29].

Szolnoki and Perc ^[30] found that the intermediate delay enhances the reciprocity of the network. Wang et al. ^[31] found that delayed reward supports the spread of cooperation, and the intermediate reward difference between time delays promotes the highest level of cooperation. We find that when the delay is large, the stable state of dynamics is broken, which is not conducive to the stable development of the population.

This paper considers that players who choose to cooperate or defect have the same probability of mutating to the other player. Considering punishment strategy can help us understand how moral behavior is established and spreads ^[32,33,34]. In the future, we will consider the evolutionary game dynamics of the prisoner's dilemma with the third strategy to punish defectors under the effect of environmental feedback ^[35,36], and, on this basis, consider adding mutation affected by environmental interference ^[37].

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No. 12271308), the Shandong Provincial Natural Science Foundation of China (No. ZR2019MA003), the Research Fund for the Taishan Scholar Project of Shandong Province of China.