Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

Abstract

The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet et al. (Heredity 116(4):362–371, 2016) introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size-change having an identical distribution of $T_{2}$ (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on $T_2$. In this paper, based on an analytical study of the rate matrix of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times $(T_3,T_2)$ for a sample of three haploid genes in a n-island model with constant size. Even if, for any $k \ge 2$, it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of $T_k$ is exactly the same as in a n-island model with constant population size, we show that the joint distribution of the coalescence times $(T_3,T_2)$ for a sample of three genes contains enough information to distinguish between a panmictic population and a n-island model of constant size.

Appendices

Appendix A Proofs for Sections 2 and 3

Proof of Lemma 1

We first observe that

$$\begin{aligned} - 2 - \displaystyle \frac{nM}{n-1}< -\frac{3}{n} < 0. \end{aligned}$$

Then the calculus gives

$$\begin{aligned} p(0)= & {} \displaystyle \frac{9M(Mn+ 2n -2)}{2(n-1)^2}> 0, \\ p\left( -\frac{3}{n}\right)= & {} - \displaystyle \frac{9(M n^2 + 2(n-1)(n-3))}{2 n^3} < 0, \\ p\left( - 2 - \displaystyle \frac{nM}{n-1} \right)= & {} \frac{(Mn + 2n-2)(M(n-3) + 2n-2)}{2 (n-1)^2} > 0 \\ \lim _{\mu \rightarrow -\infty } p(\mu )= & {} - \infty . \end{aligned}$$

The intermediate value theorem applied to $p(\mu )$ thus provides a proof of the result. $\square$

Proof of Proposition 2

In order to simplify computations, we make the following change of parameters:

$$\begin{aligned} M = 2 N, \; n = \frac{a + N}{a}. \end{aligned}$$

The new parameters a et N verify $a > 0$ and $N > 0$ and the matrix Q becomes

$$\begin{aligned} Q= \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}3&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}1\\ 0&{}6\,a&{}-6\,a&{}0&{}0\\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] . \end{aligned}$$

The matrix Q has the double eigenvalue 0 and it is easy to check that the corresponding eigenspace is generated by the following non colinear vectors:

$$\begin{aligned} V_1 = [N,N-1,N-1,2 N, N -a -1]^T, \quad V_2 = [1,1,1,1,1]^T. \end{aligned}$$

We then consider the change of basis matrix

$$\begin{aligned} P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}0&{}0&{}N&{}1\\ 0&{}\quad 1&{}\quad 0&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 1&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 0&{}\quad 2\,N&{}\quad 1 \\ 0&{}\quad 0&{}\quad 0&{}\quad N-a-1&{}\quad 1\end{array} \right] , \end{aligned}$$

whose inverse is

$$\begin{aligned} P_1^{-1} = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}\quad 0&{}\quad 0&{}\quad -{\frac{a+1}{N+a+1}}&{}\quad -{\frac{N}{ N+a+1}}\\ 0&{}\quad 1&{}\quad 0&{}\quad -{\frac{a}{N+a+1}}&{}-\quad {\frac{N+1}{N+ a+1}}\\ 0&{}\quad 0&{}\quad 1&{}\quad -{\frac{a}{N+a+1}}\quad &{}-{\frac{N+1}{N+a+ 1}}\\ 0&{}\quad 0&{}\quad 0&{}\quad \frac{1}{N+a+1} &{}\quad - \frac{1}{N+a+1}\\ 0&{}\quad 0&{}\quad 0&{}\quad -{\frac{N-a-1}{N+a+1}}&{}\quad {\frac{2N}{N+a+1}}\end{array} \right] . \end{aligned}$$

We have thus obtained a partial diagonalization (by blocks) of the matrix Q:

$$\begin{aligned} Q_2 = P_1^{-1} Q P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}0&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}0\\ 0&{}6\,a&{}-6\,a&{}0&{}0 \\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] , \end{aligned}$$

(24)

and we put

$$\begin{aligned} R := \left[ \begin{array}{c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0\\ a&{}-1+a- 2\,N&{}-2\,a+2\,N\\ 0&{}6\,a&{}-6\,a\end{array} \right] . \end{aligned}$$

The characteristic polynomial of R is the polynomial $p(\mu )$, which has the following expressions using the new parameters:

$$\begin{aligned} p(\mu ) = \mu ^3 + (5N+5a+4) \mu ^2 + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) \mu + 18 a (N + a +1), \end{aligned}$$

which has three strictly negative real roots. These roots are all distinct by Lemma 1.

If $\mu$ is an eigenvalue of R, we then determine a corresponding eigenvector $W(\mu )$ by solving the equation $R W(\mu ) = \mu W(\mu )$.

The computations show that we can choose

$$\begin{aligned} W(\mu ) = \left[ \begin{array}{c} \mu ^2 + (2N + 5a+1) \mu + 6a(a+1) \\ a(\mu + 6a) \\ 6 a^2 \end{array} \right] . \end{aligned}$$

We then consider the $3 \times 3$ passage matrix $P_2$, whose column vectors are the $W(\mu _i), \, i=1,2,3$, where the $\mu _i$ are the three eigenvalues of R:

$$\begin{aligned} P_2 = \left[ W(\mu _1), \; W(\mu _2), \; W(\mu _3) \right] . \end{aligned}$$

Some easy computations on the rows show that the determinant of $P_2$ is a Van der Monde determinant, and hence: $\mathrm {det}(P_2) = (\mu _1 - \mu _2)(\mu _1-\mu _3)(\mu _2 - \mu _3) \ne 0$.

The computations of the inverse $P_2^{-1}$ gives

$$\begin{aligned} P_2^{-1} = \left[ Z(\mu _1), Z(\mu _2), Z(\mu _3)\right] ^T, \end{aligned}$$

where

$$\begin{aligned} Z(\mu ) = \frac{1}{p'(\mu )} \; \left[ \begin{array}{c} 1 \\ \displaystyle \frac{\mu + 3N+3}{a} \\ -\displaystyle \frac{\mu ^2 + (3N+a+4)\mu + 3(N+a+1)}{a \mu } \end{array}\right] , \end{aligned}$$

with

$$\begin{aligned} p'(\mu ) = 3 \mu ^2 + 2(5 N + 5 a +4)\mu + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) . \end{aligned}$$

We further obtain

$$\begin{aligned} e^{t R}= & {} P_2 \; \left[ \begin{array}{c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0\\ 0 &{} e^{\mu _2 t} &{}0 \\ 0 &{} 0 &{} e^{\mu _3 t} \end{array} \right] \; P_2^{-1}, \\= & {} \sum _{j=1}^3 e^{\mu _j} \; W(\mu _j) Z(\mu _j)^T. \end{aligned}$$

We then introduce the matrices $A(\mu )$ and B defined by

$$\begin{aligned} A(\mu ) := P_1 \; \bar{W}(\mu ) \bar{Z}(\mu ) ^T \; P_1^{-1}, \end{aligned}$$

where $\bar{W}_j(\mu )$ (resp. $\bar{Z}(\mu )$) is a vector of length 5 obtained by adding two null coordinates to $W(\mu )$ (resp. $Z(\mu )$), and

$$\begin{aligned} B = P_1 \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 1 \end{array} \right] \; P_1^{-1}. \end{aligned}$$

Emphasizing the rank one property of matrix $A(\mu )$, let us define the vectors

$$\begin{aligned} E_1 = \left[ \begin{array}{c} \displaystyle \frac{3 M}{v} \\ 1 \\ \displaystyle \frac{3 M}{u} \\ 0 \\ 0 \end{array} \right] , \quad E_2 = \left[ \begin{array}{c} M u \\ (n-1) u v \\ (n-1)(n-2) M v \\ \displaystyle \frac{3 M u}{\mu } \\ \displaystyle \frac{(n-1) uv}{\mu } \end{array} \right] , \end{aligned}$$

where $\mu$ is an eigenvalue of R, and u, v are functions of $\mu$ defined by

$$\begin{aligned} u = (n-1)\mu + 3 M,\quad v = 2 \mu + 3(M+2). \end{aligned}$$

Note that, since

$$\begin{aligned} p\left( \displaystyle -\frac{3M}{n-1}\right)= & {} - \displaystyle \frac{9}{2} \cdot \frac{(n-2) [2(n-1) + (n-3)M] M^2}{(n-1)^3} <0,\\ p\left( \displaystyle -\frac{3(M+2)}{2}\right)= & {} \displaystyle \frac{9}{8} \cdot \frac{ [2(n-1) + (n-3)M] M^2}{(n-1)^2} >0, \end{aligned}$$

$\displaystyle -\frac{3M}{n-1}$ and $\displaystyle -\frac{3(M+2)}{2}$ cannot be eigenvalues and thus $u \ne 0$ et $v \ne 0$.

With $\delta (M,n,\mu ) = 2(n-1)^2 p'(\mu )$, we have

$$\begin{aligned} A(\mu ) = \displaystyle \frac{1}{\delta (M,n,\mu )} \; E_1 E_2^T, \end{aligned}$$

(25)

which gives the expression of $A(\mu )$ in Proposition 2.

The stated result easily follows. $\square$

Proof of Proposition 4

Using Eq. (15) and Proposition 2, we deduce that

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = -3 \ \frac{A(\mu _1)(i,4)+A(\mu _1)(i,5)}{3A(\mu _1)(i,1)+A(\mu _1)(i,2)}, \quad i=1,2,3. \end{aligned}$$

Note that the matrix $E_1 E_2^T$ introduced in the proof of Proposition 2 can also be factorized in a different manner. If we define the $3 \times 3$ matrix C, the $5 \times 3$ matrix $D_1$ and the $3 \times 5$ matrix $D_2$ by

$$\begin{aligned} C:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c} \displaystyle \frac{u}{v} &{} u &{} 1 \\ u &{} uv &{} v \\ 1 &{} v &{} \displaystyle \frac{v}{u} \end{array} \right] ,\quad D_1 := \left[ \begin{array}{c@{\quad }c@{\quad }c} 3 M &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 3M \\ 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right] ,\\ D_2:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} M &{} 0 &{} 0 &{} \displaystyle \frac{3M}{\mu } &{} 0 \\ 0 &{} n-1 &{} 0 &{} 0 &{} \displaystyle \frac{n-1}{\mu } \\ 0 &{} 0 &{} (n-1)(n-2) &{} 0 &{} 0 \end{array} \right] , \end{aligned}$$

we may check that, for any value of $\mu$, we have $E_1 E_2^T = D_1 C D_2$.

The vectors $V_1$ and $V_2$, defined by

$$\begin{aligned} V_1 := \left[ \begin{array}{c} 3 \\ 0 \\ 0 \\ vp \\ 0 \end{array} \right] , \;\; V_2 := \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ 0 \\ -\mu \end{array} \right] , \end{aligned}$$

verify $D_2 V_j = 0$ for $j=1,2$, and hence, using Eq. (25), $A(\mu )V_j = 0$, for $j=1,2$.

Therefore, for $i=1,2,3$,

$$\begin{aligned} 3A(\mu )(i,1) - \mu A(\mu )(i,4) = 0, \; A(\mu )(i,2) - \mu A(\mu )(i,5) = 0, \end{aligned}$$

and the result follows. $\square$

Proof of Lemma 2

Note that $-3\alpha$ and $-3\beta$, with $\alpha , \beta$ defined in Eq. (10), are the roots of the polynomial $q_1(X)= q(X/3)$, where

$$\begin{aligned} q(X) = X^2 + \left( 1 + \frac{nM}{n-1}\right) X + \frac{M}{n-1}, \end{aligned}$$

and on the open subset $D = \left\{ (n,M):\; n> 2, \; M>0 \right\}$ of $\mathbb {R}^2$ the polynomials p(X) and $q_1(X)$ have a common root if and only their resultant $R(n,M) = \text{ Res }(p(X),q_1(X),X)$ with respect to X is null [see e.g. Lang (2002, Chapter IV–8)].

Because $R(n,M) = - \frac{M^3}{9(n-1)^2} < 0$ on D, and because the roots $-3 \alpha$ and $\mu _3$ are continuous functions of (n, M) on D, the inequality $-3 \alpha < \mu _3$ for one value of $(n,M) \in D$ implies the inequality everywhere on D. The same is true for the inequality $\mu _1 < -3 \beta$.

This achieves the proof of the lemma. $\square$

Proof of Proposition 5

We will only give the proof of (i), which corresponds to the case of three genes sampled from the same deme. The proofs ot the analogous results (ii) and (iii), corresponding to the two other sample schemes, are similar.

When $t \rightarrow 0$, using the well-known relation $P_t = I_5 + tQ + o(t)$, where $I_5$ denotes the identity matrix, we have the following Taylor expansions:

$$\begin{aligned} P_t(1,1)&= 1 + \left( -\frac{3M}{2}-3\right) t + o(t),\\ P_t(1,2)&= \frac{3M}{2} t + o(t). \end{aligned}$$

In particular, we have

$$\begin{aligned} \frac{1}{3P_t(1,1)+P_t(1,2)} = \frac{1}{1-(M+3)t +o(t)} = 1+(M+3)t+o(t). \end{aligned}$$

Using Eqs. (7)–(11), and a Taylor expansion of order 2 of the exponential in the neighborhood of 0, we easily obtain

$$\begin{aligned} F_{2,s}(u)&= u - \frac{M+1}{2}u^2 + o\left( u^2\right) ,\\ F_{2,d}(u)&= \frac{M}{2(n-1)}u^2 + o\left( u^2\right) . \end{aligned}$$

When substituting the above expressions into Eq. (11), straightforward calculations give

$$\begin{aligned} \mathbb {P}_1\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = u - \frac{M}{2}ut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) . \end{aligned}$$

Further, using Eq. (18), let us denote

$$\begin{aligned} h(u,t):=\mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) =1 - \left( \frac{1 - P_{t+u}(1,4)-P_{t+u}(1,5)}{1 - P_{t}(1,4)-P_{t}(1,5)}\right) ^{1/3}. \end{aligned}$$

We will write a Taylor expansion of order 2 of h(u, t) in the neighborhood of (0, 0). We have $h(0,0)=0$ and direct computations give

$$\begin{aligned} \frac{\partial h}{\partial u}(u,t)&= \frac{1}{3} \times \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{-1/3} \times \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}\\&\quad \times \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) ,\\ \frac{\partial h}{\partial t}(u,t)&= \frac{\left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}}{3\left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{4/3}} \\&\quad \times \{ \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) \\&\quad - \left( P_{t}Q(1,4)+P_{t}Q(1,5) \right) \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) \}. \end{aligned}$$

Using the fact that $P_0 = I_5,$ and $Q(1,4)=3, Q(1,5)=0$, we obtain

$$\begin{aligned} \frac{\partial h}{\partial u}(0,0)=1, \frac{\partial h}{\partial t}(0,0)=0. \end{aligned}$$

We further compute the second partial derivatives of h in (0, 0). With $Q^2$ being the square matrix of the rate matrix Q, and using the fact that $Q^2(1,4) = -9\left( \frac{M}{2}+1\right)$ and $Q^2(1,5) = \frac{3M}{2},$ we easily derive

$$\begin{aligned} \frac{\partial ^2 h}{\partial u \partial t}(0,0)&= \frac{1}{3} \left\{ \left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -M,\\ \frac{\partial ^2 h}{\partial u^2}(0,0)&= \frac{1}{3} \left\{ \frac{2}{3}\left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -(M+1),\\ \frac{\partial ^2 h}{\partial t^2}(0,0)&= 0, \end{aligned}$$

The Taylor expansion of order 2 of h(u, t) near (0, 0) finally gives

$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) = u - Mut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) , \end{aligned}$$

which finishes the proof. $\square$

Proof of Proposition 6

Because $0< \beta < \alpha$, using Eqs. (7) and (8), we get

$$\begin{aligned} F_{2,s}(u) = 1 - \frac{a}{\alpha } e^{-\alpha u} - \frac{1-a}{\beta } e^{-\beta u} = 1 - \frac{1-a}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) \end{aligned}$$

and

$$\begin{aligned} F_{2,d}(u) = 1 - \frac{c}{\alpha } e^{-\alpha u} + \frac{c}{\beta } e^{-\beta u} = 1 + \frac{c}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) . \end{aligned}$$

Therefore, using Eq. (11), we obtain

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - K_{1,i}(n,M,t) e^{-\beta u} + o\left( e^{-\beta u}\right) , \end{aligned}$$

where $K_{1,i}(n,M,t)$, given by (20), is strictly positive because $0< a < 1$ and $c < 0$.

Using Proposition 2 we get

$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j) = \left( A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1(t+u)} + o\left( e^{\mu _1(t+u)}\right) . \end{aligned}$$

Therefore, using Eq. (18) and relation $1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)$, we obtain

$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) = 1 - K_{2,i}(n,M,t)\; e^{\frac{\mu _1}{3} u} + o\left( e^{\frac{\mu _1}{3} u}\right) , \end{aligned}$$

where

$$\begin{aligned} K_{2,i}(n,M,t) :=\left( \frac{A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)}{P_t(i,1) + P_t(i,2) + P_t(i,3)} \right) ^{\frac{1}{3}} \; e^{\frac{\mu _1}{3} t}. \end{aligned}$$

Because $\frac{\mu _1}{3} < - \beta$ from Lemma 2, we have $\displaystyle e^{\frac{\mu _1}{3} u} = o(e^{-\beta u}),$ which achieves the proof of (19). $\square$

Proof of Proposition 7

We will first prove a useful lemma. $\square$

Lemma 3

Let us define $\phi (\mu )$ by

$$\begin{aligned} \phi (\mu ) := \displaystyle \frac{3 A(\mu )(1,1)}{3 A(\mu )(1,1) + A(\mu )(1,2)}, \end{aligned}$$

where the matrix $A(\mu )$ is given in Proposition 2.

Then,

• (i) $0< \phi (\mu _i)< - \frac{\mu _i}{3} < \alpha , \; i=1,2,3.$

• (ii) Defining h(n) by

  $$\begin{aligned} h(n) = \frac{(n-1)\left( 25 - 9n + 5 \sqrt{9n^2 - 18n + 25}\right) }{12 n^2}, \end{aligned}$$

  we have the following results:

  • (a) If $M > h(n)$, then $\phi (\mu _1)< \beta< \phi (\mu _2)< \phi (\mu _3) < \alpha .$

  • (b) If $M = h(n)$, then $\phi (\mu _1)< \beta = \phi (\mu _2)< \phi (\mu _3) < \alpha .$

  • (c) If $M < h(n)$, then $\phi (\mu _1)< \phi (\mu _2)< \beta< \phi (\mu _3) < \alpha .$

Proof

From Proposition 2 and using $p(\mu _i)=0$, we get

$$\begin{aligned} \phi (\mu _i)= & {} \displaystyle \frac{1}{3} \frac{(n-1)\mu _i^2 + (nM + n-1)\mu _i + 6M}{Mn + 2(n-1)} \\= & {} \displaystyle \frac{3(n-1)}{nM + 2(n-1)} \; q\left( \frac{\mu _i}{3}\right) - \displaystyle \frac{M(\mu _i n + 3)}{nM + 2(n-1)} - \displaystyle \frac{\mu _i}{3}. \end{aligned}$$

From Lemma 1, it follows that $\mu n + 3 >0$. Because $-\alpha< \displaystyle \frac{\mu _i}{3} < -\beta$ from Lemma 2, we get $q\left( \frac{\mu _i}{3}\right) < 0$, that proves $\phi (\mu _i) < - \frac{\mu _i}{3}$.

Now, using again that $p(\mu _i)=0$, we obtain the following expression for $\phi (\mu _i)$

$$\begin{aligned} \phi (\mu _i) = \frac{3 M}{2(n-1)\mu _i + 3nM + 6(n-1)}. \end{aligned}$$

(26)

Introducing the new variable $\nu = \displaystyle \frac{3 M}{2(n-1)\mu + 3nM + 6(n-1)}$, and thus $\mu = \displaystyle - \frac{3}{2} \frac{(nM + 2n-2))\nu - M}{(n-1) \nu }$, we get that $\phi (\mu _i)$, for $i=1,2,3$ are the three real roots of the polynomial

$$\begin{aligned} r(\nu )= & {} 4(n-1)\left( nM + 2n-2\right) \nu ^3 - \left[ n^2 M^2 + 2(n-1)(3n+4) M + 8(n-1)^2 \right] \nu ^2 \\&+ 2 M (2 nM + 5n-5) \nu - 3M^2. \end{aligned}$$

The coefficient signs of r(X) show that r(X) has no negative roots, implying that $\phi (\mu _i) >0, \; i=1,2,3$, which achieves the proof of (i).

The set of (n, M) such that r(X) and $q(-X)$ have a common root is obtained by computing their resultant with respect to X, denoted by R(n, M):

$$\begin{aligned} R(n,M) = \text{ Res }(r(X),q(-X),X) = M^3 \left[ 6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 \right] . \end{aligned}$$

In the domain $D = \{(n,M),\; n>2, \; M>0 \}$ the curve $6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 = 0$ is identical to the graph of the function $M = h(n), \; n > 2$.

The set $D {\setminus } \{(n,h(n)): \; n > 2 \}$ has two connex open components in which the relative position of $\beta , \alpha$ and $\phi (\mu _i), \; i=1,2,3$ are the same.

In the component $D_1 := \{(n,M):\; n>2,\; M > h(n)\}$, one may choose $n=3, \; M=\frac{4}{3}$ for which

$$\begin{aligned} \beta= & {} \displaystyle \frac{3}{2} - \displaystyle \frac{\sqrt{57}}{6} \approx 0.241694, \\ \alpha= & {} \displaystyle \frac{3}{2} + \displaystyle \frac{\sqrt{57}}{6} \approx 2.758305,\\ \phi (\mu _1)= & {} \displaystyle \frac{3 - \sqrt{5}}{4} \approx 0.190983, \\ \phi (\mu _2)= & {} \displaystyle \frac{1}{3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{3 + \sqrt{5}}{4} \approx 1.3090167, \end{aligned}$$

that proves $(ii-a)$.

In the component $D_2 := \{(n,M):\; n>2,\; M < h(n)\}$, one may choose $n=3, \; M= \frac{1}{2}$ for which

$$\begin{aligned} \beta= & {} \displaystyle \frac{7 - \sqrt{33}}{8} \approx 0.156929, \\ \alpha= & {} \displaystyle \frac{7 + \sqrt{33}}{8} \approx 1.593070, \\ \phi (\mu _1)\approx & {} 0.104089, \\ \phi (\mu _2)\approx & {} 0.146359, \\ \phi (\mu _3)\approx & {} 1.118868, \end{aligned}$$

that proves $(ii-c)$.

The curve arc $\{(n,h(n)): n > 2 \}$ may be parametrized by

$$\begin{aligned} n = \displaystyle \frac{3u^2 + 8u-3}{3(u^2-1)}, \quad M = \displaystyle \frac{8u}{(u+3)^2},\quad u \in ]1,3[, \end{aligned}$$

and we obtain

$$\begin{aligned} \beta= & {} \displaystyle \frac{u-1}{u+3}, \\ \alpha= & {} \displaystyle \frac{3(u+1)}{u+3}, \\ \phi (\mu _1)= & {} \displaystyle \frac{39u + 51 -3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \\ \phi (\mu _2)= & {} \displaystyle \frac{u-1}{u+3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{39u + 51 +3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \end{aligned}$$

and $(ii-b)$ is proved. $\square$

Lemma 3 is now used to prove Proposition 7. First note that we have

$$\begin{aligned} \phi (\mu ) = \displaystyle \frac{3 A(\mu )(i,1)}{3 A(\mu )(i,1) + A(\mu )(i,2)}, \end{aligned}$$

for every $i=1,2,3$.

When $t \rightarrow + \infty$, $P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2$ and thus

$$\begin{aligned}&\mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) \\&\quad = \frac{3 A(\mu _1)(i,1)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,s}(u) + \frac{A(\mu _1)(i,2)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,d}(u) + o(1) \\&\quad = \phi (\mu _1) F_{2,s}(u) + (1 - \phi (\mu _1)) F_{2,d}(u) + o(1). \end{aligned}$$

Using the definitions of $F_{2,s}(u)$ and $F_{2,d}(u)$ in Eqs. (7) and (8), we get

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - c_1 e^{-\beta u} - c_2 e^{-\alpha u} + o(1), \end{aligned}$$

where $c_1$ and $c_2$ are given by (23).

On another hand, we have $P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2,3$, we obtain

$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 (t+u)} + o\left( e^{\mu _1 t}\right) ,\\ \sum _{j=1}^3 P_t(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) . \end{aligned}$$

Using the fact that $1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)$, this implies

$$\begin{aligned} \mathbb {P}(T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t) = 1 - e^{ \frac{\mu _1}{3} u} + o(1), \end{aligned}$$

and thus (21) holds.

It remains to show that $K_3(n,M,u) >0$.

Using $c_1 + c_2 = 1$, we may write

$$\begin{aligned} K_3(n,M,u) = \frac{\beta - \phi (\mu _1)}{\beta -\alpha } \left( e^{- \alpha u} - e^{\frac{\mu _1}{3} u}\right) + \frac{\phi (\mu _1)-\alpha }{\beta -\alpha } \left( e^{ \frac{\mu _1}{3} u} - e^{-\beta u}\right) . \end{aligned}$$

From Lemma 2, $-\alpha< \frac{\mu _1}{3} < -\beta$ and thus $e^{-\alpha u}< e^{\frac{\mu _1}{3} u} < e^{-\beta u}$. On the other hand, from Lemma 3 we have $\phi (\mu _1)< \beta < \alpha$. Therefore $K_3(n,M,u) >0$. $\square$

Appendix B The case of $n=2$ islands

If we now consider the ancestral lineage process for a sample of three genes in the case of a symmetrical n-island model with $n=2$ two islands, we only have the following four possible configurations:

• 1. the three lineages are in the same island,

• 2. two lineages are in the same island and the third one is in the other island,

• 3. there are only two ancestral lineages left and they are in the same island,

• 4. there are only two ancestral lineages left and they are in different islands.

The corresponding transition rate matrix is:

$$\begin{aligned} Q = \begin{pmatrix} -\frac{3M}{2}-3 &{}\quad \frac{3M}{2} &{}\quad 3 &{}\quad 0\\ \frac{M}{2} &{}\quad -\frac{M}{2}-1 &{}\quad 0 &{}\quad 1 \\ 0 &{}\quad 0&{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \end{pmatrix}, \end{aligned}$$

The characteristic polynomial of Q is $\chi _Q(\mu ) = -\mu ^2 p(\mu )$, with

$$\begin{aligned} p(\mu ) = \mu ^2 + 2(M+2) \mu + 3(M+1). \end{aligned}$$

The matrix Q has the double eigenvalue 0 and the corresponding eigenspace of dimension 2 can be generated by the vectors $\displaystyle \left[ \frac{M}{2},\frac{M}{2}-1, M, -1\right] ^T$ and $\displaystyle [1,1,1,1]^T$. The two other eigenvalues are $\mu _1 = - M - 2 + \sqrt{M^2+M+1}$ and $\mu _2 = - M - 2 - \sqrt{M^2+M+1}$.

An eigenvector for $\mu _1$ (resp. $\mu _2$) is $[3M, 2 \mu _1+3M+6, 0, 0]^T$ (resp. $[3M, 2 \mu _2+3M+6, 0, 0]^T$), and we may consider the following change of basis matrix P given by

$$\begin{aligned} P = \begin{pmatrix} 3M &{}\quad 3M &{}\quad \frac{M}{2} &{}\quad 1 \\ 2 \mu _1+3M+6 &{}\quad 2 \mu _2+3M+6 &{}\quad \frac{M}{2}-1 &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad M &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad -1 &{}\quad 1 \end{pmatrix}. \end{aligned}$$

From

$$\begin{aligned} P^{-1} = \begin{pmatrix} \displaystyle \frac{2 \mu _1+M+2}{12M(\mu _1 + M+2)} &{} \displaystyle \frac{1}{4(\mu _1+M+2)} &{} \displaystyle \frac{2\mu _1+M+2}{4 M \mu _1(\mu _1 + M +2)} &{} \displaystyle \frac{1}{4\mu _1(\mu _1+M+2)} \\ \displaystyle \frac{2 \mu _2+M+2}{12M(\mu _2 + M+2)} &{} \displaystyle \frac{1}{4(\mu _2+M+2)} &{} \displaystyle \frac{2\mu _2+M+2}{4 M \mu _2(\mu _2 + M +2)} &{} \displaystyle \frac{1}{4\mu _2(\mu _2+M+2)} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{} - \displaystyle \frac{1}{M+1} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{}\displaystyle \frac{M}{M+1} \end{pmatrix}, \end{aligned}$$

and the equality

$$\begin{aligned} e^{t Q} = P \; \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0 &{} 0 \\ 0 &{} e^{\mu _2} &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 1 \end{array} \right] \; P^{-1}, \end{aligned}$$

a proof of the following proposition is obtained.

Proposition 8

The transition kernel $P_t = e^{t Q}$ is given by

$$\begin{aligned} P_t = e^{\mu _1 t} A(\mu _1) + e^{\mu _2 t} A(\mu _2) + B, \end{aligned}$$

where

$$\begin{aligned} A(\mu )= & {} \displaystyle \frac{1}{\delta (M,\mu )} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 2 \mu +M+2 &{} 3 M &{} \displaystyle \frac{3(2\mu +M+2)}{\mu } &{} \displaystyle \frac{3M}{\mu } \\ M &{} 2\mu +3M+6 &{} \displaystyle \frac{3M}{\mu } &{} \displaystyle \frac{2\mu +3M+6}{\mu } \\ 0 &{} 0 &{}0 &{} 0 \\ 0 &{} 0 &{}0 &{} 0 \end{array} \right] ,\\ B= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad b &{}\quad 1 - b \\ 0 &{} \quad 0 &{}\quad 1-b &{} \quad b \\ 0 &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 \end{array}\right] , \end{aligned}$$

with $\delta (M,\mu ) = 4(\mu + M+2)$ and $b= \displaystyle \frac{M+2}{2(M+1)}$.

The IICRs $\lambda _i(\cdot )/3$, for the initial sample configurations $i=1$ and $i=2$, correspond to the size-change functions

$$\begin{aligned} \lambda _i(t) = \frac{3(1 - P_t(i,3) - P_t(i,4))}{3P_t(i,1) + P_t(i,2)}, \end{aligned}$$

and the following proposition is verified.

Proposition 9

When $t \rightarrow \infty$, $\lambda _i(\cdot ),\; i=1,2$, have the following limit

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3}{\mu _1}. \end{aligned}$$

In Fig. 7 we plot the two functions $\lambda _i(\cdot ),\; i=1,2$ for $n=2$ demes and $M=1$ (left), respectively $M=0.1$ (right). The dashed line indicates the common asymptotic value $- \frac{3}{\mu _1}$. We can also show, using Proposition 8, that $\displaystyle \lim _{M \rightarrow \infty } \lambda _i(t) = 2$ for every $t > 0$ and $i=1,2$. This corresponds to the dotted line in the left panel of Fig. 7.

Proof

Using Proposition 8 we get

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3(A(\mu _1)(i,3) + A(\mu _1)(i,4))}{3A(\mu _1)(i,1) + A(\mu _1)(i,2)}. \end{aligned}$$

Then the relations $3 A(\mu _1)(i,1) = \mu _1 A(\mu _1)(i,3)$ and $A(\mu _1)(i,2) = \mu _1 A(\mu _1)(i,4)$ allow to prove the result. $\square$

Fig. 7

Plot of the functions $\lambda _i(\cdot ),\; i=1,2$ for $n=2$ and $M=1$ (left), respectively $M=0.1$ (right). The dashed line corresponds to the asymptotic value $- 3/\mu _1$. The dotted line in the left panel corresponds to the limit value $n=2$ when $M \rightarrow \infty$

The conditional cumulative distribution functions $\mathbb {P}\left( T_2^{(3),\lambda _i} \le \cdot | T_3^{(3),\lambda _i} = t\right)$ and $\mathbb {P}_i(T_2^{(3),2,M} \le \cdot | T_3^{(3),2,M} = t)$ are given, for every $t > 0$, by formulas analogous to (11) and (18):

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t\right)&= \frac{F_{2,s}(u)\frac{d}{dt} P_t(i,3)+F_{2,d}(u)\frac{d}{dt} P_t(i,4)}{f_{T_3^{(3),2,M},i}(t)} \nonumber \\&= \frac{3F_{2,s}(u)P_t(i,1)+F_{2,d}(u)P_t(i,2)}{3P_t(i,1)+P_t(i,2)}, \end{aligned}$$

(27)

$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right)&= 1- \exp \{-(\varLambda _i(t+u) - \varLambda _i(t))\}\nonumber \\&= 1 - \left( \frac{1 - P_{t+u}(i,3)-P_{t+u}(i,4)}{1 - P_{t}(i,3)-P_{t}(i,4)}\right) ^{1/3}. \end{aligned}$$

(28)

In order to compare, for $u, t > 0$, the conditional cumulative distribution functions given in Eqs. (27) and (28), let us introduce the functions

$$\begin{aligned} g_i(u,t) := \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t \right) -\mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) , \; i=1,2. \end{aligned}$$

Proposition 10

The functions $g_i(u,t),\; i=1,2$ have the following asymptotic behaviour:

• 1. For (u, t) in the neighborhood of (0, 0), we have

  $$\begin{aligned} g_1(u,t)= & {} \displaystyle \frac{M}{2} u t + o\left( u^2+t^2\right) , \\ g_2(u,t)= & {} \displaystyle -\frac{u}{3} + \frac{6 M + 1}{18} u^2 + \frac{7 M}{6} u t + o\left( u^2+t^2\right) . \end{aligned}$$

• 2. For fixed $t > 0$, when $u \rightarrow +\infty$,

  $$\begin{aligned} g_i(u,t) = - K_{1,i} (M,t) e^{-\beta u} + o(e^{-\beta u}),\; i=1,2, \end{aligned}$$

  where $K_{1,i}(M,t) > 0$ is given by

  $$\begin{aligned} K_{1,i}(M,t) = \displaystyle \frac{3 P_t(i,1)}{3P_t(i,1) + P_t(i,2)} \frac{1-a}{\beta } - \displaystyle \frac{P_t(i,2)}{3P_t(i,1) + P_t(i,2)} \frac{c}{\beta }, \end{aligned}$$

  where constants $\beta , a, c$ are defined in Eqs. (9) and (10) with $n=2$, i.e.

  $$\begin{aligned}&\beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\quad a = \frac{1}{2} \left( 1 + \left( 4M^2+1\right) ^{-1/2}\right) ,\\&\quad c = -M \left( 4M^2+1\right) ^{-1/2}. \end{aligned}$$

• 3. For fixed $u \ge 0$, we have

  $$\begin{aligned} \displaystyle \lim _{t \rightarrow +\infty } g_i(u,t) = -K_3(M,u),\quad i=1,2, \end{aligned}$$

  where $K_3(M,u) > 0$ is given by

  $$\begin{aligned} K_3(M,u) = c_1 e^{-\beta u} + c_2 e^{-\alpha u} - e^{\frac{\mu _1}{3} u}, \end{aligned}$$

  with

  $$\begin{aligned} \alpha= & {} M + \frac{1}{2}\left( 1 + \left( 4M^2+1\right) ^{1/2}\right) ,\quad \beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\; \\ \phi \left( \mu _1\right)= & {} \displaystyle \frac{3M}{2\left( \mu _1 + 3M + 3\right) },\quad c_1 = \frac{\phi \left( \mu _1\right) - \alpha }{\beta -\alpha },\quad c_2 = \frac{\beta - \phi \left( \mu _1\right) }{\beta -\alpha }. \end{aligned}$$

The proof is similar to the one given in the case $n > 2$.

Most of the calculations of “Appendices A and B” section were made and/or verified using the computer algebra system Maple: programs and tracks of their execution are available from the authors.