Quenched coalescent for diploid population models with selfing and overlapping generations

Abstract

We introduce a general diploid population model with self-fertilization and possible overlapping generations, and study the genealogy of a sample of $n$ genes as the population size $N$ tends to infinity. Unlike traditional approach in coalescent theory which considers the unconditional (annealed) law of the gene genealogies averaged over the population pedigree, here we study the conditional (quenched) law of gene genealogies given the pedigree. We focus on the case of high selfing probability and obtain that this conditional law converges to a random probability measure, given by the random law of a system of coalescing random walks on an exchangeable fragmentation-coalescence process of \cite{berestycki04}. This system contains the system of coalescing random walks on the ancestral recombination graph as a special case, and it sheds new light on the site-frequency spectrum (SFS) of genetic data by specifying how SFS depends on the pedigree. The convergence result is proved by means of a general characterization of weak convergence for random measures on the Skorokhod space with paths taking values in a locally compact Polish space.