Joshi, K.; Halder, S.; Casanova, A. G.; Lynch, M.

For decades, population geneticists have relied upon a formula derived by Malecot and by Kimura to estimate the fixation probability of mutant alleles (phi). Among other things, this formula leads to the conclusion that in sufficiently large populations phi asymptotically approaches 2s(Ne/N) (for small s), where s is the relative selective advantage of the mutant allele, and Ne and N respectively denote the effective and absolute (haploid) population sizes. In contrast, in sufficiently small populations, phi = 1/N, with the two domains of behavior being separated at the point where s is approximately 1/(2Ne). These results hold when s, N, and Ne remain constant during the fixation process, but require modification when populations are changing in size. Here, we show that if there are ecological effects associated with competing alleles, such that the genetic composition of the population influences the total population size, the fixation probability of a beneficial allele can substantially deviate from the levels suggested above, even in the domain of effective neutrality (i.e., |s| less than 1/(2Ne)). We obtain analytical results for a variety of frequency-dependent functions for the overall population size, and show that the overall effect is largely a function of the response at low mutant-allele frequency. We also derive expressions for times to fixation and for fixation probabilities of deleterious alleles, and show how our results relate to prior work for the situation in which population sizes change in frequency-independent manners.