Backward bifurcations in spatial replicator models:when invasion criteria fail to predict coexistence

Abstract

The replicator equation is a central framework for studying frequency--dependent selection in ecology and evolutionary game theory. In well-mixed populations, the long-term outcome of a two-species system is determined by the signs of the pairwise invasion fitnesses, leading to dominance, coexistence, or bistability. However, many ecological systems are spatially structured, with environmental heterogeneity and dispersal shaping local interactions. How these spatial effects modify the classical replicator regimes remains incompletely understood. In this work, we study a spatially heterogeneous extension of the two-species replicator equation in which pairwise invasion fitnesses vary across space, and frequencies evolve under diffusion and advection. In this setting, the classical invasion fitnesses are replaced by spatial invasion rates given by the principal eigenvalues of the associated linearized operators. Using bifurcation theory, we show that these spatial invasion rates do not fully determine the qualitative dynamics of the system. In particular, spatial heterogeneity can induce a backward bifurcation, generating stable coexistence states even in parameter regimes where the well-mixed replicator predicts competitive exclusion. We derive an explicit local condition for this mechanism, characterizing when such coexistence states arise. These results show that spatial structure can fundamentally alter classical replicator dynamics and provide a concrete mechanism through which coexistence may emerge.