Global attractors and fast-slow reduction for finite-state actor-critic mean dynamics

Abstract

When a learning algorithm reshapes the data distribution it trains on, the long-run behavior depends on the joint evolution of the policy, the value estimate, and the data distribution. We study finite-state actor-critic mean dynamics on the enlarged phase space $(\theta,w,\mu)$, where $\theta$ is the actor parameter, $w$ is an auxiliary critic state, and $\mu$ is a state-law variable (the distribution over states induced by the current policy). The state-law coordinate follows the exact controlled-Markov equation $\delta \dot\mu = Q_\theta^*\mu$. Under a softmax actor with box confinement (a smooth proxy for parameter clipping), a uniformly coercive linear critic equation, and a Lipschitz generator family $\theta \mapsto Q_\theta$, we prove that for each $\delta > 0$ the resulting autonomous semiflow possesses a compact global attractor. Under a uniform exponential-mixing assumption, we prove that the invariant-law map $\theta \mapsto \mu_\theta$ is Lipschitz and that the reduced invariant-law system on $(\theta,w)$ is well posed. Under an additional pathwise exponential-stability estimate for the non-autonomous fast state equation, we show that the exact flow tracks the reduced flow on every finite time interval up to the initial layer, and that the exact attractors converge upper semicontinuously to the lifted reduced attractor as $\delta \to 0$. We also give a concrete finite-state reference-state minorization condition implying the pathwise hypothesis. All results are formalized in Lean 4 without custom axioms.