Chaos and bifurcations in continuous language-model feedback

Abstract

Feeding a pretrained language model the expected embedding of its own next-token distribution turns it into a discrete-time dynamical system with continuous states. Across Qwen2.5, OLMo-2 and Llama-3 models from 0.5B to 72B parameters, temperature and activation steering move this loop between repeating text and chaos, with positive finite-time Lyapunov exponents. The loop's derivative factorizes into the inverse temperature, the covariance of token embeddings under the next-token distribution (a cross-covariance for untied embeddings) and the network's Jacobian. In one model studied in detail, along recorded orbits, the growth that steering adds comes mainly from the readout, where next tokens compete. The periodic windows show the quantitative signatures of low-dimensional maps, from saddle-node scaling and Feigenbaum's constant to shrimp-shaped domains; passage times predicted in advance from the model's derivatives came within one return-map iterate of the measured passage times in 42 of 48 cases over two prospective stages (24 and 18 of 24), and interval arithmetic proves attracting periodic orbits of the full transformer map. Numerical reproducibility depends on the dynamical regime: changing only the batch layout changed the text of 173 of 192 chaotic float64 loops in the smallest model, compared with only 7 of 192 periodic ones. These results show that bifurcation theory can make quantitative predictions about continuous language-model feedback, with rigorous guarantees for selected periodic orbits of the full transformer.