Can Transformers predict system collapse in dynamical systems?

Abstract

Transformer architectures have recently surged as promising solutions for nonlinear dynamical systems, proposed as foundation models capable of zero-shot dynamics reconstruction and forecasting. Despite this success, it remains unclear whether they can truly serve as reliable digital twins of dynamical systems, i.e., whether they capture the underlying physical dynamics in distinct parameter regimes, especially in parameter regimes from which no training data is taken. For parameter-space extrapolation in nonlinear dynamical systems, reservoir computing has demonstrated broad success, as proper training can turn it into an intrinsic dynamical system capable of capturing not only the dynamical climate of the target system but more importantly, how the climate changes with parameter. Transformers, in contrast, rely on permutation-invariant attention mechanisms that can limit their ability to capture how temporal structure changes with parameter. To determine if Transformers have the capability of dynamics extrapolation, we take predicting catastrophic collapse, which occurs when a bifurcation parameter crosses a critical threshold, as a benchmark task. Models are trained on trajectories in normal parameter regimes and then tested on parameters in an unseen regime with system collapse. Our results show that Transformers, across configurations, consistently fail to capture collapse, while reservoir computing reliably predicts the transitions. This surprising finding raises questions about the generalization ability of Transformers to dynamical systems, a topic warranting future research.