Transformer-Based Characterization of Criticality in Directed Percolation

Abstract

Directed percolation (DP) provides a canonical framework for studying nonequilibrium phase transitions into absorbing states, yet extracting critical behavior directly from high-dimensional spatiotemporal configurations remains challenging. Here, we investigate (1 + 1)- and (2 + 1)-dimensional bond DP using a supervised Transformer encoder. Supervised learning results demonstrate that the Transformer can distinguish between the active and absorbing phases of directed percolation with high accuracy. The critical point and critical exponents estimated from the model's classification probabilities align closely with reference values from existing literature, indicating that the model not only achieves phase classification but also yields outputs containing information related to critical scaling behavior. To probe the model's internal representation, we employ attention rollout and define an attention center that quantifies the temporal centroid of the cumulative attention distribution. The attention patterns and the attention center reorganize systematically near the critical region and exhibit pronounced finite-size dependence. Finally, we examined the model's transferability between configurations generated from fully occupied initial states and those from single-seed initial states. The results indicate the presence of transferable information between the two types of data, though the transfer performance exhibits directional asymmetry. Furthermore, when using an equal amount of target-domain data, fine-tuning yielded only a marginal improvement in accuracy compared to training from scratch.