A generative pre-trained transformer with Kerr-soliton attention

Abstract

Artificial intelligence systems, particularly through generative pre-trained transformers (GPTs), have enabled capability-rich language models, but their operation incurs substantial costs in digital computation, memory, and data movement. Attention is a core operation in GPTs that computes context-dependent weights for input tokens. Since deep-learning models are defined by compositions of nonlinear transformations, identifying physical systems that can realize them offers a pathway to higher efficiency. Here, we introduce Kerr-soliton attention, harnessing driven-dissipative nonlinear dynamics in a resonator to realize, execute, and validate a deep-learning attention operation in physical hardware. We train a transformer language model using an analytic Kerr-soliton attention response and explore generative inference by streaming model-produced inputs through the experimental system. We observe high-fidelity agreement between the experimentally produced nonlinear weights and those predicted by the analytic Kerr-soliton model. Computation proceeds through streaming-in-time excitation of an ensemble of Kerr solitons, with inputs encoded as temporal signals that evolve under nonlinear dynamics. Our approach maps memory and compute onto the same physical dynamics, relaxing the need for intermediate digital storage and reducing data movement. This work points toward hybrid digital-physical learning systems in which Kerr solitons provide physical memory and high-bandwidth streaming nonlinear processing within deep-learning models.