Benjamin Levi | Searcharxiv

arXiv2025

Post-Norm can Resharpen Attention

Length Generalization is the essential capacity of autonomous agents to perform tasks in longer contexts than those encountered during training. To systematically study this feat, we test how well models can approximate the next token distributions in algorithmic tasks. This is to take into account the realistic possibility of multiple next tokens being legal. We present a prototypical benchmark for this line of study: in the Set Complement Task, the model needs to output a uniform distribution over tokens not in the input. We prove a theorem that states simple transformers can length generalize on this task, however, with performance degradation due to attention dispersion. A mechanistic reading of how dispersion takes effect lets us discover a remedy: Post-Norm can Resharpen Attention. We present experimental evidence to support this idea. We also show that Exponential Moving Averages can help the issue of noisy gradients that arises when many next tokens are legal. We validate the general applicability of our proposed methods on a suite of formal language experiments. Our source code will be available upon publication.

cs.LG↗

arXiv2004

Quantum Computation of a Complex System : the Kicked Harper Model

The simulation of complex quantum systems on a quantum computer is studied, taking the kicked Harper model as an example. This well-studied system has a rich variety of dynamical behavior depending on parameters, displays interesting phenomena such as fractal spectra, mixed phase space, dynamical localization, anomalous diffusion, or partial delocalization, and can describe electrons in a magnetic field. Three different quantum algorithms are presented and analyzed, enabling to simulate efficiently the evolution operator of this system with different precision using different resources. Depending on the parameters chosen, the system is near-integrable, localized, or partially delocalized. In each case we identify transport or spectral quantities which can be obtained more efficiently on a quantum computer than on a classical one. In most cases, a polynomial gain compared to classical algorithms is obtained, which can be quadratic or less depending on the parameter regime. We also present the effects of static imperfections on the quantities selected, and show that depending on the regime of parameters, very different behaviors are observed. Some quantities can be obtained reliably with moderate levels of imperfection, whereas others are exponentially sensitive to imperfection strength. In particular, the imperfection threshold for delocalization becomes exponentially small in the partially delocalized regime. Our results show that interesting behavior can be observed with as little as 7-8 qubits, and can be reliably measured in presence of moderate levels of internal imperfections.

quant-ph↗