Alastair James Arthur Price

arXiv2026

FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with $O(N\log^2 N)$ algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.

cs.LG↗

arXiv2024

Adaptive hybrid density functionals

Exact exchange contributions are known to crucially affect electronic states, which in turn govern covalent bond formation and breaking in chemical species. Empirically averaging the exact exchange admixture over compositional degrees of freedom, hybrid density functional approximations have been widely successful, yet have fallen short to reach high level quantum chemistry accuracy, primarily due to delocalization errors. We propose to `adaptify` hybrid functionals by generating optimal admixture ratios of exact exchange on the fly, i.e. specifically for any chemical compound, using extremely data efficient quantum machine learning models that carry negligible overhead. The adaptive Perdew-Burke-Ernzerhof based hybrid density functional (aPBE0) is shown to yield atomization energies with sufficient accuracy to effectively cure the infamous spin gap problem in open shell systems, such as carbenes. aPBE0 further improves energetics, electron densities, and HOMO-LUMO gaps in organic molecules drawn from the QM9 and QM7b data set. Obtained with aPBE0 in a large basis, we present a revision of the entire QM9 data set (revQM9) with an estimated quality vastly superior to the original containing on average, stronger covalent binding, larger band-gaps, more localized electron densities, and larger dipole-moments. While aPBE0 is applicable in the equilibrium regime, outstanding limitations include covalent bond dissociation when going beyond the Coulson-Fisher point.

physics.chem-ph↗