arXiv subjects

David Quarel

Publications and source records attributed to David Quarel.

5 recordsLinked to original sources

arXiv2026

Interpreting Reinforcement Learning Agents with Susceptibilities

Susceptibilities are a technique for neural network interpretability that studies the response of posterior expectation values of observables to perturbations of the loss. We generalize this construction to the setting of the regret in deep reinforcement learning and investigate the utility of susceptibilities in a simple gridworld model that nevertheless exhibits non-trivial stagewise development. We argue that susceptibilities reveal internal features of the development of the model in parameter space that one cannot detect purely by studying the development of the learned policy. We validate these results with activation-steering, and discuss the framework's extension to RLHF post-training.

cs.LG↗

arXiv2026

Stagewise Reinforcement Learning and the Geometry of the Regret Landscape

Singular learning theory characterizes Bayesian learning as an evolving tradeoff between accuracy and complexity, with transitions between qualitatively different solutions as sample size increases. We extend this theory to reinforcement learning, proving that the concentration of a generalized posterior over policies is governed by the local learning coefficient (LLC), an invariant of the geometry of the regret function. This theory predicts that deep reinforcement learning with SGD should proceed from simple policies with high regret to complex policies with low regret. We verify this prediction empirically in a gridworld environment exhibiting stagewise policy development: phase transitions over training manifest as "opposing staircases" where regret decreases sharply while the LLC increases.

cs.LG↗

arXiv2025

SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMs

Mechanistic interpretability aims to decompose neural networks into interpretable features and map their connecting circuits. The standard approach trains sparse autoencoders (SAEs) on each layer's activations. However, SAEs trained in isolation don't encourage sparse cross-layer connections, inflating extracted circuits where upstream features needlessly affect multiple downstream features. Current evaluations focus on individual SAE performance, leaving interaction sparsity unexamined. We introduce SCALAR (Sparse Connectivity Assessment of Latent Activation Relationships), a benchmark measuring interaction sparsity between SAE features. We also propose "Staircase SAEs", using weight-sharing to limit upstream feature duplication across downstream features. Using SCALAR, we compare TopK SAEs, Jacobian SAEs (JSAEs), and Staircase SAEs. Staircase SAEs improve relative sparsity over TopK SAEs by $59.67\% \pm 1.83\%$ (feedforward) and $63.15\% \pm 1.35\%$ (transformer blocks). JSAEs provide $8.54\% \pm 0.38\%$ improvement over TopK for feedforward layers but cannot train effectively across transformer blocks, unlike Staircase and TopK SAEs which work anywhere in the residual stream. We validate on a $216$K-parameter toy model and GPT-$2$ Small ($124$M), where Staircase SAEs maintain interaction sparsity improvements while preserving feature interpretability. Our work highlights the importance of interaction sparsity in SAEs through benchmarking and comparing promising architectures.

cs.LG↗

arXiv2023

Universal Agent Mixtures and the Geometry of Intelligence

Inspired by recent progress in multi-agent Reinforcement Learning (RL), in this work we examine the collective intelligent behaviour of theoretical universal agents by introducing a weighted mixture operation. Given a weighted set of agents, their weighted mixture is a new agent whose expected total reward in any environment is the corresponding weighted average of the original agents' expected total rewards in that environment. Thus, if RL agent intelligence is quantified in terms of performance across environments, the weighted mixture's intelligence is the weighted average of the original agents' intelligences. This operation enables various interesting new theorems that shed light on the geometry of RL agent intelligence, namely: results about symmetries, convex agent-sets, and local extrema. We also show that any RL agent intelligence measure based on average performance across environments, subject to certain weak technical conditions, is identical (up to a constant factor) to performance within a single environment dependent on said intelligence measure.

cs.AI↗

arXiv2017

On a numerical upper bound for the extended Goldbach conjecture

The Goldbach conjecture states that every even number can be decomposed as the sum of two primes. Let $D(N)$ denote the number of such prime decompositions for an even $N$. It is known that $D(N)$ can be bounded above by $$ D(N) \leq C^* Θ(N), \quad Θ(N):= \frac{N}{\log^2 N}\prod_{\substack{p|N p>2}} \left( 1 + \frac{1}{p-2}\right)\prod_{p>2}\left(1-\frac{1}{(p-1)^2}\right) $$ where $C^*$ denotes Chen's constant. It is conjectured that $C^*=2$. In 2004, Wu showed that $C^* \leq 7.8209$. We attempted to replicate his work in computing Chen's constant, and in doing so we provide an improved approximation of the Buchstab function $ω(u)$, \begin{align*} ω(u)=1/u, & \quad (1\leq u\leq 2), (u ω(u))'=ω(u-1), & \quad (u\geq 2). \end{align*} based on work done by Cheer and Goldston. For each interval $[j,j+1]$, they expressed $ω(u)$ as a Taylor expansion about $u=j+1$. We expanded about the point $u=j+0.5$, so $ω(u)$ was never evaluated more than $0.5$ away from the center of the Taylor expansion, which gave much stronger error bounds. Issues arose while using this Taylor expansion to compute the required integrals for Chen's constant, so we proceeded with solving the above differential equation to obtain $ω(u)$, and then integrating the result. Although the values that were obtained undershot Wu's results, we pressed on and refined Wu's work by discretising his integrals with finer granularity. The improvements to Chen's constant were negligible (as predicted by Wu). This provides experimental evidence, but not a proof, that were Wu's integrals computed on smaller intervals in exact form, the improvement to Chen's constant would be negligible. Thus, any substantial improvement on Chen's constant likely requires a radically different method to what Wu provided.

math.HO↗