arXiv subjects

Lucius Bushnaq

Publications and source records attributed to Lucius Bushnaq.

18 recordsLinked to original sources

arXiv2026

Functional Degeneracy in Neural Networks: Measurement and Pruning

A central question in modern machine learning is how much a trained model can be compressed without changing its behavior, to reduce the memory, compute and energy required to deploy it. To study this, we quantify functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance. Using the behavioral recovery rank as a geometric benchmark for compression, we find that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated. This gap suggests that functional redundancy is distributed across parameter directions and is not exposed by individual weights or neurons.

cs.LG↗

arXiv2026

Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds

What is the geometry of a visual percept? The most widely used protocols for decomposing neural network representations into interpretable parts treat concepts as isolated directions, yet recent work shows that concepts are often realized as geometric structures in low dimensional regions of activation space. We turn to the literature of Structured sparsity to close this gap, and show that block sparsity, which groups directions into blocks, is the prior matched to a generative model in which a representation is a sparse sum of low-dimensional manifolds: the modern, learned form of a classical idea in visual neuroscience, where a visual feature is carried by a coordinated group of neurons rather than a single tuned one. We implement three variants of block-sparse featurizers (BSFs) and, through a minimum-description-length analysis, show that all three describe activations more compactly than direction-based featurizers, with the recovered concepts typically two- to four-dimensional. We then use BSFs to (i) recontextualize prior work, showing that curve detectors in InceptionV1 actually read from a single continuous curve manifold, (ii) discover novel manifolds including shadows and lighting in DINOv3, and (iii) support interpretable control of image generation in diffusion models (SDXL) via manifold steering.

cs.CV↗

arXiv2025

From Memorization to Reasoning in the Spectrum of Loss Curvature

We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs) using a decomposition based on the loss landscape curvature. This insight is based on prior theoretical and empirical work showing that the curvature for memorized training points is much sharper than non memorized, meaning ordering weight components from high to low curvature can reveal a distinction without explicit labels. This motivates a weight editing procedure that suppresses far more recitation of untargeted memorized data more effectively than a recent unlearning method (BalancedSubnet), while maintaining lower perplexity. Since the basis of curvature has a natural interpretation for shared structure in model weights, we analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected, even though open book fact retrieval and general logical reasoning is conserved. We posit these tasks rely heavily on specialized directions in weight space rather than general purpose mechanisms, regardless of whether those individual datapoints are memorized. We support this by showing a correspondence between task data's activation strength with low curvature components that we edit out, and the drop in task performance after the edit. Our work enhances the understanding of memorization in neural networks with practical applications towards removing it, and provides evidence for idiosyncratic, narrowly-used structures involved in solving tasks like math and fact retrieval.

cs.CL↗

arXiv2025

Stochastic Parameter Decomposition

A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation. Linear parameter decomposition -- a framework that has been proposed to resolve several issues with current decomposition methods -- decomposes neural network parameters into a sum of sparsely used vectors in parameter space. However, the current main method in this framework, Attribution-based Parameter Decomposition (APD), is impractical on account of its computational cost and sensitivity to hyperparameters. In this work, we introduce \textit{Stochastic Parameter Decomposition} (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose with APD. We also show that SPD avoids other issues, such as shrinkage of the learned parameters, and better identifies ground truth mechanisms in toy models. By bridging causal mediation analysis and network decomposition methods, this demonstration opens up new research possibilities in mechanistic interpretability by removing barriers to scaling linear parameter decomposition methods to larger models. We release a library for running SPD and reproducing our experiments at https://github.com/goodfire-ai/spd/tree/spd-paper.

cs.LG↗

arXiv2025

Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition

Much of mechanistic interpretability has focused on understanding the activation spaces of large neural networks. However, activation space-based approaches reveal little about the underlying circuitry used to compute features. To better understand the circuits employed by models, we introduce a new decomposition method called Local Loss Landscape Decomposition (L3D). L3D identifies a set of low-rank subnetworks: directions in parameter space of which a subset can reconstruct the gradient of the loss between any sample's output and a reference output vector. We design a series of progressively more challenging toy models with well-defined subnetworks and show that L3D can nearly perfectly recover the associated subnetworks. Additionally, we investigate the extent to which perturbing the model in the direction of a given subnetwork affects only the relevant subset of samples. Finally, we apply L3D to a real-world transformer model and a convolutional neural network, demonstrating its potential to identify interpretable and relevant circuits in parameter space.

cs.LG↗

arXiv2025

Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition

Mechanistic interpretability aims to understand the internal mechanisms learned by neural networks. Despite recent progress toward this goal, it remains unclear how best to decompose neural network parameters into mechanistic components. We introduce Attribution-based Parameter Decomposition (APD), a method that directly decomposes a neural network's parameters into components that (i) are faithful to the parameters of the original network, (ii) require a minimal number of components to process any input, and (iii) are maximally simple. Our approach thus optimizes for a minimal length description of the network's mechanisms. We demonstrate APD's effectiveness by successfully identifying ground truth mechanisms in multiple toy experimental settings: Recovering features from superposition; separating compressed computations; and identifying cross-layer distributed representations. While challenges remain to scaling APD to non-toy models, our results suggest solutions to several open problems in mechanistic interpretability, including identifying minimal circuits in superposition, offering a conceptual foundation for 'features', and providing an architecture-agnostic framework for neural network decomposition.

cs.LG↗

arXiv2025

Open Problems in Mechanistic Interpretability

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

cs.LG↗

arXiv2024

Towards evaluations-based safety cases for AI scheming

We sketch how developers of frontier AI systems could construct a structured rationale -- a 'safety case' -- that an AI system is unlikely to cause catastrophic outcomes through scheming. Scheming is a potential threat model where AI systems could pursue misaligned goals covertly, hiding their true capabilities and objectives. In this report, we propose three arguments that safety cases could use in relation to scheming. For each argument we sketch how evidence could be gathered from empirical evaluations, and what assumptions would need to be met to provide strong assurance. First, developers of frontier AI systems could argue that AI systems are not capable of scheming (Scheming Inability). Second, one could argue that AI systems are not capable of posing harm through scheming (Harm Inability). Third, one could argue that control measures around the AI systems would prevent unacceptable outcomes even if the AI systems intentionally attempted to subvert them (Harm Control). Additionally, we discuss how safety cases might be supported by evidence that an AI system is reasonably aligned with its developers (Alignment). Finally, we point out that many of the assumptions required to make these safety arguments have not been confidently satisfied to date and require making progress on multiple open research problems.

cs.CR↗

arXiv2024

Using Degeneracy in the Loss Landscape for Mechanistic Interpretability

Mechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering neural networks is that many of the parameters inside a network are not involved in the computation being implemented by the network. These degenerate parameters may obfuscate internal structure. Singular learning theory teaches us that neural network parameterizations are biased towards being more degenerate, and parameterizations with more degeneracy are likely to generalize further. We identify 3 ways that network parameters can be degenerate: linear dependence between activations in a layer; linear dependence between gradients passed back to a layer; ReLUs which fire on the same subset of datapoints. We also present a heuristic argument that modular networks are likely to be more degenerate, and we develop a metric for identifying modules in a network that is based on this argument. We propose that if we can represent a neural network in a way that is invariant to reparameterizations that exploit the degeneracies, then this representation is likely to be more interpretable, and we provide some evidence that such a representation is likely to have sparser interactions. We introduce the Interaction Basis, a tractable technique to obtain a representation that is invariant to degeneracies from linear dependence of activations or Jacobians.

cs.LG↗

arXiv2024

The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks

Mechanistic interpretability aims to understand the behavior of neural networks by reverse-engineering their internal computations. However, current methods struggle to find clear interpretations of neural network activations because a decomposition of activations into computational features is missing. Individual neurons or model components do not cleanly correspond to distinct features or functions. We present a novel interpretability method that aims to overcome this limitation by transforming the activations of the network into a new basis - the Local Interaction Basis (LIB). LIB aims to identify computational features by removing irrelevant activations and interactions. Our method drops irrelevant activation directions and aligns the basis with the singular vectors of the Jacobian matrix between adjacent layers. It also scales features based on their importance for downstream computation, producing an interaction graph that shows all computationally-relevant features and interactions in a model. We evaluate the effectiveness of LIB on modular addition and CIFAR-10 models, finding that it identifies more computationally-relevant features that interact more sparsely, compared to principal component analysis. However, LIB does not yield substantial improvements in interpretability or interaction sparsity when applied to language models. We conclude that LIB is a promising theory-driven approach for analyzing neural networks, but in its current form is not applicable to large language models.

cs.LG↗

arXiv2023

Tuning of QCD+QED simulations with C$^{\star}$ boundary conditions

We give an update on the ongoing effort of the RC$^\star$ collaboration to generate fully dynamical QCD+QED ensembles with C$^\star$ boundary conditions using the openQ$^\star$D code. The simulations were tuned to the U-symmetric point ($m_d = m_s$) with pions at $m_{π^{\pm}} \approx 400$ MeV. The splitting of the light mesons is used as one of three tuning observables and fixed to $m_{K^{0}} - m_{K^{\pm}} \approx 5$ MeV and $m_{K^{0}} - m_{K^{\pm}} \approx 25$ MeV on ensembles with renormalized electromagnetic coupling $α_{\text{R}} \approx α_{\text{phys}}$ and $α_R \approx 5.5α_{phys}$ respectively. The tuning of the three independent quark masses to the desired lines of constant physics is particularly challenging. We will define the chosen hadronic renormalization scheme, and we will present a tuning strategy based on a combination of mass reweighting and linear interpolation to explore the parameter space. We will comment on finite-volume effects comparing meson masses on two different volumes with $m_{π^{\pm}} L \approx 3.2$ and $m_{π^{\pm}} L \approx 5.1$. We will also provide some technical details on our updated strategy to calculate the sign of the fermionic Pfaffian, which arises in presence of C$^\star$ boundary conditions in place of the standard fermionic determinant. More technical details on the generation of the configurations can be found in J. Lücke's proceedings

hep-lat↗

arXiv2023

$N_f = 1+2+1$ QCD+QED simulations with C$^\star$ boundary conditions

We give an update on the ongoing effort of the RC$^\star$ collaboration to generate fully dynamical QCD+QED configurations with C$^\star$ boundary conditions using the openQ$^\star$D code. The simulations are tuned to the U-symmetric point ($m_d=m_s$) with pions at $m_{π^\pm}\approx 400$ MeV. The splitting of the light mesons is used as one of three tuning observables and fixed to $m_{K^0} - m_{K^\pm} \approx 5$ MeV and $m_{K^0} - m_{K^\pm} \approx 25$ MeV on ensembles with renormalized electromagnetic coupling $α_\mathrm{R} \approx α_\mathrm{phys.}$ and $α_\mathrm{R}\approx 5.5 α_\mathrm{phys.}$ respectively. We will discuss some details concerning our tuning strategy and present the calculation of the meson and baryon masses. Finally, we will also present a cost analysis for our simulations. More technical details on finite-volume effects and the tuning can be found in A. Cotellucci's proceedings.

hep-lat↗

arXiv2023

Strange and charm contributions to the HVP from C* boundary conditions

We present preliminary results for the determination of the leading strange and charm quark-connected contributions to the hadronic vacuum polarization contribution to the muon's g-2. Measurements are performed on the RC* collaboration's QCD ensembles, with 3+1 flavors of O(a) improved Wilson fermions and C* boundary conditions. The HVP is computed on a single value of the lattice spacing and two lattice volumes at unphysical pion mass. In addition, we compare the signal-to-noise ratio for different lattice discretizations of the vector current.

hep-lat↗

arXiv2022

Hadronic vacuum polarization with C* boundary conditions

We present a progress report on the calculation of the connected hadronic contribution to the muon g-2 with C* boundary conditions. For that purpose we use a QCD gauge ensemble with 3+1 flavors and two QCD+QED gauge ensembles with 1+2+1 flavors of dynamical quarks generated by the RC* collaboration. We detail the calculation of the vector mass and elaborate on both statistical and systematic errors.

hep-lat↗

arXiv2022

First results on QCD+QED with C* boundary conditions

Accounting for isospin-breaking corrections is critical for achieving subpercent precision in lattice computations of hadronic observables. A way to include QED and strong-isospin-breaking corrections in lattice QCD calculations is to impose C$^\star$ boundary conditions in space. Here, we demonstrate the computation of a selection of meson and baryon masses on two QCD and five QCD+QED gauge ensembles in this setup, which preserves locality, gauge and translational invariance all through the calculation. The generation of the gauge ensembles is performed for two volumes, and three different values of the renormalized fine-structure constant at the U-symmetric point, corresponding to the SU(3)-symmetric QCD in the two ensembles where the electromagnetic coupling is turned off. We also present our tuning strategy and, to the extent possible, a cost analysis of the simulations with C$^\star$ boundary conditions.

hep-lat↗

arXiv2022

Implementing noise reduction techniques into theOpenQ*D package

We present the results of testing a new technique for stochastic noise reduction in the calculation of propagators by implementing it in OpenQ*D for two ensembles with O(a) improved Wilson fermion action, with periodic boundary conditions and pion masses of 437 MeV and 331 MeV, for the connected vector and pseudoscalar correlators. We find that the technique yields no speedup compared to traditional methods, owning to the failure of its underlying assumption that the spectra of the spatial Laplacian and Dirac operators are sufficiently similar for the technique's purposes.

hep-lat↗

arXiv2021

Baryon masses from full QCD+QED${}_\text{C}$ simulations

In these proceedings we present preliminary results for the masses of the proton, neutron and $Ω^-$ baryons obtained from QCD+QED lattice simulations performed with four dynamical quarks using C$^*$ boundary conditions. These results are part of the ongoing effort of the RC${}^*$ collaboration discussed in the companion proceedings, and have been obtained on a single ensemble in which the renormalised electromagnetic coupling is $α_{\text{em}}\sim 0.04$, the physical volume is $L\sim 1.7$ fm and the masses of the four dynamical quarks have been tuned at the $U$--spin symmetric point $m_d=m_s$. We demonstrate on this unphysical ensemble that baryon masses can be calculated with satisfactory precision when including QED without the need for gauge--fixing and perturbation theory. This makes us confident in the effectiveness of the strategy presented here also in the case of simulations closer to the physical point.

hep-lat↗

arXiv2021

An update on QCD+QED simulations with C* boundary conditions

We present two novelties in our analysis of fully dynamical QCD+QED ensembles with C* boundary conditions. The first one is the explicit computation of the sign of the Pfaffian. We present an algorithm that provides a significant speedup compared to traditional methods. The second one is a reweighting of the mass in the context of the RHMC. We have tested the techniques on both pure QCD and QCD+QED ensembles with pions at $m_{π^\pm}\approx400$ MeV, a lattice spacing of $a\approx0.05$ fm, a fine-structure constant of $α_{\mathrm{R}}=0$ and $0.04$.

hep-lat↗