arXiv subjects

Stefan Heimersheim

Publications and source records attributed to Stefan Heimersheim.

31 records · Page 2Linked to original sources

arXiv2024

Evolution of SAE Features Across Layers in LLMs

Sparse Autoencoders for transformer-based language models are typically defined independently per layer. In this work we analyze statistical relationships between features in adjacent layers to understand how features evolve through a forward pass. We provide a graph visualization interface for features and their most similar next-layer neighbors (https://stefanhex.com/spar-2024/feature-browser/), and build communities of related features across layers. We find that a considerable amount of features are passed through from a previous layer, some features can be expressed as quasi-boolean combinations of previous features, and some features become more specialized in later layers.

cs.LG↗

arXiv2024

Using Degeneracy in the Loss Landscape for Mechanistic Interpretability

Mechanistic Interpretability aims to reverse engineer the algorithms implemented by neural networks by studying their weights and activations. An obstacle to reverse engineering neural networks is that many of the parameters inside a network are not involved in the computation being implemented by the network. These degenerate parameters may obfuscate internal structure. Singular learning theory teaches us that neural network parameterizations are biased towards being more degenerate, and parameterizations with more degeneracy are likely to generalize further. We identify 3 ways that network parameters can be degenerate: linear dependence between activations in a layer; linear dependence between gradients passed back to a layer; ReLUs which fire on the same subset of datapoints. We also present a heuristic argument that modular networks are likely to be more degenerate, and we develop a metric for identifying modules in a network that is based on this argument. We propose that if we can represent a neural network in a way that is invariant to reparameterizations that exploit the degeneracies, then this representation is likely to be more interpretable, and we provide some evidence that such a representation is likely to have sparser interactions. We introduce the Interaction Basis, a tractable technique to obtain a representation that is invariant to degeneracies from linear dependence of activations or Jacobians.

cs.LG↗

arXiv2024

The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks

Mechanistic interpretability aims to understand the behavior of neural networks by reverse-engineering their internal computations. However, current methods struggle to find clear interpretations of neural network activations because a decomposition of activations into computational features is missing. Individual neurons or model components do not cleanly correspond to distinct features or functions. We present a novel interpretability method that aims to overcome this limitation by transforming the activations of the network into a new basis - the Local Interaction Basis (LIB). LIB aims to identify computational features by removing irrelevant activations and interactions. Our method drops irrelevant activation directions and aligns the basis with the singular vectors of the Jacobian matrix between adjacent layers. It also scales features based on their importance for downstream computation, producing an interaction graph that shows all computationally-relevant features and interactions in a model. We evaluate the effectiveness of LIB on modular addition and CIFAR-10 models, finding that it identifies more computationally-relevant features that interact more sparsely, compared to principal component analysis. However, LIB does not yield substantial improvements in interpretability or interaction sparsity when applied to language models. We conclude that LIB is a promising theory-driven approach for analyzing neural networks, but in its current form is not applicable to large language models.

cs.LG↗

arXiv2024

How to use and interpret activation patching

Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.

cs.LG↗

arXiv2023

Joint analysis constraints on the physics of the first galaxies with low frequency radio astronomy data

The first billion years of cosmic history remains largely unobserved. We demonstrate, using a novel machine learning technique, how combining upper limits on the spatial fluctuations in the 21-cm signal with observations of the sky-averaged 21-cm signal from neutral hydrogen can improve our understanding of this epoch. By jointly analysing data from SARAS3 (redshift $z\approx15-25$) and limits from HERA ($z\approx8$ and $10$), we show that such a synergetic analysis provides tighter constraints on the astrophysics of galaxies 200 million years after the Big Bang than can be achieved with the individual data sets. Although our constraints are weak, this is the first time data from a sky-averaged 21-cm experiment and power spectrum experiment have been analysed together. In synergy, the two experiments leave only $64.9^{+0.3}_{-0.1}$% of the explored broad theoretical parameter space to be consistent with the joint data set, in comparison to $92.3^{+0.3}_{-0.1}$% for SARAS3 and $79.0^{+0.5}_{-0.2}$% for HERA alone. We use the joint analysis to constrain star formation efficiency, minimum halo mass for star formation, X-ray luminosity of early emitters and the radio luminosity of early galaxies. The joint analysis disfavours at 68% confidence a combination of galaxies with X-ray emission that is $\lesssim 33$ and radio emission that is $\gtrsim 32$ times as efficient as present day galaxies. We disfavour at $95$% confidence scenarios in which power spectra are $\geq126$ mK$^{2}$ at $z=25$ and the sky-averaged signals are $\leq-277$ mK.

astro-ph.CO↗

arXiv2023

FlexKnot and Gaussian Process for 21 cm global signal analysis and foreground separation

The cosmological 21 cm signal is one of the most promising avenues to study the Epoch of Reionization. One class of experiments aiming to detect this signal is global signal experiments measuring the sky-averaged 21 cm brightness temperature as a function of frequency. A crucial step in the interpretation and analysis of such measurements is separating foreground contributions from the remainder of the signal, requiring accurate models for both components. Current models for the signal (non-foreground) component, which may contain cosmological and systematic contributions, are incomplete and unable to capture the full signal. We propose two new methods for extracting this component from the data: Firstly, we employ a foreground-orthogonal Gaussian Process to extract the part of the signal that cannot be explained by the foregrounds. Secondly, we use a FlexKnot parameterization to model the full signal component in a free-form manner, not assuming any particular shape or functional form. This method uses Bayesian model selection to find the simplest signal that can explain the data. We test our methods on both, synthetic data and publicly available EDGES low-band data. We find that the Gaussian Process can clearly capture the foreground-orthogonal signal component of both data sets. The FlexKnot method correctly recovers the full shape of the input signal used in the synthetic data and yields a multi-modal distribution of different signal shapes that can explain the EDGES observations.

astro-ph.CO↗

arXiv2023

Towards Automated Circuit Discovery for Mechanistic Interpretability

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under investigation, researchers can understand the functionality of each component. We automate one of the process' steps: to identify the circuit that implements the specified behavior in the model's computational graph. We propose several algorithms and reproduce previous interpretability results to validate them. For example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery.

cs.LG↗

arXiv2023

Improved Constraints on the 21 cm EoR Power Spectrum and the X-Ray Heating of the IGM with HERA Phase I Observations

We report the most sensitive upper limits to date on the 21 cm epoch of reionization power spectrum using 94 nights of observing with Phase I of the Hydrogen Epoch of Reionization Array (HERA). Using similar analysis techniques as in previously reported limits (HERA Collaboration 2022a), we find at 95% confidence that $Δ^2(k = 0.34$ $h$ Mpc$^{-1}$) $\leq 457$ mK$^2$ at $z = 7.9$ and that $Δ^2 (k = 0.36$ $h$ Mpc$^{-1}) \leq 3,496$ mK$^2$ at $z = 10.4$, an improvement by a factor of 2.1 and 2.6 respectively. These limits are mostly consistent with thermal noise over a wide range of $k$ after our data quality cuts, despite performing a relatively conservative analysis designed to minimize signal loss. Our results are validated with both statistical tests on the data and end-to-end pipeline simulations. We also report updated constraints on the astrophysics of reionization and the cosmic dawn. Using multiple independent modeling and inference techniques previously employed by HERA Collaboration (2022b), we find that the intergalactic medium must have been heated above the adiabatic cooling limit at least as early as $z = 10.4$, ruling out a broad set of so-called "cold reionization" scenarios. If this heating is due to high-mass X-ray binaries during the cosmic dawn, as is generally believed, our result's 99% credible interval excludes the local relationship between soft X-ray luminosity and star formation and thus requires heating driven by evolved low-metallicity stars.

astro-ph.CO↗

arXiv2022

HERA Phase I Limits on the Cosmic 21-cm Signal: Constraints on Astrophysics and Cosmology During the Epoch of Reionization

Recently, the Hydrogen Epoch of Reionization Array (HERA) collaboration has produced the experiment's first upper limits on the power spectrum of 21-cm fluctuations at z~8 and 10. Here, we use several independent theoretical models to infer constraints on the intergalactic medium (IGM) and galaxies during the epoch of reionization (EoR) from these limits. We find that the IGM must have been heated above the adiabatic cooling threshold by z~8, independent of uncertainties about the IGM ionization state and the nature of the radio background. Combining HERA limits with galaxy and EoR observations constrains the spin temperature of the z~8 neutral IGM to 27 K < T_S < 630 K (2.3 K < T_S < 640 K) at 68% (95%) confidence. They therefore also place a lower bound on X-ray heating, a previously unconstrained aspects of early galaxies. For example, if the CMB dominates the z~8 radio background, the new HERA limits imply that the first galaxies produced X-rays more efficiently than local ones (with soft band X-ray luminosities per star formation rate constrained to L_X/SFR = { 10^40.2, 10^41.9 } erg/s/(M_sun/yr) at 68% confidence), consistent with expectations of X-ray binaries in low-metallicity environments. The z~10 limits require even earlier heating if dark-matter interactions (e.g., through millicharges) cool down the hydrogen gas. Using a model in which an extra radio background is produced by galaxies, we rule out (at 95% confidence) the combination of high radio and low X-ray luminosities of L_{r,ν}/SFR > 3.9 x 10^24 W/Hz/(M_sun/yr) and L_X/SFR<10^40 erg/s/(M_sun/yr). The new HERA upper limits neither support nor disfavor a cosmological interpretation of the recent EDGES detection. The analysis framework described here provides a foundation for the interpretation of future HERA results.

astro-ph.CO↗

arXiv2022

What it Takes to Measure Reionization with Fast Radio Bursts

Fast Radio Bursts (FRBs) are extra-galactic radio transients which exhibit a distance-dependent dispersion of their signal, and thus can be used as cosmological probes. In this article we, for the first time, apply a model-independent approach to measure reionization from synthetic FRB data assuming these signals are detected beyond redshift 5. This method allows us to constrain the full shape of the reionization history as well as the CMB optical depth $τ$ while avoiding the problems of commonly used model-based techniques. 100 localized FRBs, originating from redshifts 5-15, could constrain (at 68% confidence level) the CMB optical depth to within 11%, and the midpoint of reionization to 4%, surpassing current state-of-the-art CMB bounds and quasar limits. Owing to the higher numbers of expected FRBs at lower redshifts, the $τ$ constraints are asymmetric (+14%, -7%) providing a much stronger lower limit. Finally, we show that the independent constraints on reionization from FRBs will improve limits on other cosmological parameters such as the amplitude of the power spectrum of primordial fluctuations.

astro-ph.CO↗

arXiv2022

Contribution to the 2022 Cosmology session of the 56th Rencontres de Moriond: Reionization constraints from HERA 21cm power spectrum limits

I present our analysis of the 21cm power spectrum upper limits from the HERA radio interferometer, published in HERA Collaboration et al. 2022. We use the recent limits to constrain reionization and the properties of the IGM and galaxies in the early universe. I focus in particular on the possibility of a radio background in addition to the CMB (e.g. produced by early galaxies) which can lead to a stronger 21cm signal and is thus easier to constrain. I show what limits the HERA observations can put on these models and the IGM, and how this compares to existing constraints on the radio and X-ray background.

astro-ph.CO↗

arXiv2022

What it takes to measure Reionization with Fast Radio Bursts

Fast Radio Bursts (FRBs) are recently discovered extra-galactic radio transients which are now used as novel cosmological probes. We show how the Bursts' Dispersion Measure can model-independently probe the history of Hydrogen reionization. Using a FlexKnot free-form parameterization to reconstruct the reionization history we predict an 11% accuracy constraint on the CMB optical depth, and 4% accuracy on the midpoint of reionization, to be achieved with 100 FRBs originating from redshifts z>5.

astro-ph.CO↗

arXiv2020

Cannibalism hinders growth: Cannibal Dark Matter and the $S_8$ tension

Many models of dark matter have been proposed in attempt to ease the $S_8$ tension between weak lensing and CMB experiments. One such exciting possibility is cannibalistic dark matter (CanDM), which has exothermal number-changing interactions allowing it to stay warm far into its non-relativistic regime. Here we investigate the cosmological implications of CanDM and how it impacts CMB anisotropies and the matter power spectrum, by implementing the model within a linear Einstein-Boltzmann solver. We show that CanDM suppresses the small scale matter power spectrum in a way very similar to light Warm Dark Matter or Hot Dark Matter. However, unlike in those models, the suppression may happen while the CanDM model still remains compatible with CMB constraints. We put strong constraints on the interaction strength of CanDM as a function of its abundance for both constant and temperature-dependent thermally-averaged cross sections. We find that the CanDM model can easily solve the $S_8$ tension (but has no impact on the Hubble tension). Indeed, it can accommodate values of $S_8$ of the order of 0.76 while being compatible with CMB+BAO data. However, as long as the $S_8$ tension remains moderate, the overall $χ^2$ improvement is relatively small given the number of extra free parameters, and the CanDM model is not significantly preferred.

astro-ph.CO↗