arXiv subjects

Gonçalo Paulo

Publications and source records attributed to Gonçalo Paulo.

16 recordsLinked to original sources

arXiv2026

The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment

Fine-tuning large language models on narrow, misaligned tasks can undo their post-training alignment and induce novel misaligned behaviors -- a phenomenon known as \emph{emergent misalignment} (EM). EM has been linked to persona-like representations, where fine-tuning might reduce loss by amplifying a harmful or 'evil' persona. It remains unclear which properties of the training data drive this effect: whether all harmful examples contribute approximately equally to misalignment and whether different models are equally affected by the same fine-tuning examples. In this work, we use training data attribution to quantitatively estimate how much each harmful example contributes to EM. We benchmark the quality of the attribution via retraining -- a sound attribution score should enable us to enhance or attenuate EM by filtering data on that score. Score-based filtering can substantially enhance or attenuate EM; we find that both data-attribution scores and a black-box harmfulness score can identify consequential examples. All models we test become misaligned when trained on the same dataset, and influence scores perform best when filtering data from the same model that computed them. We find cross-model generalization of influence scores from scores derived from the three model families we tested, but this generalization does not recover same model filtering performance.

cs.CL↗

arXiv2026

Bergson: An Open Source Library for Data Attribution

Data attribution is a promising field in interpretability that aims to explain model behavior through the influence of its training data, with applications including debugging undesirable model behavior and training dataset curation. However, significant engineering effort is required to perform it at scale, and many cutting edge techniques lack open-source tooling and support. Bergson is an open source library that aims to enable faster progress in the field by providing a host of techniques that scale to very large language models and pre-training datasets. The library natively supports on-disk gradient stores and multi-node distributed training, and provides quality of life tools for researchers. Finally, we introduce the first open-source implementations of three leading data attribution methods: MAGIC, SOURCE, and TrackStar. The library is available at https://github.com/EleutherAI/bergson .

cs.LG↗

arXiv2026

Can Data Attribution Filter Out Subliminal Learning? Not Reliably

Subliminal learning allows language models to transmit behavioral traits through training data with no obvious semantic relationship to those traits, undermining content-based data filtering as a safety intervention. Training data attribution offers an alternative: it identifies the training examples responsible for a given model behavior, independent of their semantic content, and so may apply in exactly the cases where semantic inspection fails. We evaluate three gradient-based attribution methods (GradCos, a contrastive GradCos variant, and EK-FAC) across three models, comparing them against divergence tokens, a strong baseline previously shown to localize subliminal learning (albeit one that requires access to counterfactual teacher models). Filtering at the token level, EK-FAC mitigates a significant part of the effect, the other methods provide little benefit, and all mostly fall short of divergence tokens. Filtering entire samples is less effective for every method, though EK-FAC often gives a stronger signal than divergence tokens in this setting. Success is inconsistent across methods and settings: variants that work well for some model-preference combinations fail for others, and we do not identify a consistent explanation for these differences. Our results suggest that gradient-based attribution can identify data responsible for subliminal learning in some settings, but that some approximations are more reliable than others.

cs.AI↗

arXiv2026

Automated Attribution Graph Interpretation via Probe Prompting

Even though we know the precise computations that lead from a large language model (LLM) input to its output this computation remains very hard to interpret. One way to make it easier to understand this process is by creating a sparse computational graph that captures most of the model behavior with smallest number of computational nodes. Cross-layer transcoders (CLT) decompose the dense computations of the MLP but the resulting circuits still contain thousands of nodes even for short prompts. Existing automated interpretation methods label individual features from corpus activations, and it often happens that these labels are not validated by causal intervention. We introduce probe prompting, a transparent rule-based pipeline that groups the features of an attribution graph into concept-aligned supernodes from their responses on a small set of concept-targeted probe prompts, summarized as Cross-Prompt Activation Signatures (CPAS). Across four factual domains, on Gemma-2-2B with a public CLT dictionary and 45,596 entity-swap interventions, we find that the labeled supernodes have the predicted steering behavior in every one of them. Code, datasets, and an interactive demo are released anonymously as a reusable harness for calibrating supernode labels against causal interventions.

cs.CL↗

arXiv2025

Electrodrying in nanopores: from fundamentals to iontronic and memristive applications

Iontronics is a burgeoning paradigm that employs ions in solution as information carriers for sensing and computing, e.g., in neuromorphic devices. The fundamentally different working principle as compared to electronics requires novel approaches and concepts to control the impedance of nanoscale fluidic circuit elements, such as nanopores. For instance, previous research has focused on voltage-induced pore wetting as a means to trigger conduction in nanopores. The present study explores the opposite counter-intuitive mechanism: using voltage to dry hydrophobic nanopores and, therefore, to turn off conduction. This "electrodrying" concept affords exquisite, bidirectional control over the conductance of nanopores additionally showing hysteresis in the current-voltage curve that is the fingerprint of memristors. Using an analytical model and free-energy molecular dynamics simulations, we explain the physical mechanism underlying electrodrying and provide clear design criteria for solid-state and biological nanopores with bidirectional control over conductance. The electrical behaviour of electrodrying nanopores shows two unique features: i) the hysteresis loop is shifted from the origin, accounting for the fifth, previously unreported memristor type and ii) negative differential resistance is observed over a broad voltage range in which the non-conductive state is favoured by electrodrying. These properties are demonstrated in a short-term memory task and in an iontronic oscillator circuit to showcase their potential in neuromorphic applications and iontronic devices. Finally, we validate our predictions through experiments on engineered dipolar hydrophobic CytK nanopores, whose voltage-dependent conductance substantiates the electrodrying concept.

physics.chem-ph↗

arXiv2025

Automatically Interpreting Millions of Features in Large Language Models

While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which may be more easily interpretable. However, these SAEs can have millions of distinct latent features, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of explanations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a feature, which we find explains features that are not recalled by existing methods. We propose guidelines for generating better explanations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. We use our explanations to measure the semantic similarity of independently trained SAEs, and find that SAEs trained on nearby layers of the residual stream are highly similar. Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-$k$ postprocessing. Our code is available at https://github.com/EleutherAI/sae-auto-interp, and our explanations are available at https://huggingface.co/datasets/EleutherAI/auto_interp_explanations.

cs.LG↗

arXiv2025

Evaluating SAE interpretability without explanations

Sparse autoencoders (SAEs) and transcoders have become important tools for machine learning interpretability. However, measuring how interpretable they are remains challenging, with weak consensus about which benchmarks to use. Most evaluation procedures start by producing a single-sentence explanation for each latent. These explanations are then evaluated based on how well they enable an LLM to predict the activation of a latent in new contexts. This method makes it difficult to disentangle the explanation generation and evaluation process from the actual interpretability of the latents discovered. In this work, we adapt existing methods to assess the interpretability of sparse coders, with the advantage that they do not require generating natural language explanations as an intermediate step. This enables a more direct and potentially standardized assessment of interpretability. Furthermore, we compare the scores produced by our interpretability metrics with human evaluations across similar tasks and varying setups, offering suggestions for the community on improving the evaluation of these techniques.

cs.LG↗

arXiv2025

When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts. In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}. To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators. Evaluating state-of-the-art LLMs on SPOT, we find that none surpasses 21.1\% recall or 6.1\% precision (o3 achieves the best scores, with all others near zero). Furthermore, confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability. Finally, qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings. These findings highlight the substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification.

cs.CL↗

arXiv2025

Transcoders Beat Sparse Autoencoders for Interpretability

Sparse autoencoders (SAEs) extract human-interpretable features from deep neural networks by transforming their activations into a sparse, higher dimensional latent space, and then reconstructing the activations from these latents. Transcoders are similar to SAEs, but they are trained to reconstruct the output of a component of a deep network given its input. In this work, we compare the features found by transcoders and SAEs trained on the same model and data, finding that transcoder features are significantly more interpretable. We also propose skip transcoders, which add an affine skip connection to the transcoder architecture, and show that these achieve lower reconstruction loss with no effect on interpretability.

cs.LG↗

arXiv2025

Partially Rewriting a Transformer in Natural Language

The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model's final output. With our pipeline, the model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline.

cs.LG↗

arXiv2025

Sparse Autoencoders Trained on the Same Data Learn Different Features

Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.

cs.LG↗

arXiv2024

Does Transformer Interpretability Transfer to RNNs?

Recent advances in recurrent neural network architectures, such as Mamba and RWKV, have enabled RNNs to match or exceed the performance of equal-size transformers in terms of language modeling perplexity and downstream evaluations, suggesting that future systems may be built on completely new architectures. In this paper, we examine if selected interpretability methods originally designed for transformer language models will transfer to these up-and-coming recurrent architectures. Specifically, we focus on steering model outputs via contrastive activation addition, on eliciting latent predictions via the tuned lens, and eliciting latent knowledge from models fine-tuned to produce false outputs under certain conditions. Our results show that most of these techniques are effective when applied to RNNs, and we show that it is possible to improve some of them by taking advantage of RNNs' compressed state.

cs.LG↗

arXiv2023

Hydrophobically gated memristive nanopores for neuromorphic applications

Brain-inspired computing has the potential to revolutionise the current von Neumann architecture, advancing machine learning applications. Signal transmission in the brain relies on voltage-gated ion channels, which exhibit the electrical behaviour of memristors, resistors with memory. State-of-the-art technologies currently employ semiconductor-based neuromorphic approaches, which have already demonstrated their efficacy in machine learning systems. However, these approaches still cannot match performance achieved by biological neurons in terms of energy efficiency and size. In this study, we utilise molecular dynamics simulations, continuum models, and electrophysiological experiments to propose and realise a bioinspired hydrophobically gated memristive nanopore. Our findings indicate that hydrophobic gating enables memory through an electrowetting mechanism, and we establish simple design rules accordingly. Through the engineering of a biological nanopore, we successfully replicate the characteristic hysteresis cycles of a memristor \tr{and construct a synaptic device capable of learning and forgetting}. This advancement offers a promising pathway for the realization of nanoscale, cost- and energy-effective, and adaptable bioinspired memristors.

cond-mat.mes-hall↗

arXiv2023

Voltage controlled iontronic switches: a computational method to predict electrowetting in hydrophobically gated nanopores

Reliable and controllable switches are crucial in nanofluidics and iontronics. Ion channels in nature serve as a rich source of inspiration due to their intricate mechanisms modulated by stimuli like pressure, temperature, chemicals, and voltage. The artificial replication of the properties of these channels is challenging due to their complex chemistry, limited stability range, and intricate moving parts, allosterically modulated. Nonetheless, we can harness some of their gating mechanisms for nanofluidic and iontronic purposes. This theoretical and computational study explores the use of electrowetting in hydrophobic nanopores to control their conductance using an external applied voltage. We employ restrained molecular dynamics to calculate the free energy required for wetting a model nanopore under different voltages. Utilizing a simple theory, we generate free energy profiles across a wide voltage range. We also computed transition rates between conductive and non-conductive states, showing their voltage dependence and how this behaviour can impair memory to the system, resembling the memristor behaviour voltage-gated channels in the brain. These findings offer a promising avenue for designing and controlling hydrophobic nanopores via electrowetting, enabling potential applications in neuromorphic iontronics.

cond-mat.mes-hall↗

arXiv2023

An atomistically informed multiscale approach to the intrusion and extrusion of water in hydrophobic nanopores

Understanding intrusion and extrusion in nanoporous materials is a challenging multiscale problem of utmost importance for applications ranging from energy storage and dissipation to water desalination and hydrophobic gating in ion channels. Including atomistic details in simulations is required to predict the overall behavior of such systems, because the statics and dynamics of these processes depend sensitively on microscopic features of the pore such as the surface hydrophobicity, geometry, and charge distribution and on the composition of the liquid. On the other hand, the transitions between the filled (intruded) and empty (extruded) states are rare events which often require long simulation times difficult to achieve with standard atomistic simulations. In this work, we explored the intrusion and extrusion processes by a multiscale approach in which the atomistic details of the system, extracted from molecular dynamics simulations, inform a simple Langevin model of water intrusion/extrusion in the pore. We then used the Langevin simulations to compute the transition times at different pressures, validating our coarse-grained model by comparing it with nonequilibrium molecular dynamics simulations. The proposed approach reproduces experimentally relevant features such as the time and temperature dependence of the intrusion/extrusion cycles, as well as specific details about the shape of the cycle. This approach also drastically increases the timescales that can be simulated allowing to reduce the gap between simulations and experiments and showing promise for more complex systems.

cond-mat.soft↗

arXiv2021

Binary Mixtures of Locally Coupled Mobile Oscillators

We study synchronization dynamics in binary mixtures of locally coupled Kuramoto oscillators which perform Brownian motion in a two-dimensional box. We introduce two models, where in model $\cal I$ there are two type of oscillators, say $\cal A$ and $\cal B$, and any two similar oscillators tend to synchronize their phases, while any two dissimilar ones tend to be out of phase. In model $\cal J$, in contrast, the oscillators in subpopulation $\cal A$ behave as in model $\cal I$, while the oscillators in subpopulation $\cal B$ tend to be out of phase with all the others. In the real space all the oscillators in both models interact via a soft-core repulsive potential. Both subpopulations of model $\cal I$ and subpopulation $\cal A$ of model $\cal J$, by their own, exhibit a phase coherent attractor in a certain region of model parameters. The approach to the attractor, after an initial transient regime, is exponential with some characteristic synchronization time scale $τ$. Numerical analysis reveals that the attractors of the two subpopulations survive within model $\cal I$, regardless of the composition of the mixture $ϕ$ and the strength of the cross-population negative coupling constant $H$, and that $τ$ sensitively depends on $ϕ$, $H$ and the packing fraction. In particular, the ability of the oscillators to move and exchange neighbours can significantly decrease $τ$. In contrast, model $\cal J$ predicts suppression of the synchronized state in subpopulation $\cal A$ and emergence of the coherent attractor in the "contrarians" subpopulation $\cal B$ for strong and weak cross-population coupling, respectively.

cond-mat.soft↗