arXiv subjects

Thomas Marshall

Publications and source records attributed to Thomas Marshall.

5 recordsLinked to original sources

arXiv2025

Parameterized Synthetic Text Generation with SimpleStories

We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language.

cs.CL↗

arXiv2024

Variable Rate Neural Compression for Sparse Detector Data

High-energy large-scale particle colliders generate data at extraordinary rates. Developing real-time high-throughput data compression algorithms to reduce data volume and meet the bandwidth requirement for storage has become increasingly critical. Deep learning is a promising technology that can address this challenging topic. At the newly constructed sPHENIX experiment at the Relativistic Heavy Ion Collider, a Time Projection Chamber (TPC) serves as the main tracking detector, which records three-dimensional particle trajectories in a volume of a gas-filled cylinder. In terms of occupancy, the resulting data flow can be very sparse reaching $10^{-3}$ for proton-proton collisions. Such sparsity presents a challenge to conventional learning-free lossy compression algorithms, such as SZ, ZFP, and MGARD. In contrast, emerging deep learning-based models, particularly those utilizing convolutional neural networks for compression, have outperformed these conventional methods in terms of compression ratios and reconstruction accuracy. However, research on the efficacy of these deep learning models in handling sparse datasets, like those produced in particle colliders, remains limited. Furthermore, most deep learning models do not adapt their processing speeds to data sparsity, which affects efficiency. To address this issue, we propose a novel approach for TPC data compression via key-point identification facilitated by sparse convolution. Our proposed algorithm, BCAE-VS, achieves a $75\%$ improvement in reconstruction accuracy with a $10\%$ increase in compression ratio over the previous state-of-the-art model. Additionally, BCAE-VS manages to achieve these results with a model size over two orders of magnitude smaller. Lastly, we have experimentally verified that as sparsity increases, so does the model's throughput.

physics.ins-det↗

arXiv2024

Refusal in LLMs is an Affine Function

We propose affine concept editing (ACE) as an approach for steering language models' behavior by intervening directly in activations. We begin with an affine decomposition of model activation vectors and show that prior methods for steering model behavior correspond to subsets of terms of this decomposition. We then provide a derivation of ACE and use it to control refusal behavior on ten different models, including Llama 3 70B. ACE combines affine subspace projection and activation addition to reliably control the model's refusal responses across prompt types. We evaluate the results using LLM-based scoring on a collection of harmful and harmless prompts. Our experiments demonstrate that ACE consistently achieves more precise control over model behavior than existing methods and generalizes to models where directional ablation via affine subspace projection alone produces incoherent outputs. Code for reproducing our results is available at https://github.com/EleutherAI/steering-llama3 .

cs.LG↗

arXiv2024

Does Transformer Interpretability Transfer to RNNs?

Recent advances in recurrent neural network architectures, such as Mamba and RWKV, have enabled RNNs to match or exceed the performance of equal-size transformers in terms of language modeling perplexity and downstream evaluations, suggesting that future systems may be built on completely new architectures. In this paper, we examine if selected interpretability methods originally designed for transformer language models will transfer to these up-and-coming recurrent architectures. Specifically, we focus on steering model outputs via contrastive activation addition, on eliciting latent predictions via the tuned lens, and eliciting latent knowledge from models fine-tuned to produce false outputs under certain conditions. Our results show that most of these techniques are effective when applied to RNNs, and we show that it is possible to improve some of them by taking advantage of RNNs' compressed state.

cs.LG↗

arXiv2023

Contrasting Features of Parton Energy Loss in Heavy-ion Collisions at RHIC and the LHC

Energetic quarks and gluons lose energy as they traverse the hot and dense medium created in high-energy heavy-ion collisions at the BNL Relativistic Heavy Ion Collider (RHIC) and the CERN Large Hadron Collider (LHC). The nuclear modification factor ($R_{AA}$) of leading particles quantifies parton energy loss in such collisions, with the particle spectrum in $p+p$ collisions as a reference. Previous $R_{AA}$ measurements at RHIC energies have revealed an approximately constant trend at high transverse momenta ($p_{T}$), implying a scenario where parton energy loss, $\Delta p_{T}$, scales proportionally with $p_{T}$, a feature naively expected from energy loss dynamics in elastic collisions. In this study, we investigate the LHC $R_{AA}$ measurements which exhibit a pronounced $p_{T}$ dependence of $R_{AA}$ for various particle species, and our analysis attributes this behavior to $\Delta p_T$ being approximately proportional to $\sqrt{p_{T}}$. These distinct features are consistent with model calculations of dominant radiative energy loss dynamics at the LHC, in contrast to the dominance of collisional energy loss at RHIC. Additionally, the linear increase of fractional energy loss with medium density at different $p_{T}$ magnitudes affirms the previous empirical observation that the magnitude of the energy loss depends mostly on the initial entropy density, with no significant path-length dependence. Implications on the dynamical scenarios of parton energy loss and future experimental investigations will also be discussed.

nucl-th↗