Searcharxiv⌕ Search

arXiv subjects

Lingjie Chen

Publications and source records attributed to Lingjie Chen.

22 records · Page 2Linked to original sources

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders

Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models, yet scalable training remains a significant challenge. We introduce a suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features. Modifications to a state-of-the-art SAE variant, Top-K SAEs, are evaluated across multiple dimensions. In particular, we assess the generalizability of SAEs trained on base models to longer contexts and fine-tuned models. Additionally, we analyze the geometry of learned SAE latents, confirming that \emph{feature splitting} enables the discovery of new features. The Llama Scope SAE checkpoints are publicly available at~\url{https://huggingface.co/fnlp/Llama-Scope}, alongside our scalable training, interpretation, and visualization tools at \url{https://github.com/OpenMOSS/Language-Model-SAEs}. These contributions aim to advance the open-source Sparse Autoencoder ecosystem and support mechanistic interpretability research by reducing the need for redundant SAE training.

cs.LG↗

"A good pun is its own reword": Can Large Language Models Understand Puns?

Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.

cs.CL↗

A strongly interacting, two-dimensional, dipolar spin ensemble in (111)-oriented diamond

Systems of spins with strong dipolar interactions and controlled dimensionality enable new explorations in quantum sensing and simulation. In this work, we investigate the creation of strong dipolar interactions in a two-dimensional ensemble of nitrogen-vacancy (NV) centers generated via plasma-enhanced chemical vapor deposition (PECVD) on (111)-oriented diamond substrates. We find that diamond growth on the (111) plane yields high incorporation of spins, both nitrogen and NV centers, where the density of the latter is tunable via the miscut of the diamond substrate. Our process allows us to form dense, preferentially aligned, 2D NV ensembles with volume-normalized AC sensitivity down to $η_{AC}$ = 810 pT um$^{3/2}$ Hz$^{-1/2}$. Furthermore, we show that (111) affords maximally positive dipolar interactions amongst a 2D NV ensemble, which is crucial for leveraging dipolar-driven entanglement schemes and exploring new interacting spin physics.

quant-ph↗

Engineering Spin Coherence in Core-Shell Diamond Nanocrystals

Diamond nanocrystals can harbor spin qubit sensors capable of probing the physical properties of biological systems with nanoscale spatial resolution. These diamond nanosensors can readily be delivered into intact cells and even living organisms. However, applications beyond current proof-of-principle experiments require a substantial increase in sensitivity, which is generally limited by surface-noise-induced spin dephasing and relaxation. In this work, we significantly reduce magnetic surface noise by engineering core-shell structures, which in combination with dynamical decoupling result in qubit coherence times (T2) ranging from 52us to 87us - a drastic improvement over the 1.1us to 35us seen in bare particles. This improvement in spin coherence, combined with an overall increase in particle fluorescence, corresponds to a two-order-of-magnitude reduction in integration time. Probing qubit dynamics at a single particle level, furthermore, reveals that the noise characteristics fundamentally change from a bath with spins that rearrange their spatial configuration during the course of an experiment to a more dilute static bath. The observed results shed light on the underlying mechanisms governing spin dephasing in diamond nanocrystals and offer an effective noise mitigation strategy based on engineered core-shell structures.

quant-ph↗