Searcharxiv⌕ Search

arXiv subjects

Honglu Fan

Publications and source records attributed to Honglu Fan.

18 recordsLinked to original sources
arXiv2025

Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task

We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps. Models achieve high accuracy even with very small embedding dimensions, but larger dimensions yield more faithful, consistent, and robust internal representations. In particular, higher embedding dimensions strengthen the formation of structured internal representation and lead to better interpretability. After hundreds of experiments, we observe two consistent mechanisms: (1) the last row of the attention weight matrix monotonically encodes the global ordering of tokens; and (2) the selected transposition aligns with the largest adjacent difference of these encoded values. Our results provide quantitative evidence that transformers build structured internal world models and that model size improves representation quality in addition to end performance. We release our metrics and analyses, which can be used to probe similar algorithmic tasks.

cs.LG↗
arXiv2025

Scaling Self-Supervised Representation Learning for Symbolic Piano Performance

We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions. After first pretraining on approximately 60,000 hours of music, we use a comparatively smaller, high-quality subset, to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings by adapting the SimCLR framework to symbolic music. When evaluating piano continuation coherence, our generative model outperforms leading symbolic generation techniques and remains competitive with proprietary audio generation models. On MIR classification benchmarks, frozen representations from our contrastive model achieve state-of-the-art results in linear probe experiments, while direct finetuning demonstrates the generalizability of pretrained representations, often requiring only a few hundred labeled examples to specialize to downstream tasks.

cs.SD↗
arXiv2025

The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Large language models (LLMs) are typically trained on enormous quantities of unlicensed text, a practice that has led to scrutiny due to possible intellectual property infringement and ethical concerns. Training LLMs on openly licensed text presents a first step towards addressing these issues, but prior data collection efforts have yielded datasets too small or low-quality to produce performant LLMs. To address this gap, we collect, curate, and release the Common Pile v0.1, an eight terabyte collection of openly licensed text designed for LLM pretraining. The Common Pile comprises content from 30 sources that span diverse domains including research papers, code, books, encyclopedias, educational materials, audio transcripts, and more. Crucially, we validate our efforts by training two 7 billion parameter LLMs on text from the Common Pile: Comma v0.1-1T and Comma v0.1-2T, trained on 1 and 2 trillion tokens respectively. Both models attain competitive performance to LLMs trained on unlicensed text with similar computational budgets, such as Llama 1 and 2 7B. In addition to releasing the Common Pile v0.1 itself, we also release the code used in its creation as well as the training mixture and checkpoints for the Comma v0.1 models.

cs.CL↗
arXiv2025

When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts. In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}. To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators. Evaluating state-of-the-art LLMs on SPOT, we find that none surpasses 21.1\% recall or 6.1\% precision (o3 achieves the best scores, with all others near zero). Furthermore, confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability. Finally, qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings. These findings highlight the substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification.

cs.CL↗
arXiv2023

Grokking Group Multiplication with Cosets

The complex and unpredictable nature of deep neural networks prevents their safe use in many high-stakes applications. There have been many techniques developed to interpret deep neural networks, but all have substantial limitations. Algorithmic tasks have proven to be a fruitful test ground for interpreting a neural network end-to-end. Building on previous work, we completely reverse engineer fully connected one-hidden layer networks that have ``grokked'' the arithmetic of the permutation groups $S_5$ and $S_6$. The models discover the true subgroup structure of the full group and converge on neural circuits that decompose the group arithmetic using the permutation group's subgroups. We relate how we reverse engineered the model's mechanisms and confirmed our theory was a faithful description of the circuit's functionality. We also draw attention to current challenges in conducting interpretability research by comparing our work to Chughtai et al. [4] which alleges to find a different algorithm for this same problem.

cs.LG↗
arXiv2023

YaRN: Efficient Context Window Extension of Large Language Models

Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset. Code is available at https://github.com/jquesnelle/yarn

cs.CL↗
arXiv2023

An ML approach to resolution of singularities

The solution set of a system of polynomial equations typically contains ill-behaved, singular points. Resolution is a fundamental process in geometry in which we replace singular points with smooth points, while keeping the rest of the solution set unchanged. Resolutions are not unique: the usual way to describe them involves repeatedly performing a fundamental operation known as "blowing-up", and the complexity of the resolution highly depends on certain choices. The process can be translated into various versions of a 2-player game, the so-called Hironaka game, and a winning strategy for the first player provides a solution to the resolution problem. In this paper we introduce a new approach to the Hironaka game that uses reinforcement learning agents to find optimal resolutions of singularities. In certain domains, the trained model outperforms state-of-the-art selection heuristics in total number of polynomial additions performed, which provides a proof-of-concept that recent developments in machine learning have the potential to improve performance of algorithms in symbolic computation.

cs.LG↗
arXiv2023

Stay on topic with Classifier-Free Guidance

Classifier-Free Guidance (CFG) has recently emerged in text-to-image generation as a lightweight technique to encourage prompt-adherence in generations. In this work, we demonstrate that CFG can be used broadly as an inference-time technique in pure language modeling. We show that CFG (1) improves the performance of Pythia, GPT-2 and LLaMA-family models across an array of tasks: Q\&A, reasoning, code generation, and machine translation, achieving SOTA on LAMBADA with LLaMA-7B over PaLM-540B; (2) brings improvements equivalent to a model with twice the parameter-count; (3) can stack alongside other inference-time methods like Chain-of-Thought and Self-Consistency, yielding further improvements in difficult tasks; (4) can be used to increase the faithfulness and coherence of assistants in challenging form-driven and content-driven prompts: in a human evaluation we show a 75\% preference for GPT4All using CFG over baseline.

cs.CL↗
arXiv2020

Holomorphic anomaly equation for $(\mathbb{P}^2,E)$ and the Nekrasov-Shatashvili limit of local $\mathbb{P}^2$

We prove a higher genus version of the genus $0$ local-relative correspondence of van Garrel-Graber-Ruddat: for $(X,D)$ a pair with $X$ a smooth projective variety and $D$ a nef smooth divisor, maximal contact Gromov-Witten theory of $(X,D)$ with $\lambda_g$-insertion is related to Gromov-Witten theory of the total space of $\mathcal{O}_X(-D)$ and local Gromov-Witten theory of $D$. Specializing to $(X,D)=(S,E)$ for $S$ a del Pezzo surface or a rational elliptic surface and $E$ a smooth anticanonical divisor, we show that maximal contact Gromov-Witten theory of $(S,E)$ is determined by the Gromov-Witten theory of the Calabi-Yau 3-fold $\mathcal{O}_S(-E)$ and the stationary Gromov-Witten theory of the elliptic curve $E$. Specializing further to $S=\mathbb{P}^2$, we prove that higher genus generating series of maximal contact Gromov-Witten invariants of $(\mathbb{P}^2,E)$ are quasimodular and satisfy a holomorphic anomaly equation. The proof combines the quasimodularity results and the holomorphic anomaly equations previously known for local $\mathbb{P}^2$ and the elliptic curve. Furthermore, using the connection between maximal contact Gromov-Witten invariants of $(\mathbb{P}^2,E)$ and Betti numbers of moduli spaces of semistable one-dimensional sheaves on $\mathbb{P}^2$, we obtain a proof of the quasimodularity and holomorphic anomaly equation predicted in the physics literature for the refined topological string free energy of local $\mathbb{P}^2$ in the Nekrasov-Shatashvili limit.

math.AG↗
arXiv2019

Higher genus relative Gromov--Witten theory and DR-cycles

We extend the definition of relative Gromov--Witten invariants with negative contact orders to all genera. Then we show that relative Gromov--Witten theory forms a partial CohFT. Some cycle relations on the moduli space of stable maps are also proved.

math.AG↗
arXiv2018

Variations on the theme of quantum Lefschetz

In this companion piece to 1712.03573, some variations on the main results there are sketched. In particular, the recursions in 1712.03573, which we interpreted as the quantum Lefschetz, is reformulated in terms of Givental's quantization formalism, or equivalently, a summation of finitely many graphs. Meanwhile, varieties of modification of the auxilliary spaces (masterspaces) for the fixed point localization are given, leading to different (looking) recursions. There are also some applications of this circle of ideas to derive (apparently) new relations of Gromov--Witten invariants.

math.AG↗
arXiv2018

Mirror theorems for root stacks and relative pairs

Given a smooth projective variety $X$ with a smooth nef divisor $D$ and a positive integer $r$, we construct an $I$-function, an explicit slice of Givental's Lagrangian cone, for Gromov--Witten theory of the root stack $X_{D,r}$. As an application, we also obtain an $I$-function for relative Gromov--Witten theory following the relation between relative and orbifold Gromov--Witten invariants.

math.AG↗
arXiv2018

Structures in genus-zero relative Gromov--Witten theory

In this paper, we define genus-zero relative Gromov--Witten invariants with negative contact orders. Using this, we construct relative quantum cohomology rings and Givental formalism. A version of Virasoro constraints also follows from it.

math.AG↗
arXiv2017

Towards a quantum Lefschetz hyperplane theorem in all genera

An effective algorithm of determining Gromov--Witten invariants of smooth hypersurfaces in any genus (subject to a degree bound) from Gromov--Witten invariants of the ambient space is proposed. The Appendix is joint with E. Schulte-Geers.

math.AG↗
arXiv2017

A quantum splitting principle and an application

We propose an analogy of splitting principle in genus $0$ Gromov--Witten theory. More precisely, we show how the Gromov--Witten theory of a variety $X$ can be embedded into the theory of the projectivization of a vector bundle over $X$. An application is also given.

math.AG↗
arXiv2017

Chern classes and Gromov--Witten theory of projective bundles

We prove that the Gromov--Witten theory (GWT) of a projective bundle can be determined by the Chern classes and the GWT of the base. It completely answers a question raised in a previous paper (arXiv:1607.00740). Its consequences include that the GWT of the blow-up of X at a smooth subvariety Z is uniquely determined by GWT of X, Z plus some topological data.

math.AG↗
arXiv2016

On Gromov-Witten theory of projective bundles

Given two equivariant vector bundles over an algebraic GKM manifold with the same equivariant Chern classes, we show that the genus zero equivariant Gromov--Witten theory of their projective bundles are naturally isomorphic.

math.AG↗