arXiv subjects

Michael Y. Hu

Publications and source records attributed to Michael Y. Hu.

At least 19 recordsLinked to original sources

arXiv2026

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time

Data mixing decides how to combine different sources or types of data and is a consequential problem throughout language model training. In pretraining, data composition is a key determinant of model quality; in continual learning and adaptation, it governs what is retained and acquired. Yet existing data mixing methods address only one phase of this lifecycle at a time: some require smaller proxy models tied to a single training phase, others assume a fixed domain set, and continual learning lacks principled guidance altogether. We argue that data mixing is fundamentally an online decision making problem -- one that recurs throughout training and demands a single, unified solution. We introduce OP-Mix (On-Policy Mix), a data mixing algorithm that operates across the entire language model training lifecycle. Our main insight is that candidate data mixtures can be cheaply simulated by interpolating between low-rank adapters trained directly on the current model, eliminating separate proxy models and ensuring the search is always grounded in the model's actual learning dynamics. Across pretraining, continual midtraining, and continual instruction tuning, OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of the baselines. In pretraining, OP-Mix improves upon training without mixing by 6.3% in average perplexity. For continual learning, OP-Mix matches the performance of both retraining and on-policy distillation while using 66% and 95% less overall compute, respectively. OP-Mix suggests a different view of language model training: not a sequence of distinct phases, but a single continuous process of learning from data.

cs.CL↗

arXiv2026

Neural Neural Scaling Laws

Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors: some improve monotonically, others plateau, and some even degrade with scale. We argue that predicting downstream performance from validation loss suffers from two limitations: averaging token-level losses obscures signal, and no simple parametric family can capture the full spectrum of scaling behaviors. To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without the limitations inherent in assuming a specific functional form. Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 1.99% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 44% reduction compared to logistic scaling laws (3.56% MAE). Furthermore, NeuNeu generalizes zero-shot to unseen model families, architectures, parameter counts, and downstream tasks. Our work suggests that predicting downstream scaling directly from data outperforms parametric alternatives.

cs.LG↗

arXiv2026

RELIC: Evaluating Complex Reasoning via the Recognition of Languages In-Context

Large language models (LLMs) are increasingly used to solve complex tasks where they must retrieve and compose many pieces of in-context information in long reasoning chains. For many real-world tasks it is hard to accurately gauge how model performance and strategy change as task complexity grows. To evaluate models' complex reasoning capability in a scalable and verifiable way, we introduce RELIC (Recognition of Languages In-Context), a framework that evaluates an LLM's ability to decide whether a given string belongs to the context-free language (CFL) generated by a grammar presented in-context. CFL recognition allows us to modulate the intrinsic complexity of the problem by varying grammar size and string length and translate this asymptotic complexity into predictions for ideal LLM performance. We find that even the most advanced reasoning models perform poorly on RELIC, not only failing to appropriately scale their inference compute to keep pace with task difficulty, but even reducing the number of reasoning tokens they use as task complexity increases. We find that these decreases in compute accompany changes in reasoning strategy, as models move from identifying and implementing algorithmic solutions to guessing. For models whose full completions go uninspected, this manifests as ``quiet quitting'' on hard tasks.

cs.CL↗

arXiv2026

Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting

Language model (LM) agents deployed in novel environments often exhibit poor sample efficiency when learning from sequential interactions. This significantly hinders the usefulness of such agents in environments where interaction is costly (for example, when they interact with humans or reset physical systems). While a number of existing LM agent architectures incorporate various mechanisms for experience storage and reflection, they make limited use of LMs' abilities to directly generate or reason about full counterfactual trajectories. We introduce ECHO (Experience Consolidation via Hindsight Optimization), a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents. ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts, effectively creating synthetic positive examples from unsuccessful interactions. Our approach consists of two components: a hindsight rule that uses the language model itself to identify relevant subgoals and generate optimized trajectories, and an update rule that maintains compressed trajectory representations in memory. We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation. Across both domains, ECHO outperforms vanilla language agent baselines by up to 80%; in XMiniGrid, it also outperforms a number of sophisticated agent architectures including Reflexion and AWM, demonstrating faster adaptation to novel environments through more effective utilization of past experiences.

cs.LG↗

arXiv2025

Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check

Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.

cs.CL↗

arXiv2025

Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases

Pretraining language models on formal language can improve their acquisition of natural language. Which features of the formal language impart an inductive bias that leads to effective transfer? Drawing on insights from linguistics and complexity theory, we hypothesize that effective transfer occurs when two conditions are met: the formal language should capture the dependency structures present in natural language, and it should remain within the computational limitations of the model architecture. We experiment with pre-pretraining (training on formal language before natural languages) on transformers and find that formal languages capturing hierarchical dependencies indeed enable language models to achieve lower loss on natural language and better linguistic generalization compared to other formal languages. We also find modest support for the hypothesis that the formal language should fall within the computational limitations of the architecture. Strikingly, pre-pretraining reduces loss more efficiently than training on a matched amount of natural language. For a 1B-parameter language model trained on roughly 1.6B tokens of natural language, pre-pretraining achieves the same loss and better linguistic generalization with a 33% smaller token budget. Finally, we also give mechanistic evidence of transfer from formal to natural language: attention heads acquired during pre-pretraining remain crucial for the model's performance on syntactic evaluations.

cs.CL↗

arXiv2025

Aioli: A Unified Optimization Framework for Language Model Data Mixing

Language model performance depends on identifying the optimal mixture of data groups to train on (e.g., law, code, math). Prior work has proposed a diverse set of methods to efficiently learn mixture proportions, ranging from fitting regression models over training runs to dynamically updating proportions throughout training. Surprisingly, we find that no existing method consistently outperforms a simple stratified sampling baseline in terms of average test perplexity. To understand this inconsistency, we unify existing methods into a standard framework, showing they are equivalent to solving a common optimization problem: minimize average loss subject to a method-specific mixing law -- an implicit assumption on the relationship between loss and mixture proportions. This framework suggests that measuring the fidelity of a method's mixing law can offer insights into its performance. Empirically, we find that existing methods set their mixing law parameters inaccurately, resulting in the inconsistent mixing performance we observe. Using this insight, we derive a new online method named Aioli, which directly estimates the mixing law parameters throughout training and uses them to dynamically adjust proportions. Aioli outperforms stratified sampling on 6 out of 6 datasets by an average of 0.27 test perplexity points, whereas existing methods fail to consistently beat stratified sampling, doing up to 6.9 points worse. Moreover, in a practical setting where proportions are learned on shorter runs due to computational constraints, Aioli can dynamically adjust these proportions over the full training run, consistently improving performance over existing methods by up to 12.012 test perplexity points.

cs.LG↗

arXiv2024

Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model training on a fixed language data budget of 100 million words or less. This year, we released improved text corpora, as well as a vision-and-language corpus to facilitate research into cognitively plausible vision language models. Submissions were compared on evaluation tasks targeting grammatical ability, (visual) question answering, pragmatic abilities, and grounding, among other abilities. Participants could submit to a 10M-word text-only track, a 100M-word text-only track, and/or a 100M-word and image multimodal track. From 31 submissions employing diverse methods, a hybrid causal-masked language model architecture outperformed other approaches. No submissions outperformed the baselines in the multimodal track. In follow-up analyses, we found a strong relationship between training FLOPs and average performance across tasks, and that the best-performing submissions proposed changes to the training data, training objective, and model architecture. This year's BabyLM Challenge shows that there is still significant room for innovation in this setting, in particular for image-text modeling, but community-driven research can yield actionable insights about effective strategies for small-scale language modeling.

cs.CL↗

arXiv2024

[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the competition rules will be different. The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques. Second, we are relaxing the rules around pretraining data, and will now allow participants to construct their own datasets provided they stay within the 100M-word or 10M-word budget. Third, we introduce a multimodal vision-and-language track, and will release a corpus of 50% text-only and 50% image-text multimodal data as a starting point for LM model training. The purpose of this CfP is to provide rules for this year's challenge, explain these rule changes and their rationale in greater detail, give a timeline of this year's competition, and provide answers to frequently asked questions from last year's challenge.

cs.CL↗

arXiv2024

Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction

Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.

cs.AI↗

arXiv2024

Latent State Models of Training Dynamics

The impact of randomness on model training is poorly understood. How do differences in data order and initialization actually manifest in the model, such that some training runs outperform others or converge faster? Furthermore, how can we interpret the resulting training dynamics and the phase transitions that characterize different trajectories? To understand the effect of randomness on the dynamics and outcomes of neural network training, we train models multiple times with different random seeds and compute a variety of metrics throughout training, such as the $L_2$ norm, mean, and variance of the neural network's weights. We then fit a hidden Markov model (HMM) over the resulting sequences of metrics. The HMM represents training as a stochastic process of transitions between latent states, providing an intuitive overview of significant changes during training. Using our method, we produce a low-dimensional, discrete representation of training dynamics on grokking tasks, image classification, and masked language modeling. We use the HMM representation to study phase transitions and identify latent "detour" states that slow down convergence.

cs.LG↗

arXiv2023

Crystal Structure and Phonon Density of States of FeSi up to 120 GPa

The strongly correlated material FeSi exhibits several unusual thermal, magnetic, and structural properties under varying pressure-temperature (P-T) conditions. It is a potential thermoelectric alloy and a materials of several geological implications as a possible constituent at the Earth's core mantle boundary (CMB). The phase transition behavior and lattice dynamics of FeSi under different P-T conditions remain elusive. A previous theoretical work predicted a pressure induced B20-B2 transition at ambient temperature, yet the transition is only observed at high P-T conditions in the experiments. Furthermore, the closing of the electronic gap due to a dramatic renormalization of the electronic structure and phonon anomalies has been reported based on density function calculations. In this study we have performed high pressure powder diffraction and Nuclear Resonant Inelastic X-ray Scattering (NRIXS) measurements up to 120 GPa to understand the phase stability and the lattice dynamics. Our study shows evidence for a nonhydrostatic stress induced B20-B2 transition in FeSi around 36 GPa for the first time. The Fe partial phonon density of states (PDOS) and thermal parameters were derived from NRIXS up to 120 GPa with the density function theoretical (DFT) calculations. These calculations further predict and are consistent with pressure-induced metallization and band gap closing around 12 GPa.

cond-mat.mtrl-sci↗

arXiv2023

Using Natural Language and Program Abstractions to Instill Human Inductive Biases in Machines

Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning may sometimes acquire very different strategies from humans. We show that co-training these agents on predicting representations from natural language task descriptions and programs induced to generate such tasks guides them toward more human-like inductive biases. Human-generated language descriptions and program induction models that add new learned primitives both contain abstract concepts that can compress description length. Co-training on these representations result in more human-like behavior in downstream meta-reinforcement learning agents than less abstract controls (synthetic language descriptions, program induction without learned primitives), suggesting that the abstraction supported by these representations is key.

cs.AI↗

arXiv2020

Interface-related magnetic and vibrational properties in Fe/MgO heterostructures from nuclear resonant spectroscopy and first-principles calculations

We combine $^{57}$Fe Mössbauer spectroscopy and $^{57}$Fe nuclear resonant inelastic x-ray scattering (NRIXS) in nanoscale polycrystalline [bcc-$^{57}$Fe/MgO] multilayers with various Fe layer thicknesses and layer-resolved density-functional-theory (DFT) based first-principles calculations of a (001)-oriented [Fe(8 ML)/MgO(8 ML)](001) heterostructure to unravel the interface-related atomic vibrational properties of a multilayer system. In theory and experiment, we observe consistently enhanced hyperfine magnetic fields compared to bulk which are associated with the Fe/MgO interface layers. NRIXS and DFT both reveal a strong reduction of the longitudinal acoustic (LA) phonon peak in combination with an enhancement of the low-energy vibrational density of states (VDOS) suggesting that the presence of interfaces and the associated increase in the layer-resolved magnetic moments results in drastic changes in the Fe-partial VDOS. From the experimental and calculated VDOS, vibrational thermodynamic properties have been determined as a function of Fe thickness and are found to be in excellent agreement.

cond-mat.mtrl-sci↗

arXiv2019

Influence of hydrogenation on the vibrational density of states of magnetocaloric $\mathrm{LaFe}_\mathrm{11.4}\mathrm{Si}_\mathrm{1.6}\mathrm{H}_{1.6}$

We report on the impact of magnetoelastic coupling on the magnetocaloric properties of LaFe$_{11.4}$Si$_{1.6}$H$_{1.6}$ in terms of the vibrational density of states, which we determined with $^{57}$Fe nuclear resonant inelastic X-ray scattering measurements and with density-functional-theory based first-principles calculations in the ferromagnetic low-temperature and paramagnetic high-temperature phase. In experiments and calculations, we observe pronounced differences in the shape of the Fe-partial VDOS between non-hydrogenated and hydrogenated samples. This shows that hydrogen does not only shift the temperature of the first-order phase transition, but also affects the elastic response of the Fe-subsystem significantly. In turn, the anomalous redshift of the Fe VDOS, observed by going to the low-volume PM phase, survives hydrogenation. As a consequence, the change in the Fe specific vibrational entropy $ΔS_\mathrm{lat}$ across the phase transition has the same sign as the magnetic and electronic contribution. DFT calculations show that the same mechanism, which is a consequence of the itinerant electron metamagnetism associated with the Fe subsystem, is effective in both the hydrogenated and he hydrogen-free compounds. Although reduced by 50 % as compared to the hydrogen-free system, the measured change $ΔS_\mathrm{lat}$ of 3.2\pm1.9 J/kgK across the FM to PM transition contributes with 35 % significantly and cooperatively to the total isothermal entropy change $ΔS_\mathrm{iso}$. Hydrogenation is observed to induce an overall blueshift of the Fe-VDOS with respect to the H-free compound; this effect, together with the enhanced Debye temperature observed, is a fingerprint of the hardening of the Fe sublattice by hydrogen incorporation. In addition, the mean Debye velocity of sound of LaFe$_{11.4}$Si$_{1.6}$H$_{1.6}$ was determined from the NRIXS and the DFT data.

cond-mat.mtrl-sci↗

arXiv2018

Size-dependent lattice dynamics of atomically precise cadmium selenide quantum dots

Material properties depend sensitively on the atomic arrangements and atomic bonding, but these are notoriously difficult to measure in nanosized atomic clusters due to the small size of the objects and the challenge of obtaining bulk samples of identical clusters. Here we have combined the recent ability to make gram quantities of identical semiconductor quantum-dot nanoparticles with the ability to measure lattice dynamics on small sample quantities of hydrogenated materials using high energy resolution inelastic x-ray scattering (HERIX), to measure the size-dependence of the phonon density of states (PDOS) in CdSe quantum dots. The fact that we have atomically precise structural models for these nanoparticles allows the calculation of the PDOS using Density Functional Theory (DFT), providing both experimental and theoretical confirmations of the important role that the inertia of the surface capping species plays on determining the lattice dynamics.

cond-mat.mtrl-sci↗

arXiv2018

Microscopic phase diagram of LaFeAsO single crystals under pressure

We investigated LaFeAsO single crystals by means of synchrotron Mössbauer spectroscopy under pressures up to 7.5 GPa and down to 13 K and provide a microscopic phase diagram. We found a continuous suppression of the magnetic hyperfine field with increasing pressure and it completely vanishes at $\sim$ 7.5 GPa which is in contrast to the behavior in polycrystalline samples where the magnetic order vanishes at $\sim$ 20 GPa. The different behavior of the polycrystalline samples might be due to As-vacancies. Our results are in qualitative agreement with density functional theory calculations where a reduction of the magnetic moment with increasing pressure was found. We found that among different samples at ambient pressure the magnetic phase transition temperature as well as the low-temperature magnetic hyperfine field decrease with increasing unit cell volume.

cond-mat.supr-con↗

arXiv2016

Impact of lattice dynamics on the phase stability of metamagnetic FeRh: Bulk and thin films

We present phonon dispersions, element-resolved vibrational density of states (VDOS) and corresponding thermodynamic properties obtained by a combination of density functional theory (DFT) and nuclear resonant inelastic X-ray scattering (NRIXS) across the metamagnetic transition of B2 FeRh in the bulk material and thin epitaxial films. We see distinct differences in the VDOS of the antiferromagnetic (AF) and ferromagnetic (FM) phase which provide a microscopic proof of strong spin-phonon coupling in FeRh. The FM VDOS exhibits a particular sensitivity to the slight tetragonal distortions present in epitaxial films, which is not encountered in the AF phase. This results in a notable change in lattice entropy, which is important for the comparison between thin film and bulk results. Our calculations confirm the recently reported lattice instability in the AF phase. The imaginary frequencies at the $X$-point depend critically on the Fe magnetic moment and atomic volume. Analyzing these non vibrational modes leads to the discovery of a stable monoclinic ground state structure which is robustly predicted from DFT but not verified in our thin film experiments. Specific heat, entropy and free energy calculated within the quasiharmonic approximation suggest that the new phase is possibly suppressed because of its relatively smaller lattice entropy. In the bulk phase, lattice degrees of freedom contribute with the same sign and in similar magnitude to the isostructural AF-FM phase transition as the electronic and magnetic subsystems and therefore needs to be included in thermodynamic modeling.

cond-mat.mtrl-sci↗