arXiv subjects

Nikhil Anand

Publications and source records attributed to Nikhil Anand.

At least 19 recordsLinked to original sources

arXiv2026

ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models

Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.

cs.CL↗

arXiv2025

LOTION: Smoothing the Optimization Landscape for Quantized Training

Optimizing neural networks for quantized objectives is fundamentally challenging because the quantizer is piece-wise constant, yielding zero gradients everywhere except at quantization thresholds where the derivative is undefined. Most existing methods deal with this issue by relaxing gradient computations with techniques like Straight Through Estimators (STE) and do not provide any guarantees of convergence. In this work, taking inspiration from Nesterov smoothing, we approximate the quantized loss surface with a continuous loss surface. In particular, we introduce LOTION, \textbf{L}ow-precision \textbf{O}ptimization via s\textbf{T}ochastic-no\textbf{I}se sm\textbf{O}othi\textbf{N}g, a principled smoothing framework that replaces the raw quantized loss with its expectation under unbiased randomized-rounding noise. In this framework, standard optimizers are guaranteed to converge to a local minimum of the loss surface. Moreover, when using noise derived from stochastic rounding, we show that the global minima of the original quantized loss are preserved. We empirically demonstrate that this method outperforms standard QAT on synthetic testbeds and on 150M- and 300M- parameter language models.

cs.LG↗

arXiv2025

Characterization and Mitigation of Training Instabilities in Microscaling Formats

Training large language models is an expensive, compute-bound process that must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-generation hardware accelerators increasingly support lower-precision arithmetic formats, such as the Microscaling (MX) formats introduced in NVIDIA's Blackwell architecture. These formats use a shared scale within blocks of parameters to extend representable range and perform forward/backward GEMM operations in reduced precision for efficiency gains. In this work, we investigate the challenges and viability of block-scaled precision formats during model training. Across nearly one thousand language models trained from scratch -- spanning compute budgets from $2 \times 10^{17}$ to $4.8 \times 10^{19}$ FLOPs and sweeping over a broad range of weight-activation precision combinations -- we consistently observe that training in MX formats exhibits sharp, stochastic instabilities in the loss, particularly at larger compute scales. To explain this phenomenon, we conduct controlled experiments and ablations on a smaller proxy model that exhibits similar behavior as the language model, sweeping across architectural settings, hyperparameters, and precision formats. These experiments motivate a simple model in which multiplicative gradient bias introduced by the quantization of layer-norm affine parameters and a small fraction of activations can trigger runaway divergence. Through \emph{in situ} intervention experiments on our proxy model, we demonstrate that instabilities can be averted or delayed by modifying precision schemes mid-training. Guided by these findings, we evaluate stabilization strategies in the LLM setting and show that certain hybrid configurations recover performance competitive with full-precision training. We release our code at https://github.com/Hither1/systems-scaling.

cs.LG↗

arXiv2025

Decomposing Elements of Problem Solving: What "Math" Does RL Teach?

Mathematical reasoning tasks have become prominent benchmarks for assessing the reasoning capabilities of LLMs, especially with reinforcement learning (RL) methods such as GRPO showing significant performance gains. However, accuracy metrics alone do not support fine-grained assessment of capabilities and fail to reveal which problem-solving skills have been internalized. To better understand these capabilities, we propose to decompose problem solving into fundamental capabilities: Plan (mapping questions to sequences of steps), Execute (correctly performing solution steps), and Verify (identifying the correctness of a solution). Empirically, we find that GRPO mainly enhances the execution skill-improving execution robustness on problems the model already knows how to solve-a phenomenon we call temperature distillation. More importantly, we show that RL-trained models struggle with fundamentally new problems, hitting a 'coverage wall' due to insufficient planning skills. To explore RL's impact more deeply, we construct a minimal, synthetic solution-tree navigation task as an analogy for mathematical problem-solving. This controlled setup replicates our empirical findings, confirming RL primarily boosts execution robustness. Importantly, in this setting, we identify conditions under which RL can potentially overcome the coverage wall through improved exploration and generalization to new solution paths. Our findings provide insights into the role of RL in enhancing LLM reasoning, expose key limitations, and suggest a path toward overcoming these barriers. Code is available at https://github.com/cfpark00/RL-Wall.

cs.AI↗

arXiv2024

Loss-to-Loss Prediction: Scaling Laws for All Datasets

While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply it to predict across different pre-training datasets and from pre-training data to downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More precisely, we find that there are simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses of two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law relationships can yield more accurate predictions than extrapolating single-dataset scaling laws.

cs.LG↗

arXiv2024

Mixture of Parrots: Experts improve memorization more than reasoning

The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what performance tradeoffs, if any, exist between MoEs and standard dense transformers. In this paper, we show that as we increase the number of experts (while fixing the number of active parameters), the memorization performance consistently increases while the reasoning capabilities saturate. We begin by analyzing the theoretical limitations of MoEs at reasoning. We prove that there exist graph problems that cannot be solved by any number of experts of a certain width; however, the same task can be easily solved by a dense model with a slightly larger width. On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data. We empirically validate these findings on synthetic graph problems and memory-intensive closed book retrieval tasks. Lastly, we pre-train a series of MoEs and dense transformers and evaluate them on commonly used benchmarks in math and natural language. We find that increasing the number of experts helps solve knowledge-intensive tasks, but fails to yield the same benefits for reasoning tasks.

cs.LG↗

arXiv2024

Dataset Difficulty and the Role of Inductive Bias

Motivated by the goals of dataset pruning and defect identification, a growing body of methods have been developed to score individual examples within a dataset. These methods, which we call "example difficulty scores", are typically used to rank or categorize examples, but the consistency of rankings between different training runs, scoring methods, and model architectures is generally unknown. To determine how example rankings vary due to these random and controlled effects, we systematically compare different formulations of scores over a range of runs and model architectures. We find that scores largely share the following traits: they are noisy over individual runs of a model, strongly correlated with a single notion of difficulty, and reveal examples that range from being highly sensitive to insensitive to the inductive biases of certain model architectures. Drawing from statistical genetics, we develop a simple method for fingerprinting model architectures using a few sensitive examples. These findings guide practitioners in maximizing the consistency of their scores (e.g. by choosing appropriate scoring methods, number of runs, and subsets of examples), and establishes comprehensive baselines for evaluating scores in the future.

cs.LG↗

arXiv2023

Influence Scores at Scale for Efficient Language Data Sampling

Modern ML systems ingest data aggregated from diverse sources, such as synthetic, human-annotated, and live customer traffic. Understanding \textit{which} examples are important to the performance of a learning algorithm is crucial for efficient model training. Recently, a growing body of literature has given rise to various "influence scores," which use training artifacts such as model confidence or checkpointed gradients to identify important subsets of data. However, these methods have primarily been developed in computer vision settings, and it remains unclear how well they generalize to language-based tasks using pretrained models. In this paper, we explore the applicability of influence scores in language classification tasks. We evaluate a diverse subset of these scores on the SNLI dataset by quantifying accuracy changes in response to pruning training data through random and influence-score-based sampling. We then stress-test one of the scores -- "variance of gradients" (VoG) from Agarwal et al. (2022) -- in an NLU model stack that was exposed to dynamic user speech patterns in a voice assistant type of setting. Our experiments demonstrate that in many cases, encoder-based language models can be finetuned on roughly 50% of the original data without degradation in performance metrics. Along the way, we summarize lessons learned from applying out-of-the-box implementations of influence scores, quantify the effects of noisy and class-imbalanced data, and offer recommendations on score-based sampling for better accuracy and training efficiency.

cs.LG↗

arXiv2023

Comprehensive Benchmarking of Entropy and Margin Based Scoring Metrics for Data Selection

While data selection methods have been studied extensively in active learning, data pruning, and data augmentation settings, there is little evidence for the efficacy of these methods in industry scale settings, particularly in low-resource languages. Our work presents ways of assessing prospective training examples in those settings for their "usefulness" or "difficulty". We also demonstrate how these measures can be used in selecting important examples for training supervised machine learning models. We primarily experiment with entropy and Error L2-Norm (EL2N) scores. We use these metrics to curate high quality datasets from a large pool of \textit{Weak Signal Labeled} data, which assigns no-defect high confidence hypotheses during inference as ground truth labels. We then conduct training data augmentation experiments using these de-identified datasets and demonstrate that score-based selection can result in a 2% decrease in semantic error rate and 4%-7% decrease in domain classification error rate when compared to the baseline technique of random selection.

cs.LG↗

arXiv2021

Chiral Limit of 2d QCD Revisited with Lightcone Conformal Truncation

We study the chiral limit of 2d QCD with a single quark flavor at finite $N_c$ using LCT. By modifying the LCT basis according to the quark mass in a manner motivated by 't Hooft's analysis, we are able to restore convergence for quark masses much smaller than the QCD strong coupling scale. For such small quark masses, the IR of the theory is expected to be well described by the Sine-Gordon model. We verify that LCT numerics are able to capture in detail the spectrum and correlation functions of the Sine-Gordon model. This opens up the possibility for studying deformations of various integrable CFTs using LCT by considering the chiral limit of QCD like theories.

hep-th↗

arXiv2021

Nonperturbative dynamics of (2+1)d $ϕ^4$-theory from Hamiltonian truncation

We use Lightcone Conformal Truncation (LCT) -- a version of Hamiltonian truncation -- to study the nonperturbative, real-time dynamics of $ϕ^4$-theory in 2+1 dimensions. This theory has UV divergences that need to be regulated. We review how, in a Hamiltonian framework with a total energy cutoff, renormalization is necessarily \emph{state-dependent}, and UV sensitivity cannot be canceled with standard local operator counterterms. To overcome this problem, we present a prescription for constructing the appropriate state-dependent counterterms for (2+1)d $ϕ^4$-theory in lightcone quantization. We then use LCT with this counterterm prescription to study $ϕ^4$-theory, focusing on the $\mathbb{Z}_2$ symmetry-preserving phase. Specifically, we compute the spectrum as a function of the coupling and demonstrate the closing of the mass gap at a (scheme-dependent) critical coupling. We also compute Lorentz-invariant two-point functions, both at generic strong coupling and near the critical point, where we demonstrate IR universality and the vanishing of the trace of the stress tensor.

hep-th↗

arXiv2020

Momentum space CFT correlators for Hamiltonian truncation

We consider Lorentzian CFT Wightman functions in momentum space. In particular, we derive a set of reference formulas for computing two- and three-point functions, restricting our attention to three-point functions where the middle operator (corresponding to a Hamiltonian density) carries zero spatial momentum, but otherwise allowing operators to have arbitrary spin. A direct application of our formulas is the computation of Hamiltonian matrix elements within the framework of conformal truncation, a recently proposed method for numerically studying strongly-coupled QFTs in real time and infinite volume. Our momentum space formulas take the form of finite sums over ${}_2F_1$ hypergeometric functions, allowing for efficient numerical evaluation. As a concrete application, we work out matrix elements for 3d $ϕ^4$-theory, thus providing the seed ingredients for future truncation studies.

hep-th↗

arXiv2020

Introduction to Lightcone Conformal Truncation: QFT Dynamics from CFT Data

We both review and augment the lightcone conformal truncation (LCT) method. LCT is a Hamiltonian truncation method for calculating dynamical quantities in QFT in infinite volume. This document is a self-contained, pedagogical introduction and "how-to" manual for LCT. We focus on 2D QFTs which have UV descriptions as free CFTs containing scalars, fermions, and gauge fields, providing a rich starting arena for LCT applications. Along our way, we develop several new techniques and innovations that greatly enhance the efficiency and applicability of LCT. These include the development of CFT radial quantization methods for computing Hamiltonian matrix elements and a new SUSY-inspired way of avoiding state-dependent counterterms and maintaining chiral symmetry. We walk readers through the construction of their own basic LCT code, sufficient for small truncation cutoffs. We also provide a more sophisticated and comprehensive set of Mathematica packages and demonstrations that can be used to study a variety of 2D models. We guide the reader through these packages with several examples and illustrate how to obtain QFT observables, such as spectral densities and the Zamolodchikov $C$-function. Specific models considered are finite $N_c$ QCD, scalar $ϕ^4$ theory, and Yukawa theory.

hep-th↗

arXiv2017

An Exact Operator That Knows Its Location

We use conformal symmetry to define an AdS$_3$ proto-field $ϕ$ as an exact linear combination of Virasoro descendants of a CFT$_2$ primary operator $\mathcal{O}$. We find that both symmetry considerations and a gravitational Wilson line formalism lead to the same results. The operator $ϕ$ has many desirable properties; in particular it has correlators that agree with gravitational perturbation theory when expanded at large $c$, and that automatically take the correct form in all vacuum AdS$_3$ geometries, including BTZ black hole backgrounds. In the future it should be possible to use $ϕ$ to probe bulk locality and black hole horizons at a non-perturbative level.

hep-th↗

arXiv2017

Merger delay time distribution of extended emission short GRBs

The most popular progenitor model for short duration Gamma-Ray bursts (sGRBs) is the merger of two compact objects. However, the short GRB population exhibit a certain diversity: some bursts display an extended emission (EE), continuing in soft $γ$-rays for a few hundreds of seconds post the initial short pulse. It is currently unclear whether the origin of such bursts is linked to compact object mergers. Within the merger hypothesis, the redshift ($z$) distribution of short GRBs is influenced by the merger delay time, i.e., time elapsed between the merger and the formation of the binary star system, which is dominated by the time-scale for gravitational wave losses during the compact binary phase. We examine redshift distributions of short GRBs with extended emission to see whether their formation channel requires considerable delay post the star formation episode. Our results show that the $z$ distribution of EE bursts is consistent with the merger model. We attempted to compare the delay time distribution of the EE and the non-EE short bursts. However, no statistically significant difference could be seen within the limited sample size.

astro-ph.HE↗

arXiv2017

RG Flow from $ϕ^4$ Theory to the 2D Ising Model

We study 1+1 dimensional $ϕ^4$ theory using the recently proposed method of conformal truncation. Starting in the UV CFT of free field theory, we construct a complete basis of states with definite conformal Casimir, $\mathcal{C}$. We use these states to express the Hamiltonian of the full interacting theory in lightcone quantization. After truncating to states with $\mathcal{C} \leq \mathcal{C}_{\max}$, we numerically diagonalize the Hamiltonian at strong coupling and study the resulting IR dynamics. We compute non-perturbative spectral densities of several local operators, which are equivalent to real-time, infinite-volume correlation functions. These spectral densities, which include the Zamolodchikov $C$-function along the full RG flow, are calculable at any value of the coupling. Near criticality, our numerical results reproduce correlation functions in the 2D Ising model.

hep-th↗

arXiv2015

The Goldstone Equivalence Theorem and AdS/CFT

The Goldstone equivalence theorem allows one to relate scattering amplitudes of massive gauge fields to those of scalar fields in the limit of large scattering energies. We generalize this theorem under the framework of the AdS/CFT correspondence. First, we obtain an expression of the equivalence theorem in terms of correlation functions of creation and annihilation operators by using an AdS wave function approach to the AdS/CFT dictionary. It is shown that the divergence of the non-conserved conformal current dual to the bulk gauge field is approximately primary when computing correlators for theories in which the masses of all the exchanged particles are sufficiently large. The results are then generalized to higher spin fields. We then go on to generalize the theorem using conformal blocks in two and four-dimensional CFTs. We show that when the scaling dimensions of the exchanged operators are large compared to both their spins and the dimension of the current, the conformal blocks satisfy an equivalence theorem.

hep-th↗

arXiv2014

Model-independent Analyses of Dark-Matter Particle Interactions

A model-independent treatment of dark-matter particle elastic scattering has been developed, yielding the most general interaction for WIMP-nucleon low-energy scattering, and the resulting amplitude has been embedded in the nucleus, taking into account the selection rules imposed by parity and time-reversal. One finds that, in contrast to the usual spin-independent/spin-dependent (SI/SD) formulation, the resulting cross section contains six independent nuclear response functions, three of which are associated with possible velocity-dependent interactions. We find that current experiments are four orders of magnitude more sensitive to derivative couplings than is apparent in the standard SI/SD treatment, which necessarily associates such interactions with cross sections proportional to the square of the WIMP velocity relative to the nuclear center of mass.

nucl-th↗