arXiv subjects

Julie Huang

Publications and source records attributed to Julie Huang.

3 recordsLinked to original sources

arXiv2026

When Thoughts Become Memory: Unifying In-Context Learning and Fine-Tuning

We are the first to give a certified rule for whether unseen data belong in a pretrained model's context or its weights, built on an exact law for their information value. That value, the code length saved on held-out queries, equals in expectation the bits stored about the query, plus a prior repair, minus a decoding loss, for any model, architecture, dataset and task. The law's ceilings certify the choice from one measurement, and an anytime-valid test makes it from held-out queries, with error controlled whenever it stops. Every certificate the ceilings issue on a trained transformer, TabPFN, an occupancy network and a stereo network is correct. The test picks TabPFN over XGBoost and a trained surrogate over an in-context learner on airfoil pressure, calls the better of GPT-2's cache and fine-tune in 61,948 of 64,000 query streams over 32 comparisons, and errs in at most 3.3% of null streams. The law also diagnoses when learning in context helps and when it does not. Deleting part of a context is held-out validation of that part in any model, and adding parts one at a time traces the learning curve in context: one patch that sees a shape feature cuts an occupancy network's code length there a hundredfold, while eight that miss it nearly double it, and one camera that sees the surface beats four in a stereo network. Ranked by this label-free valuation, a kriging learner's three best support cars beat four drawn at random, and on a yes-or-no claim the valuation recovers a published evidence adjudicator, with AUROC 0.91.

stat.ML↗

arXiv2026

Exact Finite Attention Responses From RoPE Derivatives

We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative $\partial_p z(p) = A z(p)$: its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term $C_{KV} = \sum_j (p'_j - p_j)\,\varepsilon_j$, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.

stat.ML↗

arXiv2026

RoPE attention is an exact forward-pass gradient step with softmax intact

We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix $ΔM_i$ satisfying $y_i = μ_i + u_i^\top ΔM_i$, where $μ_i$ is the uniform mean of the attended values and $u_i$ is the augmented query input. The construction applies the classical exponential divided difference $ρ= ϕ_1$ to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.

stat.ML↗