arXiv subjects

Nolan Miller

Publications and source records attributed to Nolan Miller.

10 recordsLinked to original sources

arXiv2025

Long Context In-Context Compression by Getting to the Gist of Gisting

Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression method with no architectural modification to the decoder transformer, is a promising approach due to its simplicity and compatibility with existing frameworks. While effective for short instructions, we demonstrate that gisting struggles with longer contexts, with significant performance drops even at minimal compression rates. Surprisingly, a simple average pooling baseline consistently outperforms gisting. We analyze the limitations of gisting, including information flow interruptions, capacity limitations and the inability to restrict its attention to subsets of the context. Motivated by theoretical insights into the performance gap between gisting and average pooling, and supported by extensive experimentation, we propose GistPool, a new in-context compression method. GistPool preserves the simplicity of gisting, while significantly boosting its performance on long context compression tasks.

cs.LG↗

arXiv2024

The hadronic vacuum polarization contribution to the muon $g-2$ at long distances

We present our lattice QCD result for the long-distance part of the hadronic vacuum polarization contribution, $(a_\mu^{\rm hvp})^{\rm LD}$, to the muon $g-2$ in the time-momentum representation. This is the numerically dominant, and at the same time the most challenging part regarding statistical precision. Our calculation is based on ensembles with dynamical up, down and strange quarks, employing the O($a$)-improved Wilson fermion action with lattice spacings ranging from $0.035-0.099$ fm. In order to reduce statistical noise in the long-distance part of the correlator to the per-mille level, we apply low-mode averaging and combine it with an explicit spectral reconstruction. Our result is $(a_\mu^{\rm hvp})^{\rm LD} = 423.2(4.2)_{\rm stat}(3.4)_{\rm syst}\times 10^{-10}$ in isospin-symmetric QCD, where the pion decay constant is used to set the energy scale. When combined with our previous results for the short- and intermediate-distance window observables and after including all sub-dominant contributions as well as isospin-breaking corrections, we obtain the total leading-order hadronic vacuum polarization contribution as $a_\mu^{\rm hvp} = 724.5(4.9)_{\rm stat}(5.2)_{\rm syst}\times 10^{-10}$. Our result displays a tension of 3.9 standard deviations with the data-driven estimate published in the 2020 White Paper, but leads to a SM prediction for the total muon anomalous magnetic moment that agrees with the current experimental average.

hep-lat↗

arXiv2024

Narrowing the Focus: Learned Optimizers for Pretrained Models

In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand-designed and tuning their hyperparameters is a big part of the training process. Learned optimizers have shown some initial promise, but are generally unsuccessful as a general optimization mechanism applicable to every problem. In this work we explore a different direction: instead of learning general optimizers, we instead specialize them to a specific training environment. We propose a novel optimizer technique that learns a layer-specific linear combination of update directions provided by a set of base optimizers, effectively adapting its strategy to the specific model and dataset. When evaluated on image classification tasks, this specialized optimizer significantly outperforms both traditional off-the-shelf methods such as Adam, as well as existing general learned optimizers. Moreover, it demonstrates robust generalization with respect to model initialization, evaluating on unseen datasets, and training durations beyond its meta-training horizon.

cs.LG↗

arXiv2023

Uncovering mesa-optimization algorithms in Transformers

Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. The origins of this phenomenon are still poorly understood. Here we analyze a series of Transformer models trained to perform synthetic sequence prediction tasks, and discover that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. We show that this process corresponds to gradient-based optimization of a principled objective function, which leads to strong generalization performance on unseen sequences. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers.

cs.LG↗

arXiv2023

Training trajectories, mini-batch losses and the curious role of the learning rate

Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. In this paper we propose to study the behavior of the loss function on fixed mini-batches along SGD trajectories. We show that the loss function on a fixed batch appears to be remarkably convex-like. In particular for ResNet the loss for any fixed mini-batch can be accurately modeled by a quadratic function and a very low loss value can be reached in just one step of gradient descent with sufficiently large learning rate. We propose a simple model that allows to analyze the relationship between the gradients of stochastic mini-batches and the full batch. Our analysis allows us to discover the equivalency between iterate aggregates and specific learning rate schedules. In particular, for Exponential Moving Average (EMA) and Stochastic Weight Averaging we show that our proposed model matches the observed training trajectories on ImageNet. Our theoretical model predicts that an even simpler averaging technique, averaging just two points a many steps apart, significantly improves accuracy compared to the baseline. We validated our findings on ImageNet and other datasets using ResNet architecture.

cs.LG↗

arXiv2022

Decentralized Learning with Multi-Headed Distillation

Decentralized learning with private data is a central problem in machine learning. We propose a novel distillation-based decentralized learning technique that allows multiple agents with private non-iid data to learn from each other, without having to share their data, weights or weight updates. Our approach is communication efficient, utilizes an unlabeled public dataset and uses multiple auxiliary heads for each client, greatly improving training efficiency in the case of heterogeneous data. This approach allows individual models to preserve and enhance performance on their private tasks while also dramatically improving their performance on the global aggregated data distribution. We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.

cs.LG↗

arXiv2022

The hyperon spectrum from lattice QCD

Hyperon decays present a promising alternative for extracting $\vert V_{us} \vert$ from lattice QCD combined with experimental measurements. Currently $\vert V_{us} \vert$ is determined from the kaon decay widths and a lattice calculation of the associated form factor. In this proceeding, I will present preliminary work on a lattice determination of the hyperon mass spectrum. I will additionally summarize future goals in which we will calculate the hyperon transition matrix elements, which will provide an alternative means for accessing $\vert V_{us} \vert$. This work is based on a particular formulation of SU(2) chiral perturbation theory for hyperons; determining the extent to which this effective field theory converges is instrumental in understanding the limits of its predictive power, especially since some hyperonic observables are difficult to calculate near the physical pion mass (e.g., hyperon-to-nucleon form factors), and thus the use of heavier than physical pion masses is likely to yield more precise results when combined with extrapolations to the physical point.}

hep-lat↗

arXiv2021

Meta-Learning Bidirectional Update Rules

In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule. In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional "genome". We show that such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.

cs.LG↗

arXiv2020

Scale setting the M\"obius Domain Wall Fermion on gradient-flowed HISQ action using the omega baryon mass and the gradient-flow scales $t_0$ and $w_0$

We report on a sub-percent scale determination using the omega baryon mass and gradient-flow methods. The calculations are performed on 22 ensembles of $N_f=2+1+1$ highly improved, rooted staggered sea-quark configurations generated by the MILC and CalLat Collaborations. The valence quark action used is M\"obius Domain-Wall fermions solved on these configurations after a gradient-flow smearing is applied with a flowtime of $t_{\rm gf}=1$ in lattice units. The ensembles span four lattice spacings in the range $0.06 \lesssim a \lesssim 0.15$ fm, six pion masses in the range $130 \lesssim m_\pi \lesssim 400$ MeV and multiple lattice volumes. On each ensemble, the gradient-flow scales $t_0/a^2$ and $w_0/a$ and the omega baryon mass $a m_\Omega$ are computed. The dimensionless product of these quantities is then extrapolated to the continuum and infinite volume limits and interpolated to the physical light, strange and charm quark mass point in the isospin limit, resulting in the determination of $\sqrt{t_0}=0.1422(14)$ fm and $w_0 = 0.1709(11)$ fm with all sources of statistical and systematic uncertainty accounted for. The dominant uncertainty in this result is the stochastic uncertainty, providing a clear path for a few-per-mille uncertainty, as recently obtained by the Budapest-Marseille-Wuppertal Collaboration.

hep-lat↗

arXiv2020

$F_K / F_π$ from Möbius domain-wall fermions solved on gradient-flowed HISQ ensembles

We report the results of a lattice quantum chromodynamics calculation of $F_K/F_π$ using Möbius domain-wall fermions computed on gradient-flowed $N_f=2+1+1$ highly-improved staggered quark (HISQ) ensembles. The calculation is performed with five values of the pion mass ranging from $130 \lesssim m_π\lesssim 400$ MeV, four lattice spacings of $a\sim 0.15, 0.12, 0.09$ and $0.06$ fm and multiple values of the lattice volume. The interpolation/extrapolation to the physical pion and kaon mass point, the continuum, and infinite volume limits are performed with a variety of different extrapolation functions utilizing both the relevant mixed-action effective field theory expressions as well as discretization-enhanced continuum chiral perturbation theory formulas. We find that the $a\sim0.06$ fm ensemble is helpful, but not necessary to achieve a subpercent determination of $F_K/F_π$. We also include an estimate of the strong isospin breaking corrections and arrive at a final result of $F_{K^\pm}/F_{π^\pm} = 1.1942(45)$ with all sources of statistical and systematic uncertainty included. This is consistent with the Flavour Lattice Averaging Group average value, providing an important benchmark for our lattice action. Combining our result with experimental measurements of the pion and kaon leptonic decays leads to a determination of $|V_{us}|/|V_{ud}| = 0.2311(10)$.

hep-lat↗