arXiv subjects

Shuyuan Fan

Publications and source records attributed to Shuyuan Fan.

4 recordsLinked to original sources

arXiv2026

GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can significantly reduce the communication volume. Existing methods quantize gradients via linear or nonlinear mappings in Euclidean space, often degrading model performance because highly anisotropic gradients incur direction-dependent distortion. We present GIFT, a geometry-informed gradient scaling method that performs low-precision communication in geometry-aware coordinates. By transforming gradients into a near-isotropic space before quantization, GIFT makes low-precision representations substantially more faithful to their high-precision counterparts. GIFT only changes the coordinate system used for low-precision gradient communication and does not change the optimizer, training recipe, communication collective, or low-precision format. We also develop a simplified geometry-aware transformation algorithm with low-rank approximation and selective application to balance the computation overhead and communication reduction. We examine the empirical convergence of GIFT using Llama-300M and Llama-600M models. Our results show that GIFT reduces the end-to-end pretraining time of Llama-600M by 7.6% on 64 NVIDIA GH200 Superchips, while improving the downstream task preservation profile over direct Euclidean FP8 communication under the same optimizer and communication path.

cs.DC↗

arXiv2025

Free energy dissipation and a decomposition of general jump diffusions on $\mathbb{R}^n$ without detailed balance

We analyze the thermodynamic structure of jump diffusions combining Brownian and Poisson noise, a class of stochastic dynamics relevant to non-equilibrium statistical physics. For such nonlocal dynamics, the free energy admits a full dissipation formula that decomposes into entropy production and housekeeping heat. A central result is a decomposition of the generator into symmetric and anti-symmetric parts with respect to the invariant measure $\rho_\mathrm{ss}$. The symmetric sector corresponds to a reversible dynamics and yields a nonlocal Fisher information governing free-energy decay, whereas the anti-symmetric sector generates a canonical conservative flow that produces circulation but no dissipation. Several numerical examples motivated by intracellular particle transports demonstrate how this decomposition clarifies the structure of non-equilibrium stationary states in jump-driven systems.

cond-mat.stat-mech↗

arXiv2025

Pier: Efficient Large Language Model pretraining with Relaxed Global Communication

Global communication, such as all-reduce and allgather, is the prominent performance bottleneck in large language model (LLM) pretraining. To address this issue, we present Pier, an efficient and scalable optimizer with relaxed global communication. Pier is built upon DiLoCo, which leverages an inner optimizer within groups of processors and an outer optimizer that requires global communication. To preserve the convergence and model performance, Pier incorporates two key techniques for the outer optimizer: momentum warmup and momentum decay. Pier employs an efficient and scalable system architecture to enable complex parallelization strategies in LLM pretraining. We examine the model performance and runtime reduction of Pier using the GPT model family (e.g., small, medium, XL, and 7B) and the OpenWebText dataset with a suite of thirteen downstream tasks. With data parallel strategy, Pier speeds up GPT-2 XL training by up to 2.7x-3.7x on 256 NVIDIA A100 GPUs and 1.2x-1.9x on 64 GH200 Superchips, respectively, without degradation of validation loss or downstream task performance. With data parallel and tensor parallel, Pier reduces the time cost GPT-2 7B model training by 54.5% on 128 A100s.

cs.DC↗

arXiv2025

Concave Comparison Functions for Accelerating Constrained Lyapunov Decay

What limits how fast a Lyapunov function can decay under input bounds? We address this question by showing how the shape of Lyapunov comparison functions governs guaranteed decay for control affine systems. Using a windowed nominal exponential rate together with the endpoint cap induced by actuator limits, we establish a strict ordering: concave comparison functions strictly outperform linear and convex ones, and strict concavity is necessary to improve the best achievable global exponential rate under a fixed endpoint cap. We derive a computable lower bound on the required actuation level for a target nominal rate and show that only concave shaping can reduce this level under the endpoint cap. We then establish a feasibility-preserving acceleration result: whenever a margin exists on a sublevel set, a feasible linear comparison can be replaced by a concave one that preserves feasibility while strictly increasing the guaranteed windowed decay. Finally, we give a tunable rational concave factor with controlled slope that yields a constructive design and integrates with CLF QP, as illustrated by examples.

math.OC↗