arXiv subjects

Wenyi Fang

Publications and source records attributed to Wenyi Fang.

12 recordsLinked to original sources

arXiv2026

Numerical stability analysis of large language models

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.

math.NA↗

arXiv2026

LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition $f(g(\mathrm{x}))$, we provide an adaptive strategy that selects a small subset of components of $g(\mathrm{x})$ to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.

cs.LG↗

arXiv2026

A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models

Delayed loss spikes have been reported in neural-network training, but existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates. We study one stylized hypothesis: normalization can postpone instability by gradually increasing the effective learning rate during otherwise stable descent. To test this hypothesis at theorem level, we analyze batch-normalized linear models. Our flagship result concerns whitened square-loss linear regression, where we derive explicit no-rising-edge and delayed-onset conditions, bound the waiting time to directional onset, and show that the rising edge self-stabilizes within finitely many iterations. Combined with a square-loss decomposition, this yields a concrete delayed-spike mechanism in the whitened regime. For logistic regression, under highly restrictive active-margin assumptions, we prove only a supporting finite-horizon directional precursor in a knife-edge regime, with an optional appendix-only loss lower bound under an extra non-degeneracy condition. The paper should therefore be read as a stylized mechanism study rather than a general explanation of neural-network loss spikes. Within that scope, the results isolate one concrete delayed-instability pathway induced by batch normalization.

stat.ML↗

arXiv2026

Fast and Accurate Probing of In-Training LLMs' Downstream Performances

The paradigm of scaling Large Language Models (LLMs) in both parameter size and test time has pushed the boundaries of AI capabilities, but at the cost of making the traditional generative evaluation paradigm prohibitively expensive, therefore making the latency of LLM's in-training downstream performance evaluation unbearable. However, simple metrics like training loss (perplexity) are not always correlated with downstream performance, as sometimes their trends diverge from the actual task outcomes. This dilemma calls for a method that is computationally efficient and sufficiently accurate in measuring model capabilities. To address this challenge, we introduce a new in-training evaluation paradigm that uses a lightweight probe for monitoring downstream performance. The probes take the internal representations of LLM checkpoints (during training) as input and directly predict the checkpoint's performance on downstream tasks measured by success probability (i.e., pass@1). We design several probe architectures, validating their effectiveness using the OLMo3-7B's checkpoints across a diverse set of downstream tasks. The probes can accurately predict a checkpoint's performance (with avg. AUROC$>$0.75), have decent generalizability across checkpoints (earlier predicts later), and reduce the computation latency from $\sim$1 hr (using conventional generative evaluation method) to $\sim$3 min. In sum, this work presents a practical and scalable in-training downstream evaluation paradigm, enabling a more agile, informed, and efficient LLM development process.

cs.LG↗

arXiv2026

Dispelling the Curse of Singularities in Neural Network Optimizations

This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.

cs.LG↗

arXiv2026

On the Spectral Flattening of Quantized Embeddings

Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data. By formalizing the connection between Zipfian statistics and random matrix theory, we prove that the power-law decay in the singular value spectra of embeddings is a fundamental requisite for semantic encoding. We derive theoretical bounds showing that uniform quantization introduces a noise floor that disproportionately truncates this spectral tail, which induces spectral flattening and a strictly provable increase in the stable rank of representations. Empirical validation across diverse architectures including GPT-2 and TinyLlama corroborates that this geometric degradation precipitates representational collapse. This work not only quantifies the spectral sensitivity of LLMs but also establishes spectral fidelity as a necessary condition for stable low-bit optimization.

cs.LG↗

arXiv2025

Transferable text data distillation by trajectory matching

In the realm of large language model (LLM), as the size of large models increases, it also brings higher training costs. There is a urgent need to minimize the data size in LLM training. Compared with data selection method, the data distillation method aims to synthesize a small number of data samples to achieve the training effect of the full data set and has better flexibility. Despite its successes in computer vision, the discreteness of text data has hitherto stymied its exploration in natural language processing (NLP). In this work, we proposed a method that involves learning pseudo prompt data based on trajectory matching and finding its nearest neighbor ID to achieve cross-architecture transfer. During the distillation process, we introduce a regularization loss to improve the robustness of our distilled data. To our best knowledge, this is the first data distillation work suitable for text generation tasks such as instruction tuning. Evaluations on two benchmarks, including ARC-Easy and MMLU instruction tuning datasets, established the superiority of our distillation approach over the SOTA data selection method LESS. Furthermore, our method demonstrates a good transferability over LLM structures (i.e., OPT to Llama).

cs.CL↗

arXiv2024

PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Models

Pre-trained Vision-Language (VL) models such as CLIP have demonstrated their excellent performance across numerous downstream tasks. A recent method, Context Optimization (CoOp), further improves the performance of VL models on downstream tasks by introducing prompt learning. CoOp optimizes a set of learnable vectors, aka prompt, and freezes the whole CLIP model. However, relying solely on CLIP loss to fine-tune prompts can lead to models that are prone to overfitting on downstream task. To address this issue, we propose a plug-in prompt-regularization method called PLPP (Prompt Learning with PerPlexity), which use perplexity loss to regularize prompt learning. PLPP designs a two-step operation to compute the perplexity for prompts: (a) calculating cosine similarity between the weight of the embedding layer and prompts to get labels, (b) introducing a language model (LM) head that requires no training behind text encoder to output word probability distribution. Meanwhile, we unveil that the essence of PLPP is inherently a form of self-distillation. To further prevent overfitting as well as to reduce the additional computation introduced by PLPP, we turn the hard label to soft label and choose top-$k$ values for calculating the perplexity loss. For accelerating model convergence, we introduce mutual self-distillation learning, that is perplexity and inverted perplexity loss. The experiments conducted on four classification tasks indicate that PLPP exhibits superior performance compared to existing methods.

cs.CL↗

arXiv2024

Training Overhead Ratio: A Practical Reliability Metric for Large Language Model Training Systems

Large Language Models (LLMs) are revolutionizing the AI industry with their superior capabilities. Training these models requires large-scale GPU clusters and significant computing time, leading to frequent failures that significantly increase training costs. Despite its significance, this field lacks a metric for evaluating reliability. In this work, we introduce a novel reliability metric called \emph{Training Overhead Ratio} (TOR) to evaluate the reliability of fault-tolerant LLM training systems. TOR is defined as the ratio of optimal training time to the observed training time of a system, serving as a practical tool for users to estimate the actual time required to train an LLM on a given system. Furthermore, our investigation identifies the key factor for enhancing reliability and present TOR equations for various types of failures encountered in practice.

cs.DC↗

arXiv2020

Community Detection through Vector-label Propagation Algorithms

Community detection is a fundamental and important problem in network science, as community structures often reveal both topological and functional relationships between different components of the complex system. In this paper, we first propose a gradient descent framework of modularity optimization called vector-label propagation algorithm (VLPA), where a node is associated with a vector of continuous community labels instead of one label. Retaining weak structural information in vector-label, VLPA outperforms some well-known community detection methods, and particularly improves the performance in networks with weak community structures. Further, we incorporate stochastic gradient strategies into VLPA to avoid stuck in the local optima, leading to the stochastic vector-label propagation algorithm (sVLPA). We show that sVLPA performs better than Louvain Method, a widely used community detection algorithm, on both artificial benchmarks and real-world networks. Our theoretical scheme based on vector-label propagation can be directly applied to high-dimensional networks where each node has multiple features, and can also be used for optimizing other partition measures such as modularity with resolution parameters.

physics.soc-ph↗

arXiv2020

Community detection based on first passage probabilities

Community detection is of fundamental significance for understanding the topology characters and the spreading dynamics on complex networks. While random walk is widely used and is proven effective in many community detection algorithms, there still exists two major defects: (i) the maximal length of random walk is too large to distinguish the clustering information if using the average step of all possible random walks; (ii) the useful community information at all other step lengths are missed if using a pre-assigned maximal length. In this paper, we propose a novel community detection method based on the first passage probabilities (FPPM), equipped with a new similarity measure that incorporates the complete structural information within the maximal step length. Here the diameter of the network is chosen as an appropriate boundary of random walks which is adaptive to different networks. Then we use the hierarchical clustering to group the vertices into communities and further select the best division through the corresponding modularity values. Finally, a post-processing strategy is designed to integrate the unreasonable small communities, which significantly improves the accuracy of community division. Surprisingly, the numerical simulations show that FPPM performs best compared to several classic algorithms on both synthetic benchmarks and real-world networks, which reveals the universality and effectiveness of our method.

physics.soc-ph↗

arXiv2019

Homophily on social networks changes evolutionary advantage in competitive information diffusion

Competitive information diffusion on large-scale social networks reveals fundamental characteristics of rumor contagions and has profound influence on public opinion formation. There has been growing interest in exploring dynamical mechanisms of the competing evolutions recently. Nevertheless, the impacts of population homophily, which determines powerful collective human behaviors, remains unclear. In this paper, we incorporate homophily effects into a modified competitive ignorant-spreader-ignorant (SIS) rumor diffusion model with generalized population preference. Using microscopic Markov chain approach, we first derive the phase diagram of competing diffusion results and examine how competitive information spreads and evolves on social networks. We then explore the detailed effects of homophily, which is modeled by a rewiring mechanism. Results show that homophily promotes the formation of divided "echo chambers" and protects the disadvantaged information from extinction, which further changes or even reverses the evolutionary advantage, i.e., the difference of final proportions of the competitive information. We highlight the conclusion that the reversals may happen only when the initially disadvantaged information has stronger transmission ability, owning diffusion advantage over the other one. Our framework provides profound insight into competing dynamics with population homophily, which may pave ways for further controlling misinformation and guiding public belief systems. Moreover, the reversing condition sheds light on designing effective competing strategies in many real scenarios.

physics.soc-ph↗