arXiv subjects

Longbin Zeng

Publications and source records attributed to Longbin Zeng.

6 recordsLinked to original sources

arXiv2026

ARGUS: Production-Scale Tracing and Performance Diagnosis for over 10,000-GPU Clusters

Large-scale LLM training requires always-on, fine-grained observability for effective performance diagnosis at scale. Coarse resource monitors alone cannot localize root causes, and fine-grained profilers incur prohibitive (5%-30%) overheads and massive trace volumes, making always-on deployment impractical in large production clusters. We propose ARGUS, a low-overhead, fine-grained, always-on tracing and real-time analysis system for training workloads in 10,000+ GPU-scale production clusters. ARGUS decomposes observation along the training call hierarchy into CPU call stacks, framework semantics, and GPU kernel execution, with always-on collection under a combined overhead of less than 2%. It builds a unified data pipeline and compresses raw kernel events by approximately 3,700x from 10 MB to 2.7 KB per rank per step. Its progressive diagnosis framework automatically isolates anomalous windows, straggler ranks, and degraded kernels through iteration-time, phase-level, and kernel-level analysis. Deployed for over six months on a 10,000+ GPU production cluster, ARGUS has supported continuous fail-slow detection and performance optimization. Our case studies further demonstrate its effectiveness across representative anomalies, including compute stragglers, link degradation, pipeline-bubble amplification, FlashAttention JIT stalls, and compute stragglers masked by communication symptoms.

cs.DC↗

arXiv2025

Numerical stability analysis of large language models

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.

math.NA↗

arXiv2023

A General Description of Criticality in Neural Network Models

Recent experimental observations have supported the hypothesis that the cerebral cortex operates in a dynamical regime near criticality, where the neuronal network exhibits a mixture of ordered and disordered patterns. However, A comprehensive study of how criticality emerges and how to reproduce it is still lacking. In this study, we investigate coupled networks with conductance-based neurons and illustrate the co-existence of different spiking patterns, including asynchronous irregular (AI) firing and synchronous regular (SR) state, along with a scale-invariant neuronal avalanche phenomenon (criticality). We show that fast-acting synaptic coupling can evoke neuronal avalanches in the mean-dominated regime but has little effect in the fluctuation-dominated regime. In a narrow region of parameter space, the network exhibits avalanche dynamics with power-law avalanche size and duration distributions. We conclude that three stages which may be responsible for reproducing the synchronized bursting: mean-dominated subthreshold dynamics, fast-initiating a spike event, and time-delayed inhibitory cancellation. Remarkably, we illustrate the mechanisms underlying critical avalanches in the presence of noise, which can be explained as a stochastic crossing state around the Hopf bifurcation under the mean-dominated regime. Moreover, we apply the ensemble Kalman filter to determine and track effective connections for the neuronal network. The method is validated on noisy synthetic BOLD signals and could exactly reproduce the corresponding critical network activity. Our results provide a special perspective to understand and model the criticality, which can be useful for large-scale modeling and computation of brain dynamics.

q-bio.NC↗

arXiv2023

Digital Twin Brain: a simulation and assimilation platform for whole human brain

In this work, we present a computing platform named digital twin brain (DTB) that can simulate spiking neuronal networks of the whole human brain scale and more importantly, a personalized biological brain structure. In comparison to most brain simulations with a homogeneous global structure, we highlight that the sparseness, couplingness and heterogeneity in the sMRI, DTI and PET data of the brain has an essential impact on the efficiency of brain simulation, which is proved from the scaling experiments that the DTB of human brain simulation is communication-intensive and memory-access intensive computing systems rather than computation-intensive. We utilize a number of optimization techniques to balance and integrate the computation loads and communication traffics from the heterogeneous biological structure to the general GPU-based HPC and achieve leading simulation performance for the whole human brain-scaled spiking neuronal networks. On the other hand, the biological structure, equipped with a mesoscopic data assimilation, enables the DTB to investigate brain cognitive function by a reverse-engineering method, which is demonstrated by a digital experiment of visual evaluation on the DTB. Furthermore, we believe that the developing DTB will be a promising powerful platform for a large of research orients including brain-inspiredintelligence, rain disease medicine and brain-machine interface.

cs.NE↗

arXiv2023

Rethinking Precision of Pseudo Label: Test-Time Adaptation via Complementary Learning

In this work, we propose a novel complementary learning approach to enhance test-time adaptation (TTA), which has been proven to exhibit good performance on testing data with distribution shifts such as corruptions. In test-time adaptation tasks, information from the source domain is typically unavailable and the model has to be optimized without supervision for test-time samples. Hence, usual methods assign labels for unannotated data with the prediction by a well-trained source model in an unsupervised learning framework. Previous studies have employed unsupervised objectives, such as the entropy of model predictions, as optimization targets to effectively learn features for test-time samples. However, the performance of the model is easily compromised by the quality of pseudo-labels, since inaccuracies in pseudo-labels introduce noise to the model. Therefore, we propose to leverage the "less probable categories" to decrease the risk of incorrect pseudo-labeling. The complementary label is introduced to designate these categories. We highlight that the risk function of complementary labels agrees with their Vanilla loss formula under the conventional true label distribution. Experiments show that the proposed learning algorithm achieves state-of-the-art performance on different datasets and experiment settings.

cs.CV↗

arXiv2022

Simulation and assimilation of the digital human brain

Here, we present the Digital Brain (DB), a platform for simulating spiking neuronal networks at the large neuron scale of the human brain based on personalized magnetic-resonance-imaging data and biological constraints. The DB aims to reproduce both the resting state and certain aspects of the action of the human brain. An architecture with up to 86 billion neurons and 14,012 GPUs, including a two-level routing scheme between GPUs to accelerate spike transmission up to 47.8 trillion neuronal synapses, was implemented as part of the simulations. We show that the DB can reproduce blood-oxygen-level-dependent signals of the resting-state of the human brain with a high correlation coefficient, as well as interact with its perceptual input, as demonstrated in a visual task. These results indicate the feasibility of implementing a digital representation of the human brain, which can open the door to a broad range of potential applications.

q-bio.NC↗