arXiv subjects

Y. X. Wei

Publications and source records attributed to Y. X. Wei.

3 recordsLinked to original sources

arXiv2025

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model co-design can effectively address these challenges, enabling cost-efficient training and inference at scale. This paper presents an in-depth analysis of the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting key innovations such as Multi-head Latent Attention (MLA) for enhanced memory efficiency, Mixture of Experts (MoE) architectures for optimized computation-communication trade-offs, FP8 mixed-precision training to unlock the full potential of hardware capabilities, and a Multi-Plane Network Topology to minimize cluster-level network overhead. Building on the hardware bottlenecks encountered during DeepSeek-V3's development, we engage in a broader discussion with academic and industry peers on potential future hardware directions, including precise low-precision computation units, scale-up and scale-out convergence, and innovations in low-latency communication fabrics. These insights underscore the critical role of hardware and model co-design in meeting the escalating demands of AI workloads, offering a practical blueprint for innovation in next-generation AI systems.

cs.DC↗

arXiv2025

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Long-context modeling is crucial for next-generation language models, yet the high computational cost of standard attention mechanisms poses significant computational challenges. Sparse attention offers a promising direction for improving efficiency while maintaining model capabilities. We present NSA, a Natively trainable Sparse Attention mechanism that integrates algorithmic innovations with hardware-aligned optimizations to achieve efficient long-context modeling. NSA employs a dynamic hierarchical sparse strategy, combining coarse-grained token compression with fine-grained token selection to preserve both global context awareness and local precision. Our approach advances sparse attention design with two key innovations: (1) We achieve substantial speedups through arithmetic intensity-balanced algorithm design, with implementation optimizations for modern hardware. (2) We enable end-to-end training, reducing pretraining computation without sacrificing model performance. As shown in Figure 1, experiments show the model pretrained with NSA maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves substantial speedups over Full Attention on 64k-length sequences across decoding, forward propagation, and backward propagation, validating its efficiency throughout the model lifecycle.

cs.CL↗

arXiv2024

Prediction of Fishbone Linear Instability in Tokamaks with Machine Learning Methods

A machine learning based surrogate model for fishbone linear instability in tokamaks is constructed. Hybrid simulations with the kinetic-magnetohydrodynamic (MHD) code M3D-K is used to generate the database of fishbone linear instability, through scanning the four key parameters which are thought to determine the fishbone physics. The four key parameters include (1) central total beta of both thermal plasma and fast ions, (2) the fast ion pressure fraction, (3) central value of safety factor $q$ and (4) the radius of $q=1$ surface. Four machine learning methods including linear regression, support vector machines (SVM) with linear kernel, SVM with nonlinear kernel and multi-layer perceptron are used to predict the fishbone instability, growth rate and real frequency, mode structure respectively. Among the four methods, SVM with nonlinear kernel performs very well to predict the linear instability with accuracy $\approx$95%, growth rate and real frequency with $R^2\approx$98%, mode structure with $R^2\approx$98%.

physics.plasm-ph↗