arXiv subjects

Qiaorui Chen

Publications and source records attributed to Qiaorui Chen.

3 recordsLinked to original sources

arXiv2026

MiniMax Sparse Attention

Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.

cs.AI↗

arXiv2026

MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches

Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%).

cs.LG↗

arXiv2025

ByteQC: GPU-Accelerated Quantum Chemistry Package for Large-Scale Systems

Applying quantum chemistry algorithms to large-scale systems requires substantial computational resources scaled with the system size and the desired accuracy. To address this, ByteQC, a fully-functional and efficient package for large-scale quantum chemistry simulations, has been open-sourced at https://github.com/bytedance/byteqc, leveraging recent advances in computational power and many-body algorithms. Regarding computational power, several standard algorithms are efficiently implemented on modern GPUs, ranging from mean-field calculations (Hartree-Fock and density functional theory) to post-Hartree-Fock methods such as Møller-Plesset perturbation theory, random phase approximation, coupled cluster methods, and quantum Monte Carlo methods. For the algorithmic approach, we also employ a quantum embedding method, which significantly expands the tractable system size while preserving high accuracy at the gold-standard level. All these features have been systematically benchmarked. For standalone algorithms, the benchmark results demonstrate up to a 60$\times$ speedup when compared to 100-core CPUs. Additionally, the tractable system sizes have been significantly expanded: 1,610 orbitals for coupled cluster with single and double excitations (1,380 orbitals with perturbative triple excitations), 11,040 orbitals for Møller-Plesset perturbation theory of second order, 37,120 orbitals for mean-field calculations under open boundary conditions, and over 100,000 orbitals for periodic boundary conditions. For the advanced quantum embedding feature, two representative examples are demonstrated: the water cluster problem (2,752 orbitals) and a water monomer adsorbed on a boron nitride surface (3,929 orbitals), achieving the gold-standard accuracy.

physics.chem-ph↗