Searcharxiv⌕ Search

arXiv subjects

Ao Sun

Publications and source records attributed to Ao Sun.

At least 37 records · Page 2Linked to original sources
arXiv2025

InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation

Long-sequence processing is a critical capability for modern large language models. However, the self-attention mechanism in the standard Transformer architecture faces severe computational and memory bottlenecks when processing long sequences. While trainable sparse attention methods offer a promising solution, existing approaches such as NSA introduce excessive extra parameters and disrupt the conventional \textit{pretrain-on-short, finetune-on-long} workflow, resulting in slow convergence and difficulty in acceleration. To overcome these limitations, we introduce dense-sparse switchable attention framework, termed as InfLLM-V2. InfLLM-V2 is a trainable sparse attention that seamlessly adapts models from short to long sequences. Specifically, InfLLM-V2 reuses dense attention parameters through parameter-free architecture modification, maintaining consistency between short and long sequence processing. Additionally, InfLLM-V2 ensures computational efficiency across all sequence lengths, by using dense attention for short inputs and smoothly transitioning to sparse attention for long sequences. To achieve practical acceleration, we further introduce an efficient implementation of InfLLM-V2 that significantly reduces the computational overhead. Our experiments on long-context understanding and chain-of-thought reasoning demonstrate that InfLLM-V2 is 4$\times$ faster than dense attention while retaining 98.1% and 99.7% of the performance, respectively. Based on the InfLLM-V2 framework, we have trained and open-sourced MiniCPM4.1 (https://huggingface.co/openbmb/MiniCPM4.1-8B), a hybrid reasoning model, providing a reproducible implementation for the research community.

cs.CL↗
arXiv2025

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens

Existing methods for training LLMs on long-sequence data, such as Tensor Parallelism and Context Parallelism, exhibit low Model FLOPs Utilization as sequence lengths and number of GPUs increase, especially when sequence lengths exceed 1M tokens. To address these challenges, we propose BurstEngine, an efficient framework designed to train LLMs on long-sequence data. BurstEngine introduces BurstAttention, an optimized distributed attention with lower communication cost than RingAttention. BurstAttention leverages topology-aware ring communication to fully utilize network bandwidth and incorporates fine-grained communication-computation overlap. Furthermore, BurstEngine introduces sequence-level selective checkpointing and fuses the language modeling head with the loss function to reduce memory cost. Additionally, BurstEngine introduces workload balance optimization for various types of attention masking. By integrating these optimizations, BurstEngine achieves a $1.2\times$ speedup with much lower memory overhead than the state-of-the-art baselines when training LLMs on extremely long sequences of over 1M tokens. We have made our code publicly available on GitHub: https://github.com/thunlp/BurstEngine.

cs.DC↗
arXiv2025

MOSAIC: Minimax-Optimal Sparsity-Adaptive Inference for Change Points in Dynamic Networks

We propose a new inference framework, named MOSAIC, for change-point detection in dynamic networks with the simultaneous low-rank and sparse-change structure. We establish the minimax rate of detection boundary, which relies on the sparsity of changes. We then develop an eigen-decomposition-based test with screened signals that approaches the minimax rate in theory, with only a minor logarithmic loss. For practical implementation of MOSAIC, we adjust the theoretical test by a novel residual-based technique, resulting in a pivotal statistic that converges to a standard normal distribution via the martingale central limit theorem under the null hypothesis and achieves full power under the alternative hypothesis. We also analyze the minimax rate of testing boundary for dynamic networks without the low-rank structure, which almost aligns with the results in high-dimensional mean-vector change-point inference. We showcase the effectiveness of MOSAIC and verify our theoretical results with several simulation examples and a real data application.

stat.ML↗
arXiv2025

MiniCPM4: Ultra-Efficient LLMs on End Devices

This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems. Specifically, in terms of model architecture, we propose InfLLM v2, a trainable sparse attention mechanism that accelerates both prefilling and decoding phases for long-context processing. Regarding training data, we propose UltraClean, an efficient and accurate pre-training data filtering and generation strategy, and UltraChat v2, a comprehensive supervised fine-tuning dataset. These datasets enable satisfactory model performance to be achieved using just 8 trillion training tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient pre-training strategy search, and improve existing post-training methods by introducing chunk-wise rollout for load-balanced reinforcement learning and data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose CPM.cu that integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding. To meet diverse on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively. Furthermore, we construct a hybrid reasoning model, MiniCPM4.1, which can be used in both deep reasoning mode and non-reasoning mode. Evaluation results demonstrate that MiniCPM4 and MiniCPM4.1 outperform similar-sized open-source models across benchmarks, with the 8B variants showing significant speed improvements on long sequence understanding and generation.

cs.CL↗
arXiv2025

Biomarkers selection and combination based on the weighted Youden index

In clinical practice, multiple biomarkers are used for disease diagnosis, but their individual accuracies are often suboptimal, with only a few proving directly relevant. Effectively selecting and combining biomarkers can significantly improve diagnostic accuracy. Existing methods often optimize metrics like the Area Under the ROC Curve (AUC) or the Youden index. However, optimizing AUC does not yield estimates for optimal cutoff values, and the Youden index assumes equal weighting of sensitivity and specificity, which may not reflect clinical priorities where these metrics are weighted differently. This highlights the need for methods that can flexibly accommodate such requirements. In this paper, we present a novel framework for selecting and combining biomarkers to maximize a weighted version of the Youden index. We introduce a smoothed estimator based on the weighted Youden index and propose a penalized version using the SCAD penalty to enhance variable selection. To handle the non-convexity of the objective function and the non-smoothness of the penalty, we develop an efficient algorithm, also applicable to other non-convex optimization problems. Simulation studies demonstrate the performance and efficiency of our method, and we apply it to construct a diagnostic scale for dermatitis.

stat.ME↗
arXiv2025

Regularity of cylindrical singular sets of mean curvature flow

In this paper, we study the $k$-cylindrical singular set of mean curvature flow in $\mathbb R^{n+1}$ for each $1\leq k\leq n-1$. We prove that they are locally contained in a $k$-dimensional $C^{2,α}$-submanifold after removing some lower-dimensional parts. Moreover, if the $k$-cylindrical singular set is a $k$-submanifold, then its curvature is determined by the asymptotic profile of the flow at these singularities. As a byproduct, we provide a detailed asymptotic profile and graphical radius estimate at these singularities. The proof is based on a new $L^2$-distance non-concentration property that we introduced in [SWX25], modified into a relative version that allows us to modulo those low eigenmodes that are not decaying fast enough and do not contribute to the curvature of the singular set.

math.DG↗
arXiv2025

Generic mean curvature flows with cylindrical singularities I: the normal forms and nondegeneracy

This paper studies the dynamics of mean curvature flow as it approaches a cylindrical singularity. We proved that the rescaled mean curvature flow converging to a smooth generalized cylinder can be written as a graph over the cylinder in a ball of radius $K\sqrt{t}$, and a normal form of the asymptotics. Using the normal form, we can define the nondegeneracy of cylindrical singularities, and we show that nondegenerate cylindrical singularities are isolated in space, have a mean convex neighborhood, and are type-I.

math.DG↗
arXiv2025

On mean curvature flow translators with prescribed ends

Given a smooth closed embedded self-shrinker $S$ with index $I$ in $\mathbb{R}^{n}$, we construct an $I$-dimensional family of complete translators polynomially asymptotic to $S\times\mathbb{R}$ at infinity, which answers a long-standing question by Ilmanen. We further prove that $\mathbb{R}^{n+1}$ can be decomposed in many ways into a one-parameter family of closed sets $\coprod_{a\in \mathbb{R}} T_a$, and each closed set $T_a$ contains a complete translator asymptotic to $S\times\mathbb{R}$ at infinity. If the closed set $T_a$ fattens, namely it has nonempty interior, then there are at least two translators asymptotic to each other at an exponential rate, which can be viewed as a kind of nonuniqueness. We show that this fattening phenomenon is non-generic but indeed happens.

math.DG↗
arXiv2025

Generic Dynamics of Mean Curvature Flows with Asymptotically Conical Singularities

This is the second paper in the series to study the generic dynamics of mean curvature flows. We study the initial perturbation of mean curvature flows, whose first singularity is modeled by an asymptotically conical shrinker. The noncompactness of the limiting shrinker creates essential difficulties. We introduce the Feynman-Kac formula to get precise asymptotic behaviour of the linearized rescaled mean curvature equation along an orbit. We also develop the invariant cone method for the noncompact setting for the local dynamics near the shrinker. As a consequence, we prove that after a generic initial perturbation, the perturbed rescaled mean curvature flow avoids the conical singularity.

math.DG↗
arXiv2025

ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoning

LLM-based translation agents have achieved highly human-like translation results and are capable of handling longer and more complex contexts with greater efficiency. However, they are typically limited to text-only inputs. In this paper, we introduce ViDove, a translation agent system designed for multimodal input. Inspired by the workflow of human translators, ViDove leverages visual and contextual background information to enhance the translation process. Additionally, we integrate a multimodal memory system and long-short term memory modules enriched with domain-specific knowledge, enabling the agent to perform more accurately and adaptively in real-world scenarios. As a result, ViDove achieves significantly higher translation quality in both subtitle generation and general translation tasks, with a 28% improvement in BLEU scores and a 15% improvement in SubER compared to previous state-of-the-art baselines. Moreover, we introduce DoveBench, a new benchmark for long-form automatic video subtitling and translation, featuring 17 hours of high-quality, human-annotated data. Our code is available here: https://github.com/pigeonai-org/ViDove

cs.AI↗
arXiv2025

LLM-Powered CPI Prediction Inference with Online Text Time Series

Forecasting the Consumer Price Index (CPI) is an important yet challenging task in economics, where most existing approaches rely on low-frequency, survey-based data. With the recent advances of large language models (LLMs), there is growing potential to leverage high-frequency online text data for improved CPI prediction, an area still largely unexplored. This paper proposes LLM-CPI, an LLM-based approach for CPI prediction inference incorporating online text time series. We collect a large set of high-frequency online texts from a popularly used Chinese social network site and employ LLMs such as ChatGPT and the trained BERT models to construct continuous inflation labels for posts that are related to inflation. Online text embeddings are extracted via LDA and BERT. We develop a joint time series framework that combines monthly CPI data with LLM-generated daily CPI surrogates. The monthly model employs an ARX structure combining observed CPI data with text embeddings and macroeconomic variables, while the daily model uses a VARX structure built on LLM-generated CPI surrogates and text embeddings. We establish the asymptotic properties of the method and provide two forms of constructed prediction intervals. The finite-sample performance and practical advantages of LLM-CPI are demonstrated through both simulation and real data examples.

stat.ML↗
arXiv2025

An effcient variational quantum Korkin-Zolotarev algorithm for solving shortest vector problems

Noisy intermediate-scale quantum cryptanalysis focuses on the capability of near-term quantum devices to solve the mathematical problems underlying cryptography, and serves as a cornerstone for the design of post-quantum cryptographic algorithms. For the shortest vector problem (SVP), which is one of the computationally hard problems in lattice-based cryptography, existing near-term quantum cryptanalysis algorithms map the problem onto a fully-connected quantum Ising Hamiltonian, and obtain the solution by optimizing for the first excited state. However, as the quantum system scales with the problem size, determining the first excited state becomes intractable due to the exponentially increased complexity for large-scale SVP instances. In this paper, we propose a variational quantum Korkin-Zolotarev (VQKZ) algorithm, which significantly reduces the qubit requirement for solving the SVP. Specifically, by transforming the original SVP into a series of subproblems on projected sublattices, the proposed VQKZ algorithm enables near-term quantum devices to solve SVP instances with lattice dimensions 61.39% larger than those solvable by previous methods. Furthermore, numerical simulations demonstrate that the proposed VQKZ algorithm can significantly outperform existing methods in terms of the length of solution vectors.

quant-ph↗
arXiv2025

High-rate continuous-variable quantum key distribution over 100 km fiber with composable security

Quantum key distribution (QKD), providing a way to generate secret keys with information-theoretic security,is arguably one of the most significant achievements in quantum information. The continuous-variable QKD (CV-QKD) offers the potential advantage of achieving a higher secret key rate (SKR) within a metro area, as well as being compatible with the mature telecom industry. However, the SKR and transmission distance of state-of-the-art CV-QKD systems are currently limited. Here, based on the novelly proposed orthogonal-frequency-division-multiplexing (OFDM) CV-QKD protocol, we demonstrate for the first time a high-rate multi-carrier (MC) CV-QKD with a 10 GHz symbol rate that chieves Gbps SKR within 10km and Mbps SKR over 100 km in the finite-size regime under composable security against collective attacks. The record-breaking results are achieved by suitable optimization of subcarrier number and modulation variance, well-controlled excess noise induced by both OFDM mechanism and efficient DSP scheme, and high-performance post-processing capacity realized by heterogeneous computing scheme. The composable finite-size SKR reaches 1779.45 Mbps@5km, 1025.49 Mbps@10km, 370.50 Mbps@25km, 99.93 Mbps@50km, 25.70 Mbps@75km,and 2.25 Mbps@100km, which improves the SKR by two orders of magnitude and quintuples the maximal transmission distance compared to most recently reported CV-QKD results [Nature Communications, 13, 4740 (2022)]. Interestingly, it is experimentally verified that the SKR of the proposed MC CV-QKD can approach five times larger than that of the single-carrier CV-QKD with the same symbol rate without additional hardware costs. Our work constitutes a critical step towards future high-speed quantum metropolitan and access networks.

quant-ph↗
arXiv2025

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling

Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec.

cs.CL↗
arXiv2025

BASIC: Bipartite Assisted Spectral-clustering for Identifying Communities in Large-scale Networks

Community detection, which focuses on recovering the group structure within networks, is a crucial and fundamental task in network analysis. However, the detection process can be quite challenging and unstable when community signals are weak. Motivated by a newly collected large-scale academic network dataset from the Web of Science, which includes multi-layer network information, we propose a Bipartite Assisted Spectral-clustering approach for Identifying Communities (BASIC), which incorporates the bipartite network information into the community structure learning of the primary network. The accuracy and stability enhancement of BASIC is validated theoretically on the basis of the degree-corrected stochastic block model framework, as well as numerically through extensive simulation studies. We rigorously study the convergence rate of BASIC even under weak signal scenarios and prove that BASIC yields a tighter upper error bound than that based on the primary network information alone. We utilize the proposed BASIC method to analyze the newly collected large-scale academic network dataset from statistical papers. During the author collaboration network structure learning, we incorporate the bipartite network information from author-paper, author-institution, and author-region relationships. From both statistical and interpretative perspectives, these bipartite networks greatly aid in identifying communities within the primary collaboration network.

math.ST↗
arXiv2025

Genus one singularities in mean curvature flow

We show that for certain one-parameter families of initial conditions in $\mathbb R^3$, when we run mean curvature flow, a genus one singularity must appear in one of the flows. Moreover, such a singularity is robust under perturbation of the family of initial conditions. This contrasts sharply with the case of just a single flow. As an application, we construct an embedded, genus one self-shrinker with entropy lower than a shrinking doughnut.

math.DG↗