arXiv subjects

Wen Sun

Publications and source records attributed to Wen Sun.

At least 37 records · Page 2Linked to original sources

arXiv2025

On the moderate deviation principles in the sparse multi-type Erdős Rényi random graph

This paper investigate the sparse multi-type Erdős Rényi random graphs studied in Söderberg~\cite{soderberg2002general} and also Bollobás et al.~\cite{bollobas2007phase}. Although the corresponding central limit results are currently unknown, we establish moderate deviation principles for the size of the largest connected component, the number of specific types of connected components, and the total number of connected components. The associated rate functions are provided explicitly. As a byproduct of this work, we present the law of large numbers for the total number of connected components. Our proof methodology relies on representing the multi-type random graph using a conditional multi-dimensional compound Poisson process. We also discuss the properties of related multi-type branching processes and the properties of the matrices in the rate functions.

math.PR↗

arXiv2025

Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack

Large language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities. Investigating these weaknesses is crucial for robust safety mechanisms. Existing attacks primarily distract LLMs by introducing additional context or adversarial tokens, leaving the core harmful intent unchanged. In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks. More specifically, we establish a taxonomy of intent transformations and leverage them to generate attacks that may be misperceived by LLMs as benign requests for information. Unlike prior methods relying on complex tokens or lengthy context, our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless prompts. Extensive experiments on both open-source and commercial LLMs show that ISA achieves over 70% improvement in attack success rate compared to direct harmful prompts. More critically, fine-tuning models on only benign data reformulated with ISA templates elevates success rates to nearly 100%. For defense, we evaluate existing methods and demonstrate their inadequacy against ISA, while exploring both training-free and training-based mitigation strategies. Our findings reveal fundamental challenges in intent inference for LLMs safety and underscore the need for more effective defenses. Our code and datasets are available at https://github.com/NJUNLP/ISA.

cs.CL↗

arXiv2025

Theory of Topological Superconductivity and Antiferromagnetic Correlated Insulators in Twisted Bilayer WSe$_2$

Since the very recent discovery of unconventional superconductivity in twisted WSe$_2$ homobilayers at filling $ν=-1$, considerable interest has arisen in revealing its mechanism. In this paper, we developed a three-band tight-binding model with non-trivial band topology by direct Wannierization of the low-energy continuum model. Incorporating both onsite Hubbard repulsion and next-nearest-neighbor attraction, we then performed a mean-field analysis of the microscopic model and obtained a phase diagram qualitatively consistent with the experiment results. For zero or weak displacement field, the ground state is a Chern number $C=\pm 2$ topological superconductor in the Altland-Zirnbauer A-class (breaking time-reversal but preserving total $S_z$ symmetry) with inter-valley pairing dominant in $d_{xy}\pm id_{x^2-y^2}$-wave (mixing with a subdominant $p_x\mp i p_y$-wave) component. For a relatively strong displacement field, the ground state is a correlated insulator with the $120^\circ$ antiferromagnetic order. Our results provide new insights into the nature of the twisted WSe$_2$ systems and suggest the need for further theoretical and experimental explorations.

cond-mat.str-el↗

arXiv2025

Pair-density-wave phase of strongly interacting electrons on the triangular lattice: A variational Monte Carlo study

A robust theory of the mechanism of pair density wave (PDW) superconductivity (i.e. where Cooper pairs have nonzero center of mass momentum) remains elusive. Here we explore the triangular lattice $t$-$J$-$V$ model, a low-energy effective theory derived from the strong-coupling limit of the Holstein-Hubbard model, by large-scale variational Monte Carlo simulations. When the electron density is sufficiently low, the favored ground state is an s-wave PDW, consistent with results obtained from previous studies in this limit. Additionally, a PDW ground state with nematic d-wave pairing emerges in the intermediate range of electron densities and phonon frequencies. For these s-wave and d-wave PDWs arising in states with spontaneous breaking of time-reversal and inversion symmetries, PDW formation derives from valley-polarization and intra-pocket pairing.

cond-mat.str-el↗

arXiv2025

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training

Reinforcement learning (RL) post-training is crucial for LLM alignment and reasoning, but existing policy-based methods, such as PPO and DPO, can fall short of fixing shortcuts inherited from pre-training. In this work, we introduce $Q\sharp$, a value-based algorithm for KL-regularized RL that guides the reference policy using the optimal regularized $Q$ function. We propose to learn the optimal $Q$ function using distributional RL on an aggregated online dataset. Unlike prior value-based baselines that guide the model using unregularized $Q$-values, our method is theoretically principled and provably learns the optimal policy for the KL-regularized RL problem. Empirically, $Q\sharp$ outperforms prior baselines in math reasoning benchmarks while maintaining a smaller KL divergence to the reference policy. Theoretically, we establish a reduction from KL-regularized RL to no-regret online learning, providing the first bounds for deterministic MDPs under only realizability. Thanks to distributional RL, our bounds are also variance-dependent and converge faster when the reference policy has small variance. In sum, our results highlight $Q\sharp$ as an effective approach for post-training LLMs, offering both improved performance and theoretical guarantees. The code can be found at https://github.com/jinpz/q_sharp.

cs.LG↗

arXiv2025

All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning

From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.

cs.LG↗

arXiv2025

Disturbed cold gas in galaxy and structure formation

Cold and cool gas (T $\leq 10^4$ K) in the circumgalactic medium (CGM) and its interaction with galaxies remain poorly understood. Simulations predict that cold gas flows into galaxies through cosmic filaments, determining the disk formation and galaxy evolution. The cold gas accretion modes in the CGM and their dependence on dark matter halo mass and redshift remain puzzling. Resolving the kiloparsec-scale kinematics and dynamics of cold gas interacting with the disk, dust, and metals in different environments is particularly lacking at z > 2. Here we report two disturbed cold gas structures traced by ultra-strong MgII absorbers (rest-frame equivalent width Wr > 2 Å) exhibiting high kinematic velocities (> 500 km/s) and their environments at z ~ 4.9 and z ~ 2.6. Observations were conducted with VLT/MUSE, JWST/NIRCam, and ALMA to detect Lya and nebular emission lines, as well as dust continuum emission in the vicinity of these two absorbing gas structures. We identify two Lya emitters associated with a strong MgII absorber pair separated by ~1000 km/s at z~ 4.87. The pair exhibits relative differences in metallicity, dust content, and ionization states, suggesting internal metal and dust exchange within the ultra-large cold gas structure. For the strong MgII absorber at z = 2.5652, we detect a dusty star-forming galaxy at a projected distance of $D = 38$ kpc. This galaxy exhibits prominent HeI, [SIII], and Paschen$γ$ lines, along with significant dust continuum. It has a star formation rate of ~ 121 +/- 33 $M_{\odot}$/yr and likely harbors a rotating disk. These findings tentatively suggest that cold gas at high redshifts plays a critical role in driving disk formation and actively participates in the transfer of metals and dust within the overdense regions of the CGM.

astro-ph.GA↗

arXiv2025

Expressive Value Learning for Scalable Offline Reinforcement Learning

Reinforcement learning (RL) is a powerful paradigm for learning to make sequences of decisions. However, RL has yet to be fully leveraged in robotics, principally due to its lack of scalability. Offline RL offers a promising avenue by training agents on large, diverse datasets, avoiding the costly real-world interactions of online RL. Scaling offline RL to increasingly complex datasets requires expressive generative models such as diffusion and flow matching. However, existing methods typically depend on either backpropagation through time (BPTT), which is computationally prohibitive, or policy distillation, which introduces compounding errors and limits scalability to larger base policies. In this paper, we consider the question of how to develop a scalable offline RL approach without relying on distillation or backpropagation through time. We introduce Expressive Value Learning for Offline Reinforcement Learning (EVOR): a scalable offline RL approach that integrates both expressive policies and expressive value functions. EVOR learns an optimal, regularized Q-function via flow matching during training. At inference-time, EVOR performs inference-time policy extraction via rejection sampling against the expressive value function, enabling efficient optimization, regularization, and compute-scalable search without retraining. Empirically, we show that EVOR outperforms baselines on a diverse set of offline RL tasks, demonstrating the benefit of integrating expressive value learning into offline RL.

cs.LG↗

arXiv2025

Prompt Curriculum Learning for Efficient LLM Post-Training

We introduce Prompt Curriculum Learning (PCL), a lightweight reinforcement learning (RL) algorithm that selects intermediate-difficulty prompts using a learned value model to post-train language models. Since post-training LLMs via RL remains sensitive to batching and prompt selection strategies, we first conduct a series of systematic experiments where we (1) determine the optimal training batch size that balances generation efficiency and gradient quality and (2) establish the importance of focusing on prompts of intermediate difficulty for the policy. We build upon these results to design PCL, which identifies prompts of intermediate difficulty for the current policy in an on-policy manner by using a value model that is concurrently updated based on the current policy. By focusing on informative prompts that yield high effective ratios, PCL achieves either the highest performance or requires significantly less time to reach comparable performance to its counterparts. Compared to rollout-based filtering methods, PCL avoids costly rollouts and achieves $12.1\times$ and $16.9\times$ faster speed on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR, respectively. We further demonstrate that our value model accurately predicts prompt difficulty and allows PCL to focus on progressively more challenging prompts during RL. Our results present a new methodology that delivers improved tradeoff between upper-bound performance and efficiency for reasoning-focused RL.

cs.LG↗

arXiv2025

Value-Guided Search for Efficient Chain-of-Thought Reasoning

In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.

cs.LG↗

arXiv2025

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment. This paper studies the setting of online RLHF and focus on improving sample efficiency. All existing algorithms in online RLHF, whether doing passive exploration or active exploration, suffer from a sample complexity that scales exponentially with the scale of the reward function. This fundamental limitation hinders their effectiveness in scenarios with heavily skewed preferences, e.g. questions with a unique correct solution. To address this, we introduce Self-Exploring Preference-Incentive Online Preference Optimization (SE-POPO), an online RLHF algorithm that for the first time achieves a sample complexity that scales polynomially with the reward scale, answering an open problem raised by Xie et al. (2024).. Theoretically, we demonstrate that the sample complexity of SE-POPO dominates that of existing exploration algorithms. Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design. The code is available at https://github.com/MYC000801/SE-POPO.

cs.LG↗

arXiv2025

A High Incidence of Central Star Formation Inferred from the Color Gradients of Galaxies at $z>4$

We study the rest-frame ultraviolet-optical color gradients of 441 galaxies at $4<z<8$ by characterizing the wavelength dependence of their structural parameters derived from simultaneously fitting the seven-band NIRCam images acquired with the James Webb Space Telescope. Distinct from trends observed at lower redshifts, where most galaxies exhibit negative color gradients whereby galaxy centers are redder than their outskirts, in high-redshift galaxies positive color gradients are just as common as or even outnumber negative color gradients. Varying stellar population, dust, and active galactic nuclei can contribute to the observed color gradient. We show that for the majority of our sample, the observed color gradients principally reflect radial variations in stellar population, without strong contribution from dust reddening or contamination from active galactic nuclei. The sign and magnitude of the color profile depend systematically on the global properties of the galaxy: positive color gradients, characteristic of centrally concentrated star formation or outside-in growth, are found preferentially in galaxies of lower stellar mass, smaller size, and bluer spectral energy distribution.

astro-ph.GA↗

arXiv2025

Step-Audio 2 Technical Report

This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.

cs.CL↗

arXiv2025

SDGO: Self-Discrimination-Guided Optimization for Consistent Safety in Large Language Models

Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks that induce harmful content generation. In this paper, we reveal a critical safety inconsistency: LLMs can more effectively identify harmful requests as discriminators than defend against them as generators. This insight inspires us to explore aligning the model's inherent discrimination and generation capabilities. To this end, we propose SDGO (Self-Discrimination-Guided Optimization), a reinforcement learning framework that leverages the model's own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. Our method does not require any additional annotated data or external models during the training phase. Extensive experiments demonstrate that SDGO significantly improves model safety compared to both prompt-based and training-based baselines while maintaining helpfulness on general benchmarks. By aligning LLMs' discrimination and generation capabilities, SDGO brings robust performance against out-of-distribution (OOD) jailbreaking attacks. This alignment achieves tighter coupling between these two capabilities, enabling the model's generation capability to be further enhanced with only a small amount of discriminative samples. Our code and datasets are available at https://github.com/NJUNLP/SDGO.

cs.CL↗

arXiv2025

HieraEdgeNet: A Multi-Scale Edge-Enhanced Framework for Automated Pollen Recognition

Automated pollen recognition is vital to paleoclimatology, biodiversity monitoring, and public health, yet conventional methods are hampered by inefficiency and subjectivity. Existing deep learning models often struggle to achieve the requisite localization accuracy for microscopic targets like pollen, which are characterized by their minute size, indistinct edges, and complex backgrounds. To overcome this limitation, we introduce HieraEdgeNet, a multi-scale edge-enhancement framework. The framework's core innovation is the introduction of three synergistic modules: the Hierarchical Edge Module (HEM), which explicitly extracts a multi-scale pyramid of edge features that corresponds to the semantic hierarchy at early network stages; the Synergistic Edge Fusion (SEF) module, for deeply fusing these edge priors with semantic information at each respective scale; and the Cross Stage Partial Omni-Kernel Module (CSPOKM), which maximally refines the most detail-rich feature layers using an Omni-Kernel operator - comprising anisotropic large-kernel convolutions and mixed-domain attention - all within a computationally efficient Cross-Stage Partial (CSP) framework. On a large-scale dataset comprising 120 pollen classes, HieraEdgeNet achieves a mean Average Precision (mAP@.5) of 0.9501, significantly outperforming state-of-the-art baseline models such as YOLOv12n and RT-DETR. Furthermore, qualitative analysis confirms that our approach generates feature representations that are more precisely focused on object boundaries. By systematically integrating edge information, HieraEdgeNet provides a robust and powerful solution for high-precision, high-efficiency automated detection of microscopic objects.

cs.CV↗

arXiv2025

Scaling Offline RL via Efficient and Expressive Shortcut Models

Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models - a novel class of generative models - to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute. We release the code at nico-espinosadice.github.io/projects/sorl.

cs.LG↗

arXiv2025

Accelerating RL for LLM Reasoning with Optimal Advantage Regression

Reinforcement learning (RL) has emerged as a powerful tool for fine-tuning large language models (LLMs) to improve complex reasoning abilities. However, state-of-the-art policy optimization methods often suffer from high computational overhead and memory consumption, primarily due to the need for multiple generations per prompt and the reliance on critic networks or advantage estimates of the current policy. In this paper, we propose $A$*-PO, a novel two-stage policy optimization framework that directly approximates the optimal advantage function and enables efficient training of LLMs for reasoning tasks. In the first stage, we leverage offline sampling from a reference policy to estimate the optimal value function $V$*, eliminating the need for costly online value estimation. In the second stage, we perform on-policy updates using a simple least-squares regression loss with only a single generation per prompt. Theoretically, we establish performance guarantees and prove that the KL-regularized RL objective can be optimized without requiring complex exploration strategies. Empirically, $A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks, while reducing training time by up to 2$\times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL. Implementation of $A$*-PO can be found at https://github.com/ZhaolinGao/A-PO.

cs.LG↗

arXiv2025

Efficient Controllable Diffusion via Optimal Classifier Guidance

The controllable generation of diffusion models aims to steer the model to generate samples that optimize some given objective functions. It is desirable for a variety of applications including image generation, molecule generation, and DNA/sequence generation. Reinforcement Learning (RL) based fine-tuning of the base model is a popular approach but it can overfit the reward function while requiring significant resources. We frame controllable generation as a problem of finding a distribution that optimizes a KL-regularized objective function. We present SLCD -- Supervised Learning based Controllable Diffusion, which iteratively generates online data and trains a small classifier to guide the generation of the diffusion model. Similar to the standard classifier-guided diffusion, SLCD's key computation primitive is classification and does not involve any complex concepts from RL or control. Via a reduction to no-regret online learning analysis, we show that under KL divergence, the output from SLCD provably converges to the optimal solution of the KL-regularized objective. Further, we empirically demonstrate that SLCD can generate high quality samples with nearly the same inference time as the base model in both image generation with continuous diffusion and biological sequence generation with discrete diffusion. Our code is available at https://github.com/Owen-Oertell/slcd

cs.LG↗