Searcharxiv⌕ Search

arXiv subjects

Shiliang Wu

Publications and source records attributed to Shiliang Wu.

5 recordsLinked to original sources
arXiv2026

A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation

Decoupled PPO has been a successful reinforcement learning (RL) algorithm to deal with the high data staleness under the asynchronous RL setting. Decoupled loss used in decoupled PPO improves coupled-loss style of algorithms' (e.g., standard PPO, GRPO) learning stability by introducing a proximal policy to decouple the off-policy correction (importance weight) from the policy update constraint (trust region). However, the proximal policy requires an extra forward pass through the model at each training step, creating a computational overhead for large language models training. We observe that since the proximal policy only serves as a trust region anchor between the behavior and target policies, we can approximate it through simple interpolation without explicit computation. We call this approach A-3PO (APproximated Proximal Policy Optimization). A-3PO eliminates this overhead, accelerating training by 1.8x speedup while maintaining comparable performance. Code \& off-the-shelf example are contributed to the open-source RL training system AReaL at: https://github.com/areal-project/AReaL/blob/v1.0.0.rc1/docs/algorithms/prox_approx.md

cs.LG↗
arXiv2026

Decomposing MXFP4 quantization error for LLM reinforcement learning: reducible bias, recoverable deadzone, and an irreducible floor

MXFP4 arithmetic can dramatically accelerate reinforcement learning (RL) post-training of large language models (LLMs), yet the quantization error introduces severe accuracy degradation. Existing work treats the quantization error as a monolithic noise term, missing the distinct mechanisms upon interpreting how quantization error damages training. We prove an exact three-way decomposition of quantization error and show how each component dominates a distinct RL training pathway. Our theoretical and empirical analysis decomposes the MXFP4 quantization error into three additive components: "scale bias" from power-of-two rounding, "deadzone truncation" from zeroing small values, and "grid noise" from rounding to the nearest 4-bit grid. Each component dominates a distinct RL failure mode: scale bias accumulates multiplicatively through the backward pass, affecting gradient accuracy; deadzone truncation degrades rollout quality; and grid noise raises the policy's entropy. We combine corrections that are RL failure mode-targeted but not component-exclusive: Macro-block scaling to reduce scale bias, Outlier Fallback recovers deadzone entries, but also partially reduces scale bias induced error, and Adaptive Quantization Noise (AQN) for controlling the policy entropy. On Qwen2.5-3B dense and Qwen3-30B-A3B-Base mixture-of-experts model, the targeted corrections recover BF16 accuracy to within 0.7% and exceed BF16 by +1.0% respectively.

cs.LG↗
arXiv2022

The perturbation bound of the extended vertical linear complementarity problem

In this paper, we discuss the perturbation analysis of the extended vertical linear complementarity problem (EVLCP). Under the assumption of the row $\mathcal{W}$-property, several absolute and relative perturbation bounds of EVLCP are given, which can be reduced to some existing results. Some numerical examples are given to show the proposed bounds.

math.NA↗
arXiv2022

New error bounds for the extended vertical LCP

In this paper, by making use of this fact that for $a_{j}, b_{j}\in \mathbb{R}$, $j=1,2,\ldots,n$, there are $λ_{j}\in [0,1]$ with $\sum_{j=1}^{n}λ_{j}=1$ such that \[ \min_{1\leq j\leq n}\{a_{j}\}-\min_{1\leq j\leq n}\{b_{j}\}=\sum_{j=1}^{n}λ_{j}(a_{j}-b_{j}), \] some new error bounds of the extended vertical LCP under the row $\mathcal{W}$-property are obtained, which cover the error bounds in [Math. Program., 106 (2006) 513-525] and [Comput. Optim. Appl., 42 (2009) 335-352]. Not only that, these new error bounds skillfully avoid the inconvenience caused by the row rearrangement technique for error bounds to achieve the goal of reducing the computation workload, which was introduced in the latter paper mentioned above. Besides, with respect to the row $\mathcal{W}$-property, two new sufficient and necessary conditions are obtained.

math.NA↗