arXiv subjects

Guohua Liu

Publications and source records attributed to Guohua Liu.

At least 19 recordsLinked to original sources

arXiv2026

Hindsight Credit Assignment over Trees for Agentic RL

Reinforcement learning for long-horizon LLM agents faces a fundamental dilemma: successful experiences are essential for policy improvement, yet limited policy capabilities make such experiences difficult to discover. Sparse outcome rewards compound this challenge, leaving sampled trajectories with little relative feedback for learning. In this paper, we propose TriGRPO, a tree-based reinforcement learning framework that turns scarce successes into exploration and learning signals through hindsight credit assignment within and across rollout trees. TriGRPO constructs rollout trees through hindsight-guided branching: when an ongoing trajectory reaches a matched decision state, a successful action discovered during rollout initiates a sibling branch that continues autonomously, enabling further exploration and parent-sibling comparisons. Three complementary groups organize policy optimization over these trees: root groups capture within-prompt relative returns, node groups assign credit from parent-sibling outcome differences, and cross groups supplement sparse feedback with historical performance on related tasks. Experimental results demonstrate consistent gains in task performance and rollout efficiency across model scales, highlighting structured hindsight credit assignment as an effective approach to long-horizon agent learning.

cs.LG↗

arXiv2026

Bounds on the minimum orbital period of general static spherical black holes: a sharp upper bound and a conditional lower bound

We investigate bounds on the minimum orbital period around four-dimensional general static spherical black holes. For all static spherically symmetric black holes with an event horizon, asymptotic flatness and non-negative energy density outside the horizon, we prove $T_{\min}\leq 6\sqrt{3}\,πM$. The bound is sharp and is saturated by the Schwarzschild solution. The lower bound requires an additional condition on the exterior matter: under the tangential pressure condition $p_τ\geq(κ/2)ρ$ with $κ>1$, imposed for all $r\geq r_{g}$, where $r_{g}$ is the largest radius at which $T(r)$ attains its global minimum, we prove $T_{\min}\geq c(κ)M$ with $c(κ)=6\sqrt{3}π(κ-1)\sqrt{κ+2}/κ^{3/2}$. In summary, both bounds apply to the same geometric class and differ only in what they demand of the matter: non-negative energy density suffices for the upper bound, while the lower bound is established under the additional tangential pressure condition.

gr-qc↗

arXiv2026

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.

cs.LG↗

arXiv2026

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

On-Policy Distillation (OPD) trains a student on its own generations using teacher feedback. With different tokenizers, comparing teacher and student predictions requires alignment at both sequence and vocabulary levels. In this paper, we examine whether expanding this alignment coverage improves learning. Across three heterogeneous teacher--student pairs on mathematical reasoning and code generation, strict 1:1 groups already cover most student-generated tokens despite substantial vocabulary mismatch. On responses sampled from the students before distillation, the shared vocabulary retains nearly all teacher and student probability mass at strictly aligned positions on average. Restricting reverse KL to a student-selected top-16 subset of the shared vocabulary at each strict position achieves accuracy comparable to full shared-vocabulary OPD, outperforming the evaluated cross-tokenizer baselines. Adding mean squared error supervision on span log-probabilities in mismatch groups gives complete supervision coverage, yet reduces accuracy. At checkpoints from training with only the strict loss, the span gradients show weak or negative directional agreement with the strict gradients and grow in magnitude relative to them. These diagnostics may help explain the accuracy drop from adding span supervision. Our findings motivate a shift from maximizing alignment coverage to prioritizing supervision reliability: compact supervision at strict positions can be more effective than broader coverage that introduces weakly aligned or conflicting training signals.

cs.CL↗

arXiv2026

Analysis of minimum orbital periods around d-dimensional charged black holes

This paper investigates the bounds on the minimum orbital period for test objects around d-dimensional charged black holes in asymptotically flat spacetimes. We derive the exact critical radius and the minimum orbital period. We then prove analytically that the minimum orbital period decreases strictly as the charge of the black hole increases. Thus, the upper limit is reached for an uncharged black hole, while the lower limit is attained for a maximally charged one, and the two bounds take the closed form $\frac{2π(d-2)}{d-3}[(d-2)M]^{\frac{1}{d-3}}\leqslant T_{min} \leqslant 2π\sqrt{\frac{d-1}{d-3}}\,[(d-1)M]^{\frac{1}{d-3}}$. Since the minimum period equals $2π$ times the shadow radius, the upper bound is equivalently a universal upper bound on the shadow radius. These results improve our understanding of dynamics around d-dimensional black holes and impose constraints on candidate gravity theories.

gr-qc↗

arXiv2026

Bounds on the minimum orbital period of four-dimensional black holes: From singular to non-singular spacetimes

It was conjectured that the lower bound $T_{\min}\geq 4πM $ may be a universal property of four-dimensional black holes, where $M$ is the black hole mass and $T_{\min}$ is the minimum orbital period around black holes. Recently, we established an upper bound $T_{\min}\le 6\sqrt{3}πM$ for Schwarzschild, Reissner--Nordström and Kerr--Newman black holes [23]. In this work we incorporate that analysis, which is not intended for separate publication, into a unified treatment and extend it to the non-singular Hayward and Bardeen black holes, demonstrating that the combined bounds $4πM\le T_{\min}\le 6\sqrt{3}πM$ hold for all five types of four-dimensional black holes investigated here. For all types of studied black holes, the upper bound is saturated by the standard Schwarzschild black hole and the lower bound by the maximally rotating Kerr black hole, suggesting that these bounds $4πM \leq T_{\min} \leq 6\sqrt{3}πM$ may be a universal property of four-dimensional black holes with horizons.

gr-qc↗

arXiv2026

Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training

Large language models offer broad capabilities, but adapting them to evolving domains, tools, and requirements often entails repeated post-training. Autonomous systems automate parts of this process by proposing updates, training candidates, and using evaluation feedback to select subsequent proposals. As evidence accumulates, a central problem emerges: which past update evidence remains actionable after subsequent training has changed the parent model? An update's effect depends on its parent, data, and training stage. Treating past success as context-free permission can waste compute. If the resulting child is promoted, it can also degrade the subsequent training trajectory. We formulate this problem as conditional experience transfer and introduce Boundary-Calibrated Intervention Transfer (BCIT), a method that authorizes experience reuse before weight-changing training. BCIT binds an observed effect to its source context, checks applicability conditions, vetoes candidates with named hard conflicts, and obtains current-state evidence through a bounded training trial when needed. Fully trained candidates still face a shared adoption rule, and only observed events extend memory. On one 4B model adapted across finance reasoning, text-to-SQL, and function calling, candidate updates exhibit heterogeneous target and retention effects across the evaluated contexts. Under matched candidates, evidence, and compute, BCIT authorizes fewer harmful updates and attains higher equal-budget final-model quality than the evaluated alternatives. These results support treating experience authorization as a distinct problem in autonomous post-training.

cs.AI↗

arXiv2026

DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning

Reinforcement Learning has become a standard paradigm for aligning Large Language Models with human intent and task requirements. While Group Relative Policy Optimization offers an efficient, value-model-free alternative to Proximal Policy Optimization, adapting it to real-world multi-reward settings remains challenging. Standard scalarization practices, such as Reward Combination and Advantage Combination, suffer from significant drawbacks: Reward Combination frequently generates advantages with excessively large squared magnitudes that lead to training instability, while Advantage Combination relies on static hyperparameters and ignores cross-objective correlations. To address these limitations, we propose Dynamic Variance-adaptive Advantage Optimization (DVAO), which dynamically adjusts combination weights based on the empirical reward variance of each objective within a rollout group, effectively up-weighting objectives with a stronger learning signal while suppressing noisy ones. We mathematically prove that DVAO maintains bounded advantage magnitudes for stable training and introduces a self-adaptive cross-objective regularization mechanism. Extensive experiments on mathematical reasoning and tool-use benchmarks using Qwen3 and Qwen2.5 models demonstrate that DVAO significantly outperforms baseline methods, achieving a superior multi-objective Pareto frontier and robust training stability.

cs.CL↗

arXiv2026

Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search

Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.

cs.AI↗

arXiv2026

Evo-Retriever: LLM-Guided Curriculum Evolution with Viewpoint-Pathway Collaboration for Multimodal Document Retrieval

Visual-language models (VLMs) excel at data mappings, but real-world document heterogeneity and unstructuredness disrupt the consistency of cross-modal embeddings. Recent late-interaction methods enhance image-text alignment through multi-vector representations, yet traditional training with limited samples and static strategies cannot adapt to the model's dynamic evolution, causing cross-modal retrieval confusion. To overcome this, we introduce Evo-Retriever, a retrieval framework featuring an LLM-guided curriculum evolution built upon a novel Viewpoint-Pathway collaboration. First, we employ multi-view image alignment to enhance fine-grained matching via multi-scale and multi-directional perspectives. Then, a bidirectional contrastive learning strategy generates "hard queries" and establishes complementary learning paths for visual and textual disambiguation to rebalance supervision. Finally, the model-state summary from the above collaboration is fed into an LLM meta-controller, which adaptively adjusts the training curriculum using expert knowledge to promote the model's evolution. On ViDoRe V2 and MMEB (VisDoc), Evo-Retriever achieves state-of-the-art performance, with nDCG@5 scores of 65.2% and 77.1%.

cs.CV↗

arXiv2026

FAQ: Mitigating Quantization Error via Regenerating Calibration Data with Family-Aware Quantization

Although post-training quantization (PTQ) provides an efficient numerical compression scheme for deploying large language models (LLMs) on resource-constrained devices, the representativeness and universality of calibration data remain a core bottleneck in determining the accuracy of quantization parameters. Traditional PTQ methods typically rely on limited samples, making it difficult to capture the activation distribution during the inference phase, leading to biases in quantization parameters. To address this, we propose \textbf{FAQ} (Family-Aware Quantization), a calibration data regeneration framework that leverages prior knowledge from LLMs of the same family to generate high-fidelity calibration samples. Specifically, FAQ first inputs the original calibration samples into a larger LLM from the same family as the target model, regenerating a series of high-fidelity calibration data using a highly consistent knowledge system. Subsequently, this data, carrying Chain-of-Thought reasoning and conforming to the expected activation distribution, undergoes group competition under expert guidance to select the best samples, which are then re-normalized to enhance the effectiveness of standard PTQ. Experiments on multiple model series, including Qwen3-8B, show that FAQ reduces accuracy loss by up to 28.5\% compared to the baseline with original calibration data, demonstrating its powerful potential and contribution.

cs.LG↗

arXiv2025

VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.

cs.LG↗

arXiv2025

PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning

Critic-free reinforcement learning methods, particularly group policies, have attracted considerable attention for their efficiency in complex tasks. However, these methods rely heavily on multiple sampling and comparisons within the policy to estimate advantage, which may cause the policy to fall into local optimum and increase computational cost. To address these issues, we propose PVPO, an efficient reinforcement learning method enhanced by an advantage reference anchor and data pre-sampling. Specifically, we use the reference model to rollout in advance and employ the calculated reward score as a reference anchor. Our approach effectively corrects the cumulative bias introduced by intra-group comparisons and significantly reduces reliance on the number of rollouts during training. Meanwhile, the reference model can assess sample difficulty during data pre-sampling, enabling effective selection of high-gain data to improve training efficiency. Moreover, PVPO is orthogonal to other advanced critic-free RL algorithms, making it compatible with and complementary to these methods. Experiments conducted on nine datasets across two domains demonstrate that PVPO achieves State-Of-The-Art (SOTA) performance. Our approach not only demonstrates robust generalization across multiple tasks, but also exhibits scalable performance across models of varying scales.

cs.LG↗

arXiv2024

Optical vortex-antivortex crystallization in free space

Stable vortex lattices are basic dynamical patterns which have been demonstrated in physical systems including superconductor physics, Bose-Einstein condensates, hydrodynamics and optics. Vortex-antivortex (VAV) ensembles can be produced, self-organizing into the respective polar lattices. However, these structures are in general highly unstable due to the strong VAV attraction. Here, we demonstrate that multiple optical VAV clusters nested in the propagating coherent field can crystallize into patterns which preserve their lattice structures over distance up to several Rayleigh lengths. To explain this phenomenon, we present a model for effective interactions between the vortices and antivortices at different lattice sites. The observed VAV crystallization is a consequence of the globally balanced VAV couplings. As the crystallization does not require the presence of nonlinearities and appears in free space, it may find applications to high-capacity optical communications and multiparticle manipulations. Our findings suggest possibilities for constructing VAV complexes through the orbit-orbit couplings, which differs from the extensively studied spin-orbit couplings.

physics.optics↗

arXiv2024

Upper bound on the radius of the innermost photonsphere in the regular compact star spacetime

We study properties of the innermost photonsphere in the regular compact star background. We take the traceless energy-momentum tensor and dominant energy conditions. In the regular compact star background, we analytically obtain an upper bound on the radius of the innermost photonsphere as $r_γ^{in}\leqslant \frac{12}{5}M$, where $r_γ^{in}$ is the radius of the innermost photonsphere and $M$ is the total ADM mass of the asymptotically flat compact star spacetime.

gr-qc↗

arXiv2024

Photonic Spin-Orbit Coupling Induced by Deep-Subwavelength Structured Light

We demonstrate both theoretically and experimentally beam-dependent photonic spin-orbit coupling in a two-wave mixing process described by an equivalent of the Pauli equation in quantum mechanics. The considered structured light in the system is comprising a superposition of two orthogonal spin-orbit-coupled states defined as spin up and spin down equivalents. The spin-orbit coupling is manifested by prominent pseudo spin precession as well as spin-transport-induced orbital angular momentum generation in a photonic crystal film of wavelength thickness. The coupling effect is significantly enhanced by using a deep-subwavelength carrier envelope, different from previous studies which depend on materials. The beam-dependent coupling effect can find intriguing applications; for instance, it is used in precisely measuring variation of light with spatial resolution up to 15 nm.

physics.optics↗

arXiv2023

No long hair behaviors of ultra-compact objects

We investigate distributions of matter fields outside spherically symmetric ultra-compact objects in the asymptotically flat background. Based on the dominant energy condition and the non-negative trace condition, we analytically find a no long hair behavior, which states that the effective radius of matter field hairs cannot extend beyond the outermost null circular orbit.

gr-qc↗

arXiv2022

A no-go theorem for scalar fields with couplings from Ginzburg-Landau models

Recently Hod proved a no-go theorem that static scalar fields cannot form spherically symmetric boson stars in the asymptotically flat background. On the other side, scalar fields can be coupled to the gradient according to next-to-leading order Ginzburg-Landau models. In the present work, we extend Hod's discussions by considering couplings between static scalar fields and the field gradient. For a non-negative coupling parameter, we show that there is no asymptotically flat spherically symmetric boson stars made of coupled static scalar fields.

gr-qc↗