arXiv subjects

Guojian Wang

Publications and source records attributed to Guojian Wang.

10 recordsLinked to original sources

arXiv2026

Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shift

External trajectories can improve offline decision-sequence learning, but dynamics shift may make some source subsequences inconsistent with the target environment. We study how to fuse such trajectories with limited target data for Decision Transformer learning under dynamics shift. We propose Target-Aligned Fusion (TAF), a principled framework that derives source-data fusion from a target-domain Bellman-risk criterion. Our analysis bounds this risk by two measurable data-alignment quantities: $Δ_m$, the state-structure mismatch of retained fragments, and $Δ_w$, the weighted transport cost from source to target transitions. This decomposition yields a gate--then--weight rule: source fragments are first filtered by target-side state-structure alignment, and retained transitions are then reweighted by local target feasibility. We instantiate this principle as TAF-DT, which uses maximum mean discrepancy (MMD) for fragment selection, optimal transport for feasibility-aware weighting, and the resulting fused law for advantage-token relabeling and Q-regularized Transformer training. Across gravity, kinematic, and morphology shifts on D4RL-style control tasks, TAF-DT achieves the strongest aggregate performance against strong offline RL and sequence-model baselines and produces more stable stitch-junction sequence semantics. Overall, these results indicate that aligning external trajectories to target-domain structure and feasibility is a practical way to exploit source data under dynamics shift.

cs.LG↗

arXiv2026

VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning

Offline reinforcement learning (RL) learns effective policies from pre-collected datasets, offering a practical solution for applications where online interactions are risky or costly. Model-based approaches are particularly advantageous for offline RL, owing to their data efficiency and generalizability. However, due to inherent model errors, model-based methods often artificially introduce conservatism guided by heuristic uncertainty estimation, which can be unreliable. In this paper, we introduce VIPO, a novel model-based offline RL algorithm that incorporates self-supervised feedback from value estimation to enhance model training. Specifically, the model is learned by additionally minimizing the inconsistency between the value learned directly from the offline data and the value estimated from the model. We perform comprehensive evaluations from multiple perspectives to show that VIPO can learn a highly accurate model efficiently and consistently outperform existing methods. In particular, it achieves state-of-the-art performance on almost all tasks in both D4RL and NeoRL benchmarks. Overall, VIPO offers a general framework that can be readily integrated into existing model-based offline RL algorithms to systematically enhance model accuracy.

cs.LG↗

arXiv2025

Preference-Guided Reinforcement Learning for Efficient Exploration

In this paper, we investigate preference-based reinforcement learning (PbRL), which enables reinforcement learning (RL) agents to learn from human feedback. This is particularly valuable when defining a fine-grain reward function is not feasible. However, this approach is inefficient and impractical for promoting deep exploration in hard-exploration tasks with long horizons and sparse rewards. To tackle this issue, we introduce LOPE: \textbf{L}earning \textbf{O}nline with trajectory \textbf{P}reference guidanc\textbf{E}, an end-to-end preference-guided RL framework that enhances exploration efficiency in hard-exploration tasks. Our intuition is that LOPE directly adjusts the focus of online exploration by considering human feedback as guidance, thereby avoiding the need to learn a separate reward model from preferences. Specifically, LOPE includes a two-step sequential policy optimization technique consisting of trust-region-based policy improvement and preference guidance steps. We reformulate preference guidance as a trajectory-wise state marginal matching problem that minimizes the maximum mean discrepancy distance between the preferred trajectories and the learned policy. Furthermore, we provide a theoretical analysis to characterize the performance improvement bound and evaluate the effectiveness of the LOPE. When assessed in various challenging hard-exploration environments, LOPE outperforms several state-of-the-art methods in terms of convergence rate and overall performance.The code used in this study is available at https://github.com/buaawgj/LOPE.

cs.LG↗

arXiv2024

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

The sparsity of reward feedback remains a challenging problem in online deep reinforcement learning (DRL). Previous approaches have utilized offline demonstrations to achieve impressive results in multiple hard tasks. However, these approaches place high demands on demonstration quality, and obtaining expert-like actions is often costly and unrealistic. To tackle these problems, we propose a simple and efficient algorithm called Policy Optimization with Smooth Guidance (POSG), which leverages a small set of state-only demonstrations (where expert action information is not included in demonstrations) to indirectly make approximate and feasible long-term credit assignments and facilitate exploration. Specifically, we first design a trajectory-importance evaluation mechanism to determine the quality of the current trajectory against demonstrations. Then, we introduce a guidance reward computation technology based on trajectory importance to measure the impact of each state-action pair, fusing the demonstrator's state distribution with reward information into the guidance reward. We theoretically analyze the performance improvement caused by smooth guidance rewards and derive a new worst-case lower bound on the performance improvement. Extensive results demonstrate POSG's significant advantages in control performance and convergence speed in four sparse-reward environments, including the grid-world maze, Hopper-v4, HalfCheetah-v4, and Ant maze. Notably, the specific metrics and quantifiable results are investigated to demonstrate the superiority of POSG.

cs.LG↗

arXiv2024

Trajectory-Oriented Policy Optimization with Sparse Rewards

Mastering deep reinforcement learning (DRL) proves challenging in tasks featuring scant rewards. These limited rewards merely signify whether the task is partially or entirely accomplished, necessitating various exploration actions before the agent garners meaningful feedback. Consequently, the majority of existing DRL exploration algorithms struggle to acquire practical policies within a reasonable timeframe. To address this challenge, we introduce an approach leveraging offline demonstration trajectories for swifter and more efficient online RL in environments with sparse rewards. Our pivotal insight involves treating offline demonstration trajectories as guidance, rather than mere imitation, allowing our method to learn a policy whose distribution of state-action visitation marginally matches that of offline demonstrations. We specifically introduce a novel trajectory distance relying on maximum mean discrepancy (MMD) and cast policy optimization as a distance-constrained optimization problem. We then illustrate that this optimization problem can be streamlined into a policy-gradient algorithm, integrating rewards shaped by insights from offline demonstrations. The proposed algorithm undergoes evaluation across extensive discrete and continuous control tasks with sparse and misleading rewards. The experimental findings demonstrate the significant superiority of our proposed algorithm over baseline methods concerning diverse exploration and the acquisition of an optimal policy.

cs.LG↗

arXiv2024

Learning Diverse Policies with Soft Self-Generated Guidance

Reinforcement learning (RL) with sparse and deceptive rewards is challenging because non-zero rewards are rarely obtained. Hence, the gradient calculated by the agent can be stochastic and without valid information. Recent studies that utilize memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods often require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This paper develops an approach that uses diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm combines a policy improvement step with an additional exploration step using offline demonstration data. The main contribution of this paper is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories while still being able to learn without rewards and approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the team's diversity and regulate exploration. The proposed algorithm is evaluated on discrete and continuous control tasks with sparse and deceptive rewards. Compared with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and avoiding local optima.

cs.LG↗

arXiv2023

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Deep reinforcement learning (DRL) faces significant challenges in addressing the hard-exploration problems in tasks with sparse or deceptive rewards and large state spaces. These challenges severely limit the practical application of DRL. Most previous exploration methods relied on complex architectures to estimate state novelty or introduced sensitive hyperparameters, resulting in instability. To mitigate these issues, we propose an efficient adaptive trajectory-constrained exploration strategy for DRL. The proposed method guides the policy of the agent away from suboptimal solutions by leveraging incomplete offline demonstrations as references. This approach gradually expands the exploration scope of the agent and strives for optimality in a constrained optimization manner. Additionally, we introduce a novel policy-gradient-based optimization algorithm that utilizes adaptively clipped trajectory-distance rewards for both single- and multi-agent reinforcement learning. We provide a theoretical analysis of our method, including a deduction of the worst-case approximation error bounds, highlighting the validity of our approach for enhancing exploration. To evaluate the effectiveness of the proposed method, we conducted experiments on two large 2D grid world mazes and several MuJoCo tasks. The extensive experimental results demonstrate the significant advantages of our method in achieving temporally extended exploration and avoiding myopic and suboptimal behaviors in both single- and multi-agent settings. Notably, the specific metrics and quantifiable results further support these findings. The code used in the study is available at \url{https://github.com/buaawgj/TACE}.

cs.LG↗

arXiv2023

Development of low-threshold detectors for low-mass dark matter searches with a p-type germanium detector operated at cryogenic temperature

This study investigates new technology for enhancing the sensitivity of low-mass dark matter detection by analyzing charge transport in a p-type germanium detector at 5.2 K. To achieve low-threshold detectors, precise calculations of the binding energies of dipole and cluster dipole states, as well as the cross-sections of trapping affected by the electric field, are essential. The detector was operated in two modes: depleted at 77 K before cooling to 5.2 K and cooled directly to 5.2 K with various bias voltages. Our results indicate that the second mode produces lower binding energies and suggests different charge states under varying operating modes. Notably, our measurements of the dipole and cluster dipole state binding energies at zero fields were $8.716\pm 0.435$ meV and $6.138\pm 0.308$ meV, respectively. These findings have strong implications for the development of low-threshold detectors for detecting low-mass dark matter in the future.

physics.ins-det↗

arXiv2023

Development of Low-Threshold Detectors for Low-Mass Dark Matter Searches Using an N-Type Germanium Detector at 5.2 K

We investigated charge transport in an n-type germanium detector at 5.2 K to explore new technology for enhancing low-mass dark matter detection sensitivity. Calculations of dipole and cluster dipole state binding energies and electric field-dependent trapping cross-sections are critical to developing low-threshold detectors. The detector operates in two modes: depleting at 77K before cooling, or directly cooling to 5.2 K and applying different bias voltages. Results indicated lower binding energy of charge states in the second mode, at zero field and under an electric field, suggesting different charge states formed under different operating modes. Measured cluster dipole and dipole state binding energies at zero field were 7.884$\pm$0.644 meV and 8.369$\pm$0.748 meV, respectively, signifying high low-threshold potential for low-mass dark matter searches in the future.

physics.ins-det↗

arXiv2015

Crystal growth and detector performance of large size high-purity Ge crystals

High-purity germanium crystals approximately 12 cm in diameter were grown in a hydrogen atmosphere using the Czochralski method. The dislocation density of the crystals was determined to be in the range of 2000 - 4200 cm-2, which meets a requirement for use as a radiation detector. The axial and radial distributions of impurities in the crystals were measured and are discussed. A planar detector was also fabricated from one of the crystals and then evaluated for electrical and spectral performance. Measurements of gamma-ray spectra from Cs-137 and Am-241 sources demonstrate that the detector has excellent energy resolution.

physics.ins-det↗