arXiv subjects

Zeshen Tang

Publications and source records attributed to Zeshen Tang.

3 recordsLinked to original sources

arXiv2024

HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Control

Goal-conditioned hierarchical reinforcement learning (HRL) decomposes complex reaching tasks into a sequence of simple subgoal-conditioned tasks, showing significant promise for addressing long-horizon planning in large-scale environments. This paper bridges the goal-conditioned HRL based on graph-based planning to brain mechanisms, proposing a hippocampus-striatum-like dual-controller hypothesis. Inspired by the brain mechanisms of organisms (i.e., the high-reward preferences observed in hippocampal replay) and instance-based theory, we propose a high-return sampling strategy for constructing memory graphs, improving sample efficiency. Additionally, we derive a model-free lower-level Q-function gradient penalty to resolve the model dependency issues present in prior work, improving the generalization of Lipschitz constraints in applications. Finally, we integrate these two extensions, High-reward Graph and model-free Gradient Penalty (HG2P), into the state-of-the-art framework ACLG, proposing a novel goal-conditioned HRL framework, HG2P+ACLG. Experimentally, the results demonstrate that our method outperforms state-of-the-art goal-conditioned HRL algorithms on a variety of long-horizon navigation tasks and robotic manipulation tasks.

cs.LG↗

arXiv2023

Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened inter-level communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting inter-level cooperation. Here, we propose a goal-conditioned HRL framework named Guided Cooperation via Model-based Rollout (GCMR), aiming to bridge inter-layer information synchronization and cooperation by exploiting forward dynamics. Firstly, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Secondly, to prevent disruption by the unseen subgoals and states, lower-level Q-function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Thirdly, we propose a one-step rollout-based planning, using higher-level critics to guide the lower-level policy. Specifically, we estimate the value of future states of the lower-level policy using the higher-level critic function, thereby transmitting global task information downwards to avoid local pitfalls. These three critical components in GCMR are expected to facilitate inter-level cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of HIGL, namely ACLG, yields more stable and robust policy improvement compared to various baselines and significantly outperforms previous state-of-the-art algorithms.

cs.LG↗

arXiv2018

SVM-Based Sea-Surface Small Target Detection: A False-Alarm-Rate-Controllable Approach

In this letter, we consider the varying detection environments to address the problem of detecting small targets within sea clutter. We first extract three simple yet practically discriminative features from the returned signals in the time and frequency domains and then fuse them into a 3-D feature space. Based on the constructed space, we then adopt and elegantly modify the support vector machine (SVM) to design a learning-based detector that enfolds the false alarm rate (FAR). Most importantly, our proposed detector can flexibly control the FAR by simply adjusting two introduced parameters, which facilitates to regulate detector's sensitivity to the outliers incurred by the sea spikes and to fairly evaluate the performance of different detection algorithms. Experimental results demonstrate that our proposed detector significantly improves the detection probability over several existing classical detectors in both low signal to clutter ratio (SCR) (up to 58%) and low FAR (up to 40%) cases.

cs.IT↗