arXiv subjects

Zhihan Liu

Publications and source records attributed to Zhihan Liu.

27 records · Page 2Linked to original sources

arXiv2024

How Can LLM Guide RL? A Value-Based Approach

Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. However, RL algorithms may require extensive trial-and-error interactions to collect useful feedback for improvement. On the other hand, recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities for planning tasks, lacking the ability to autonomously refine their responses based on feedback. Therefore, in this paper, we study how the policy prior provided by the LLM can enhance the sample efficiency of RL algorithms. Specifically, we develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning, particularly when the difference between the ideal policy and the LLM-informed policy is small, which suggests that the initial policy is close to optimal, reducing the need for further exploration. Additionally, we present a practical algorithm SLINVIT that simplifies the construction of the value function and employs subgoals to reduce the search complexity. Our experiments across three interactive environments ALFWorld, InterCode, and BlocksWorld demonstrate that our method achieves state-of-the-art success rates and also surpasses previous RL and LLM approaches in terms of sample efficiency. Our code is available at https://github.com/agentification/Language-Integrated-VI.

cs.LG↗

arXiv2023

Non-topological logarithmic corrections in minimal gauged supergravity

We compute the logarithmic correction to the entropy of asymptotically AdS$_4$ black holes in minimal ${\cal N}=2$ gauged supergravity. We show that for extremal black holes the logarithmic correction computed in the near horizon geometry agrees with the result in the full geometry up to zero mode contributions, thus clarifying where the quantum degrees of freedom lie in AdS spacetimes. In contrast to flat space, we observe that the logarithmic correction for supersymmetric black holes can be non-topological in AdS as it is controlled by additional four-derivative terms other than the Euler density. The available microscopic data and results in 11d supergravity indicate that the full logarithmic correction is topological, which suggests that the topological nature of logarithmic corrections could serve as a diagnosis of whether a low-energy gravity theory admits an ultraviolet completion.

hep-th↗

arXiv2023

A Principled Framework for Knowledge-enhanced Large Language Model

Large Language Models (LLMs) are versatile, yet they often falter in tasks requiring deep and reliable reasoning due to issues like hallucinations, limiting their applicability in critical scenarios. This paper introduces a rigorously designed framework for creating LLMs that effectively anchor knowledge and employ a closed-loop reasoning process, enhancing their capability for in-depth analysis. We dissect the framework to illustrate the contribution of each component to the LLMs' performance, offering a theoretical assurance of improved reasoning under well-defined assumptions.

cs.CL↗

arXiv2023

Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration

In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration. However, in order to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as optimization within data-dependent level-sets or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called \textit{Maximize to Explore} (\texttt{MEX}), which only needs to optimize \emph{unconstrainedly} a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that \texttt{MEX} achieves a sublinear regret with general function approximations for Markov decision processes (MDP) and is further extendable to two-player zero-sum Markov games (MG). Meanwhile, we adapt deep RL baselines to design practical versions of \texttt{MEX}, in both model-free and model-based manners, which can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards. Compared with existing sample-efficient online RL algorithms with general function approximations, \texttt{MEX} achieves similar sample efficiency while enjoying a lower computational cost and is more compatible with modern deep RL methods.

cs.LG↗

arXiv2023

Sample-Efficient Multi-Agent RL: An Optimization Perspective

We study multi-agent reinforcement learning (MARL) for the general-sum Markov Games (MGs) under the general function approximation. In order to find the minimum assumption for sample-efficient learning, we introduce a novel complexity measure called the Multi-Agent Decoupling Coefficient (MADC) for general-sum MGs. Using this measure, we propose the first unified algorithmic framework that ensures sample efficiency in learning Nash Equilibrium, Coarse Correlated Equilibrium, and Correlated Equilibrium for both model-based and model-free MARL problems with low MADC. We also show that our algorithm provides comparable sublinear regret to the existing works. Moreover, our algorithm combines an equilibrium-solving oracle with a single objective optimization subprocedure that solves for the regularized payoff of each deterministic joint policy, which avoids solving constrained optimization problems within data-dependent constraints (Jin et al. 2020; Wang et al. 2023) or executing sampling procedures with complex multi-objective optimization problems (Foster et al. 2023), thus being more amenable to empirical implementation.

cs.LG↗

arXiv2023

Guarded Policy Optimization with Imperfect Online Demonstrations

The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

cs.LG↗

arXiv2022

The Phase Diagram of BPS Black Holes in AdS$_5$

Motivated by recent studies of supersymmetric black holes, we revisit the phase diagram of AdS black holes, whether BPS or not, with particular emphasis on the role of rotation. We develop BPS thermodynamics systematically and, in many explicit examples, we study its striking similarities with more familiar AdS black holes, as well as some differences. We highlight an important fugacity that preserves BPS saturation but is not captured by the supersymmetric index.

hep-th↗

arXiv2021

Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation

In generative adversarial imitation learning (GAIL), the agent aims to learn a policy from an expert demonstration so that its performance cannot be discriminated from the expert policy on a certain predefined reward set. In this paper, we study GAIL in both online and offline settings with linear function approximation, where both the transition and reward function are linear in the feature maps. Besides the expert demonstration, in the online setting the agent can interact with the environment, while in the offline setting the agent only accesses an additional dataset collected by a prior. For online GAIL, we propose an optimistic generative adversarial policy optimization algorithm (OGAP) and prove that OGAP achieves $\widetilde{\mathcal{O}}(H^2 d^{3/2}K^{1/2}+KH^{3/2}dN_1^{-1/2})$ regret. Here $N_1$ represents the number of trajectories of the expert demonstration, $d$ is the feature dimension, and $K$ is the number of episodes. For offline GAIL, we propose a pessimistic generative adversarial policy optimization algorithm (PGAP). For an arbitrary additional dataset, we obtain the optimality gap of PGAP, achieving the minimax lower bound in the utilization of the additional dataset. Assuming sufficient coverage on the additional dataset, we show that PGAP achieves $\widetilde{\mathcal{O}}(H^{2}dK^{-1/2} +H^2d^{3/2}N_2^{-1/2}+H^{3/2}dN_1^{-1/2} \ )$ optimality gap. Here $N_2$ represents the number of trajectories of the additional dataset with sufficient coverage.

cs.LG↗

arXiv2019

3D Trajectory Optimization for Secure UAV Communication with CoMP Reception

This paper studies a secrecy unmanned aerial vehicle (UAV) communication system with coordinated multi-point (CoMP) reception, in which one UAV sends confidential messages to a set of distributed ground nodes (GNs) that can cooperate in signal detection, in the presence of several colluding suspicious eavesdroppers. Different from prior works considering the two-dimensional (2D) horizontal trajectory design in the non-CoMP scenario, this paper additionally exploits the UAV's vertical trajectory (or altitude) control for further improving the secrecy communication performance with CoMP. In particular, we jointly optimize the three dimensional (3D) trajectory and transmit power allocation of the UAV to maximize the average secrecy rate at GNs over a particular flight period, subject to the UAV's maximum flight speed and maximum transmit power constraints. To solve the non-convex optimization problem, we propose an alternating-optimization-based approach, which optimizes the transmit power allocation and trajectory design in an alternating manner, by convex optimization and successive convex approximation (SCA), respectively. Numerical results show that in the scenario with CoMP reception, our proposed 3D trajectory optimization significantly outperforms the conventional 2D horizontal trajectory design, by exploiting the additional degree of freedom in vertical trajectory.

eess.SP↗