arXiv subjects

Xuyang Chen

Publications and source records attributed to Xuyang Chen.

42 records · Page 3Linked to original sources

arXiv2024

Finite-time analysis of single-timescale actor-critic

Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an $ε$-approximate stationary point with $\widetilde{\mathcal{O}}(ε^{-2})$ sample complexity under standard assumptions, which can be further improved to $\mathcal{O}(ε^{-2})$ under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well.

cs.LG↗

arXiv2023

Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedback

In recent times, significant advancements have been made in delving into the optimization landscape of policy gradient methods for achieving optimal control in linear time-invariant (LTI) systems. Compared with state-feedback control, output-feedback control is more prevalent since the underlying state of the system may not be fully observed in many practical settings. This paper analyzes the optimization landscape inherent to policy gradient methods when applied to static output feedback (SOF) control in discrete-time LTI systems subject to quadratic cost. We begin by establishing crucial properties of the SOF cost, encompassing coercivity, L-smoothness, and M-Lipschitz continuous Hessian. Despite the absence of convexity, we leverage these properties to derive novel findings regarding convergence (and nearly dimension-free rate) to stationary points for three policy gradient methods, including the vanilla policy gradient method, the natural policy gradient method, and the Gauss-Newton method. Moreover, we provide proof that the vanilla policy gradient method exhibits linear convergence towards local minima when initialized near such minima. The paper concludes by presenting numerical examples that validate our theoretical findings. These results not only characterize the performance of gradient descent for optimizing the SOF problem but also provide insights into the effectiveness of general policy gradient methods within the realm of reinforcement learning.

math.OC↗

arXiv2023

Efficient Q-Learning over Visit Frequency Maps for Multi-agent Exploration of Unknown Environments

The robot exploration task has been widely studied with applications spanning from novel environment mapping to item delivery. For some time-critical tasks, such as rescue catastrophes, the agent is required to explore as efficiently as possible. Recently, Visit Frequency-based map representation achieved great success in such scenarios by discouraging repetitive visits with a frequency-based penalty. However, its relatively large size and single-agent settings hinder its further development. In this context, we propose Integrated Visit Frequency Map, which encodes identical information as Visit Frequency Map with a more compact size, and a visit frequency-based multi-agent information exchange and control scheme that is able to accommodate both representations. Through tests in diverse settings, the results indicate our proposed methods can achieve a comparable level of performance of VFM with lower bandwidth requirements and generalize well to different multi-agent setups including real-world environments.

cs.RO↗

arXiv2023

Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulator

The actor-critic (AC) reinforcement learning algorithms have been the powerhouse behind many challenging applications. Nevertheless, its convergence is fragile in general. To study its instability, existing works mostly consider the uncommon double-loop variant or basic models with finite state and action space. We investigate the more practical single-sample two-timescale AC for solving the canonical linear quadratic regulator (LQR) problem, where the actor and the critic update only once with a single sample in each iteration on an unbounded continuous state and action space. Existing analysis cannot conclude the convergence for such a challenging case. We develop a new analysis framework that allows establishing the global convergence to an $ε$-optimal solution with at most an $\mathcal{O}(ε^{-2.5})$ sample complexity. To our knowledge, this is the first finite-time convergence analysis for the single sample two-timescale AC for solving LQR with global optimality. The sample complexity improves those of other variants by orders, which sheds light on the practical wisdom of single sample algorithms. We also further validate our theoretical findings via comprehensive simulation comparisons.

cs.LG↗

arXiv2021

Phase transition and cascading collapse in binary decision-making dynamics

Binary decision-making process is ubiquitous in social life and is of vital significance in many real-world issues, ranging from public health to political campaigns. While continuous opinion evolution independent of discrete choice behavior has been extensively studied, few works unveil how the group binary decision-making result is determined by the coupled dynamics of these two processes. To this end, we propose an agent-based model to study the collective behaviors of individual binary decision-making process through competitive opinion dynamics on social networks. Three key factors are considered: bounded confidence that describes the cognitive scope of the population, stubbornness level that characterizes the opinion updating speed, and the opinion strength that represents the asymmetry power or attractiveness of the two choices. We find that bounded confidence plays an important role in determining competing evolution results. As bounded confidence grows, population opinions experience polarization to consensus, leading to the emergence of phase transition from co-existence to winner-takes-all state under binary decisions. Of particular interest, we show how the combined effects of bounded confidence and asymmetry opinion strength may reverse the initial supportive advantage in competitive dynamics. Notably, our model qualitatively reproduces the important dynamical pattern during a brutal competition, namely, cascading collapse, as observed by real data. Finally and intriguingly, we find that individual cognitive heterogeneity can bring about randomness and unpredictability in binary decision-making process, leading to the emergence of indeterministic oscillation. Our results reveal how the diverse behavioral patterns of binary decision-making can be interpreted by the complicated interactions of the proposed elements, which provides important insights toward competitive dynamics

physics.soc-ph↗

arXiv2018

AXNet: ApproXimate computing using an end-to-end trainable neural network

Neural network based approximate computing is a universal architecture promising to gain tremendous energy-efficiency for many error resilient applications. To guarantee the approximation quality, existing works deploy two neural networks (NNs), e.g., an approximator and a predictor. The approximator provides the approximate results, while the predictor predicts whether the input data is safe to approximate with the given quality requirement. However, it is non-trivial and time-consuming to make these two neural network coordinate---they have different optimization objectives---by training them separately. This paper proposes a novel neural network structure---AXNet---to fuse two NNs to a holistic end-to-end trainable NN. Leveraging the philosophy of multi-task learning, AXNet can tremendously improve the invocation (proportion of safe-to-approximate samples) and reduce the approximation error. The training effort also decrease significantly. Experiment results show 50.7% more invocation and substantial cuts of training time when compared to existing neural network based approximate computing framework.

cs.LG↗