arXiv subjects

Kaiqing Zhang

Publications and source records attributed to Kaiqing Zhang.

84 records · Page 5Linked to original sources

arXiv2019

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

This paper extends off-policy reinforcement learning to the multi-agent case in which a set of networked agents communicating with their neighbors according to a time-varying graph collaboratively evaluates and improves a target policy while following a distinct behavior policy. To this end, the paper develops a multi-agent version of emphatic temporal difference learning for off-policy policy evaluation, and proves convergence under linear function approximation. The paper then leverages this result, in conjunction with a novel multi-agent off-policy policy gradient theorem and recent work in both multi-agent on-policy and single-agent off-policy actor-critic methods, to develop and give convergence guarantees for a new multi-agent off-policy actor-critic algorithm.

cs.LG↗

arXiv2019

A Convergence Result for Regularized Actor-Critic Methods

In this paper, we present a probability one convergence proof, under suitable conditions, of a certain class of actor-critic algorithms for finding approximate solutions to entropy-regularized MDPs using the machinery of stochastic approximation. To obtain this overall result, we prove the convergence of policy evaluation with general regularizers when using linear approximation architectures and show convergence of entropy-regularized policy improvement.

cs.LG↗

arXiv2019

Online Planning for Decentralized Stochastic Control with Partial History Sharing

In decentralized stochastic control, standard approaches for sequential decision-making, e.g. dynamic programming, quickly become intractable due to the need to maintain a complex information state. Computational challenges are further compounded if agents do not possess complete model knowledge. In this paper, we take advantage of the fact that in many problems agents share some common information, or history, termed partial history sharing. Under this information structure the policy search space is greatly reduced. We propose a provably convergent, online tree-search based algorithm that does not require a closed-form model or explicit communication among agents. Interestingly, our algorithm can be viewed as a generalization of several existing heuristic solvers for decentralized partially observable Markov decision processes. To demonstrate the applicability of the model, we propose a novel collaborative intrusion response model, where multiple agents (defenders) possessing asymmetric information aim to collaboratively defend a computer network. Numerical results demonstrate the performance of our algorithm.

cs.LG↗

arXiv2019

A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning

This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors. A randomized communication-efficient multi-agent actor-critic algorithm is proposed for possibly unidirectional communication relationships depicted by a directed graph. It is shown that the algorithm can solve the problem for strongly connected graphs by allowing each agent to transmit only two scalar-valued variables at one time.

cs.LG↗

arXiv2018

Distributed Learning of Average Belief Over Networks Using Sequential Observations

This paper addresses the problem of distributed learning of average belief with sequential observations, in which a network of $n>1$ agents aim to reach a consensus on the average value of their beliefs, by exchanging information only with their neighbors. Each agent has sequentially arriving samples of its belief in an online manner. The neighbor relationships among the $n$ agents are described by a graph which is possibly time-varying, whose vertices correspond to agents and whose edges depict neighbor relationships. Two distributed online algorithms are introduced for undirected and directed graphs, which are both shown to converge to the average belief almost surely. Moreover, the sequences generated by both algorithms are shown to reach consensus with an $O(1/t)$ rate with high probability, where $t$ is the number of iterations. For undirected graphs, the corresponding algorithm is modified for the case with quantized communication and limited precision of the division operation. It is shown that the modified algorithm causes all $n$ agents to either reach a quantized consensus or enter a small neighborhood around the average of their beliefs. Numerical simulations are then provided to corroborate the theoretical results.

cs.MA↗

arXiv2018

Distributed Equilibrium-Learning for Power Network Voltage Control With a Locally Connected Communication Network

In current power distribution systems, one of the most challenging operation tasks is to coordinate the network- wide distributed energy resources (DERs) to maintain the stability of voltage magnitude of the system. This voltage control task has been investigated actively under either distributed optimization-based or local feedback control-based characterizations. The former architecture requires a strongly-connected communication network among all DERs for implementing the optimization algorithms, a scenario not yet realistic in most of the existing distribution systems with under-deployed communication infrastructure. The latter one, on the other hand, has been proven to suffer from loss of network-wide op- erational optimality. In this paper, we propose a game-theoretic characterization for semi-local voltage control with only a locally connected communication network. We analyze the existence and uniqueness of the generalized Nash equilibrium (GNE) for this characterization and develop a fully distributed equilibrium-learning algorithm that relies on only neighbor-to-neighbor information exchange. Provable convergence results are provided along with numerical tests which corroborate the robust convergence property of the proposed algorithm.

eess.SY↗

arXiv2018

Dynamic Power Distribution System Management With a Locally Connected Communication Network

Coordinated optimization and control of distribution-level assets can enable a reliable and optimal integration of massive amount of distributed energy resources (DERs) and facilitate distribution system management (DSM). Accordingly, the objective is to coordinate the power injection at the DERs to maintain certain quantities across the network, e.g., voltage magnitude, line flows, or line losses, to be close to a desired profile. By and large, the performance of the DSM algorithms has been challenged by two factors: i) the possibly non strongly connected communication network over DERs that hinders the coordination; ii) the dynamics of the real system caused by the DERs with heterogeneous capabilities, time-varying operating conditions, and real-time measurement mismatches. In this paper, we investigate the modeling and algorithm design and analysis with the consideration of these two factors. In particular, a game-theoretic characterization is first proposed to account for a locally connected communication network over DERs, along with the analysis of the existence and uniqueness of the Nash equilibrium (NE) therein. To achieve the equilibrium in a distributed fashion, a projected-gradient-based asynchronous DSM algorithm is then advocated. The algorithm performance, including the convergence speed and the tracking error, is analytically guaranteed under the dynamic setting. Extensive numerical tests on both synthetic and realistic cases corroborate the analytical results derived.

eess.SY↗

arXiv2018

Optimal Joint Bidding and Pricing of Profit-seeking Load Serving Entity

The demand response provides an opportunity for load serving entities (LSEs) that operate retail electricity markets (REMs) to strategically purchase energy and provide reserves in wholesale electricity markets (WEMs). This paper concerns with the problem of simultaneously determining the optimal energy bids and reserve offers an LSE submits to the WEM as well as the optimal energy and reserve prices it sets in the REM so as to maximize its profit. To this end, we explicitly model the tri-layer market structure that consists of a WEM, a REM, and a set of end user customers, so as to capture the coupling between the bidding problem and the pricing problem. Based on the tri-layer market model, we then formulate the joint bidding and pricing problem as a bi-level programming problem and further transform it into a single-level mixed integer linear programming problem, which can be solved efficiently. Numerical studies using the IEEE test cases are presented to illustrate the application of the proposed methodology as well as to reveal several interesting characteristics of the LSE's profit-seeking behavior.

math.OC↗

arXiv2018

Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

We consider the problem of \emph{fully decentralized} multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.

cs.LG↗

arXiv2017

Extending the photon energy coverage of an x-ray self-seeding FEL via the reverse taper enhanced harmonic generation technique

In this paper, a simple method is proposed to extend the photon energy range of a soft x-ray self-seeding free-electron laser (FEL). A normal monochromator is first applied to purify the FEL spectrum and provide a coherent seeding signal. This coherent signal then interacts with the electron beam in the following reverse tapered undulator section to generate strong coherent microbunchings while maintain the good quality of the electron beam. After that, the pre-bunched electron beam is sent into the third undulator section which resonates at a target high harmonic of the seed to amplify the coherent radiation at shorter wavelength. Three dimensional simulations have been performed and the results demonstrate that the photon energy gap between 1.5 keV and 4.5 keV of the self-seeding scheme can be fully covered and 100 GW-level peak power can be achieved by using the proposed technique.

physics.acc-ph↗

arXiv2015

Indoor Localization Algorithm For Smartphones

Increasing sources of sensor measurements and prior knowledge have become available for indoor localization on smartphones. How to effectively utilize these sources for enhancing localization accuracy is an important yet challenging problem. In this paper, we present an area state-aided localization algorithm that exploits various sources of information. Specifically, we introduce the concept of area state to indicate the area where the user is on an indoor map. The position of the user is then estimated using inertial measurement unit (IMU) measurements with the aid of area states. The area states are in turn updated based on the position estimates. To avoid accumulated errors of IMU measurements, our algorithm uses WiFi received signal strength indicator (RSSI) to indicate the vicinity of the user to the routers. The experiment results show that our system can achieve satisfactory localization accuracy in a typical indoor environment.

cs.NI↗

arXiv2015

Enhanced Multi-Parameter Cognitive Architecture for Future Wireless Communications

The very original concept of cognitive radio (CR) raised by Mitola targets at all the environment parameters, including those in physical layer, MAC layer, application layer as well as the information extracted from reasoning. Hence the first CR is also referred to as "full cognitive radio". However, due to its difficult implementation, FCC and Simon Haykin separately proposed a much more simplified definition, in which CR mainly detects one single parameter, i.e., spectrum occupancy, and is also called as "spectrum sensing cognitive radio". With the rapid development of wireless communication, the infrastructure of a wireless system becomes much more complicated while the functionality at every node is desired to be as intelligent as possible, say the self-organized capability in the approaching 5G cellular networks. It is then interesting to re-look into Mitola's definition and think whether one could, besides obtaining the "on/off" status of the licensed user only, achieve more parameters in a cognitive way. In this article, we propose a new cognitive architecture targeting at multiple parameters in future cellular networks, which is a one step further towards the "full cognition" compared to the most existing CR research. The new architecture is elaborated in detailed stages, and three representative examples are provided based on the recent research progress to illustrate the feasibility as well as the validity of the proposed architecture.

cs.IT↗