arXiv subjects

Tamer Başar

Publications and source records attributed to Tamer Başar.

At least 127 records · Page 7Linked to original sources

arXiv2019

Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances

Multi-agent reinforcement learning (MARL) has long been a significant and everlasting research topic in both machine learning and control. With the recent development of (single-agent) deep RL, there is a resurgence of interests in developing new MARL algorithms, especially those that are backed by theoretical analysis. In this paper, we review some recent advances a sub-area of this topic: decentralized MARL with networked agents. Specifically, multiple agents perform sequential decision-making in a common environment, without the coordination of any central controller. Instead, the agents are allowed to exchange information with their neighbors over a communication network. Such a setting finds broad applications in the control and operation of robots, unmanned vehicles, mobile sensor networks, and smart grid. This review is built upon several our research endeavors in this direction, together with some progresses made by other researchers along the line. We hope this review to inspire the devotion of more research efforts to this exciting yet challenging area.

cs.LG↗

arXiv2019

Strategic Inference with a Single Private Sample

Motivated by applications in cyber security, we develop a simple game model for describing how a learning agent's private information influences an observing agent's inference process. The model describes a situation in which one of the agents (attacker) is deciding which of two targets to attack, one with a known reward and another with uncertain reward. The attacker receives a single private sample from the uncertain target's distribution and updates its belief of the target quality. The other agent (defender) knows the true rewards, but does not see the sample that the attacker has received. This leads to agents possessing asymmetric information: the attacker is uncertain over the parameter of the distribution, whereas the defender is uncertain about the observed sample. After the attacker updates its belief, both the attacker and the defender play a simultaneous move game based on their respective beliefs. We offer a characterization of the pure strategy equilibria of the game and explain how the players' decisions are influenced by their prior knowledge and the payoffs/costs.

cs.GT↗

arXiv2019

Dynamic Contract Design for Systemic Cyber Risk Management of Interdependent Enterprise Networks

The interconnectivity of cyber and physical systems and Internet of things has created ubiquitous concerns of cyber threats for enterprise system managers. It is common that the asset owners and enterprise network operators need to work with cybersecurity professionals to manage the risk by remunerating them for their efforts that are not directly observable. In this paper, we use a principal-agent framework to capture the service relationships between the two parties, i.e., the asset owner (principal) and the cyber risk manager (agent). Specifically, we consider a dynamic systemic risk management problem with asymmetric information where the principal can only observe cyber risk outcomes of the enterprise network rather than directly the efforts that the manager expends on protecting the resources. Under this information pattern, the principal aims to minimize the systemic cyber risks by designing a dynamic contract that specifies the compensation flows and the anticipated efforts of the manager by taking into account his incentives and rational behaviors. We formulate a bi-level mechanism design problem for dynamic contract design within the framework of a class of stochastic differential games. We show that the principal has rational controllability of the systemic risk by designing an incentive compatible estimator of the agent's hidden efforts. We characterize the optimal solution by reformulating the problem as a stochastic optimal control program which can be solved using dynamic programming. We further investigate a benchmark scenario with complete information and identify conditions that yield zero information rent and lead to a new certainty equivalence principle for principal-agent problems. Finally, case studies over networked systems are carried out to illustrate the theoretical results obtained.

eess.SY↗

arXiv2019

Online Planning for Decentralized Stochastic Control with Partial History Sharing

In decentralized stochastic control, standard approaches for sequential decision-making, e.g. dynamic programming, quickly become intractable due to the need to maintain a complex information state. Computational challenges are further compounded if agents do not possess complete model knowledge. In this paper, we take advantage of the fact that in many problems agents share some common information, or history, termed partial history sharing. Under this information structure the policy search space is greatly reduced. We propose a provably convergent, online tree-search based algorithm that does not require a closed-form model or explicit communication among agents. Interestingly, our algorithm can be viewed as a generalization of several existing heuristic solvers for decentralized partially observable Markov decision processes. To demonstrate the applicability of the model, we propose a novel collaborative intrusion response model, where multiple agents (defenders) possessing asymmetric information aim to collaboratively defend a computer network. Numerical results demonstrate the performance of our algorithm.

cs.LG↗

arXiv2019

Asymptotic Behavior of Conjunctive Boolean Networks Over Weakly Connected Digraphs

A conjunctive Boolean network (CBN) is a finite state dynamical system, whose variables take values from a binary set, and the value update rule for each variable is a Boolean function consisting only of logic AND operations. We investigate the asymptotic behavior of CBNs by computing their periodic orbits. When the underlying digraph is strongly connected, the periodic orbits of the associated CBN has been completely understood, one-to-one corresponding to binary necklaces of a certain length given by the loop number of the graph. We characterize in the paper the periodic orbits of CBNs over an arbitrary weakly connected digraphs. We establish, among other things, a new method to investigate their asymptotic behavior. Specifically, we introduce a graphical-approach, termed system reduction, which turns the underlying digraph into a special weakly connected digraph whose strongly connected components are all cycles. We show that the reduced system uniquely determines the asymptotic behavior of the original system. Moreover, we provide a constructive method for computing the periodic orbit of the reduced system, which the system will enter for a given but arbitrary initial condition.

math.DS↗

arXiv2019

A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning

This paper considers a distributed reinforcement learning problem in which a network of multiple agents aim to cooperatively maximize the globally averaged return through communication with only local neighbors. A randomized communication-efficient multi-agent actor-critic algorithm is proposed for possibly unidirectional communication relationships depicted by a directed graph. It is shown that the algorithm can solve the problem for strongly connected graphs by allowing each agent to transmit only two scalar-valued variables at one time.

cs.LG↗

arXiv2018

Distributed Discrete-time Optimization in Multi-agent Networks Using only Sign of Relative State

This paper proposes distributed discrete-time algorithms to cooperatively solve an additive cost optimization problem in multi-agent networks. The striking feature lies in the use of only the sign of relative state information between neighbors, which substantially differentiates our algorithms from others in the existing literature. We first interpret the proposed algorithms in terms of the penalty method in optimization theory and then perform non-asymptotic analysis to study convergence for static network graphs. Compared with the celebrated distributed subgradient algorithms, which however use the exact relative state information, the convergence speed is essentially not affected by the loss of information. We also study how introducing noise into the relative state information and randomly activated graphs affect the performance of our algorithms. Finally, we validate the theoretical results on a class of distributed quantile regression problems.

eess.SY↗

arXiv2018

Distributed Learning of Average Belief Over Networks Using Sequential Observations

This paper addresses the problem of distributed learning of average belief with sequential observations, in which a network of $n>1$ agents aim to reach a consensus on the average value of their beliefs, by exchanging information only with their neighbors. Each agent has sequentially arriving samples of its belief in an online manner. The neighbor relationships among the $n$ agents are described by a graph which is possibly time-varying, whose vertices correspond to agents and whose edges depict neighbor relationships. Two distributed online algorithms are introduced for undirected and directed graphs, which are both shown to converge to the average belief almost surely. Moreover, the sequences generated by both algorithms are shown to reach consensus with an $O(1/t)$ rate with high probability, where $t$ is the number of iterations. For undirected graphs, the corresponding algorithm is modified for the case with quantized communication and limited precision of the division operation. It is shown that the modified algorithm causes all $n$ agents to either reach a quantized consensus or enter a small neighborhood around the average of their beliefs. Numerical simulations are then provided to corroborate the theoretical results.

cs.MA↗

arXiv2018

Distributed Equilibrium-Learning for Power Network Voltage Control With a Locally Connected Communication Network

In current power distribution systems, one of the most challenging operation tasks is to coordinate the network- wide distributed energy resources (DERs) to maintain the stability of voltage magnitude of the system. This voltage control task has been investigated actively under either distributed optimization-based or local feedback control-based characterizations. The former architecture requires a strongly-connected communication network among all DERs for implementing the optimization algorithms, a scenario not yet realistic in most of the existing distribution systems with under-deployed communication infrastructure. The latter one, on the other hand, has been proven to suffer from loss of network-wide op- erational optimality. In this paper, we propose a game-theoretic characterization for semi-local voltage control with only a locally connected communication network. We analyze the existence and uniqueness of the generalized Nash equilibrium (GNE) for this characterization and develop a fully distributed equilibrium-learning algorithm that relies on only neighbor-to-neighbor information exchange. Provable convergence results are provided along with numerical tests which corroborate the robust convergence property of the proposed algorithm.

eess.SY↗

arXiv2018

Dynamic Power Distribution System Management With a Locally Connected Communication Network

Coordinated optimization and control of distribution-level assets can enable a reliable and optimal integration of massive amount of distributed energy resources (DERs) and facilitate distribution system management (DSM). Accordingly, the objective is to coordinate the power injection at the DERs to maintain certain quantities across the network, e.g., voltage magnitude, line flows, or line losses, to be close to a desired profile. By and large, the performance of the DSM algorithms has been challenged by two factors: i) the possibly non strongly connected communication network over DERs that hinders the coordination; ii) the dynamics of the real system caused by the DERs with heterogeneous capabilities, time-varying operating conditions, and real-time measurement mismatches. In this paper, we investigate the modeling and algorithm design and analysis with the consideration of these two factors. In particular, a game-theoretic characterization is first proposed to account for a locally connected communication network over DERs, along with the analysis of the existence and uniqueness of the Nash equilibrium (NE) therein. To achieve the equilibrium in a distributed fashion, a projected-gradient-based asynchronous DSM algorithm is then advocated. The algorithm performance, including the convergence speed and the tracking error, is analytically guaranteed under the dynamic setting. Extensive numerical tests on both synthetic and realistic cases corroborate the analytical results derived.

eess.SY↗

arXiv2018

Generalized Colonel Blotto Game

Competitive resource allocation between adversarial decision makers arises in a wide spectrum of real-world applications such as in communication systems, cyber-physical systems security, as well as financial, political, and electoral competition. As such, developing analytical tools to model and analyze competitive resource allocation is crucial for devising optimal allocation strategies and anticipating the potential outcomes of the competition. To this end, the Colonel Blotto game is one of the most popular game-theoretic frameworks for modeling and analyzing such competitive resource allocation problems. However, in many real-world competitive situations, the Colonel Blotto game does not admit solutions in deterministic strategies and, hence, one must rely on analytically complex mixed-strategies with their associated tractability, applicability, and practicality challenges. In this paper, a generalization of the Colonel Blotto game which enables the derivation of deterministic, practical, and implementable equilibrium strategies is proposed while accounting for the heterogeneity of the battlefields. In addition, the proposed generalized game enables accounting for the consumed resources in each battlefield, a feature that is not considered in the classical Blotto game. For the generalized game, the existence of a Nash equilibrium in pure-strategies is shown. Then, closed-form analytical expressions of the equilibrium strategies, are derived and the outcome of the game is characterized; based on the number of resources of each player as well as the valuation of each battlefield. The generated results provide invaluable insights on the outcome of the competition. For example, the results show that, when both players are fully rational, the more resourceful player can achieve a better total payoff at the Nash equilibrium, a result that is not mimicked in the classical Blotto game.

cs.GT↗

arXiv2018

Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents

We consider the problem of \emph{fully decentralized} multi-agent reinforcement learning (MARL), where the agents are located at the nodes of a time-varying communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. Within this setting, the collective goal of the agents is to maximize the globally averaged return over the network through exchanging information with their neighbors. To this end, we propose two decentralized actor-critic algorithms with function approximation, which are applicable to large-scale MARL problems where both the number of states and the number of agents are massively large. Under the decentralized structure, the actor step is performed individually by each agent with no need to infer the policies of others. For the critic step, we propose a consensus update via communication over the network. Our algorithms are fully incremental and can be implemented in an online fashion. Convergence analyses of the algorithms are provided when the value functions are approximated within the class of linear functions. Extensive simulation results with both linear and nonlinear function approximations are presented to validate the proposed algorithms. Our work appears to be the first study of fully decentralized MARL algorithms for networked agents with function approximation, with provable convergence guarantees.

cs.LG↗

arXiv2018

Reliable Intersection Control in Non-cooperative Environments

We propose a reliable intersection control mechanism for strategic autonomous and connected vehicles (agents) in non-cooperative environments. Each agent has access to his/her earliest possible and desired passing times, and reports a passing time to the intersection manager, who allocates the intersection temporally to the agents in a First-Come-First-Serve basis. However, the agents might have conflicting interests and can take actions strategically. To this end, we analyze the strategic behaviors of the agents and formulate Nash equilibria for all possible scenarios. Furthermore, among all Nash equilibria we identify a socially optimal equilibrium that leads to a fair intersection allocation, and correspondingly we describe a strategy-proof intersection mechanism, which achieves reliable intersection control such that the strategic agents do not have any incentive to misreport their passing times strategically.

cs.AI↗

arXiv2018

Secure Sensor Design Against Undetected Infiltration: Minimum Impact-Minimum Damage

We propose a new defense mechanism against undetected infiltration into controllers in cyber-physical systems. To this end, we cautiously design the outputs of the sensors that monitor the state of the system. Different from the defense mechanisms that seek to detect infiltration, the proposed approach seeks to minimize the damage of possible attacks before they have been detected. Controller of a cyber-physical system could have been infiltrated into by an undetected attacker at any time of the operation. Disregarding such a possibility and disclosing system's state without caution benefits the attacker in his/her malicious objective. Therefore, secure sensor design can improve the security of cyber-physical systems further when incorporated along with other defense mechanisms. We, specifically, consider a controlled Gauss-Markov process, where the controller could have been infiltrated into at any time within the system's operation. In the sense of game-theoretic hierarchical equilibrium, we provide a semi-definite programming based algorithm to compute the optimal linear secure sensor outputs and analyze the performance for various scenarios numerically.

eess.SY↗

arXiv2017

Countries' Survival in Networked International Environments

This paper applies a recently developed power allocation game in Li and Morse (2017) to study the countries' survival problem in networked international environments. In the game, countries strategically allocate their power to support the survival of themselves and their friends and to oppose that of their foes, where by a country's survival is meant when the country's total support equals or exceeds its total threats. This paper establishes conditions that characterize different types of networked international environments in which a country may survive, such as when all the antagonism among countries makes up a complete or bipartite graph.

cs.SI↗

arXiv2017

Strategic Communication Between Prospect Theoretic Agents over a Gaussian Test Channel

In this paper, we model a Stackelberg game in a simple Gaussian test channel where a human transmitter (leader) communicates a source message to a human receiver (follower). We model human decision making using prospect theory models proposed for continuous decision spaces. Assuming that the value function is the squared distortion at both the transmitter and the receiver, we analyze the effects of the weight functions at both the transmitter and the receiver on optimal communication strategies, namely encoding at the transmitter and decoding at the receiver, in the Stackelberg sense. We show that the optimal strategies for the behavioral agents in the Stackelberg sense are identical to those designed for unbiased agents. At the same time, we also show that the prospect-theoretic distortions at both the transmitter and the receiver are both larger than the expected distortion, thus making behavioral agents less contended than unbiased agents. Consequently, the presence of cognitive biases increases the need for transmission power in order to achieve a given distortion at both transmitter and receiver.

cs.GT↗

arXiv2017

Discrete-Time Polar Opinion Dynamics with Susceptibility

This paper considers a discrete-time opinion dynamics model in which each individual's susceptibility to being influenced by others is dependent on her current opinion. We assume that the social network has time-varying topology and that the opinions are scalars on a continuous interval. We first propose a general opinion dynamics model based on the DeGroot model, with a general function to describe the functional dependence of each individual's susceptibility on her own opinion, and show that this general model is analogous to the Friedkin-Johnsen model, which assumes a constant susceptibility for each individual. We then consider two specific functions in which the individual's susceptibility depends on the \emph{polarity} of her opinion, and provide motivating social examples. First, we consider stubborn positives, who have reduced susceptibility if their opinions are at one end of the interval and increased susceptibility if their opinions are at the opposite end. A court jury is used as a motivating example. Second, we consider stubborn neutrals, who have reduced susceptibility when their opinions are in the middle of the spectrum, and our motivating examples are social networks discussing established social norms or institutionalized behavior. For each specific susceptibility model, we establish the initial and graph topology conditions in which consensus is reached, and develop necessary and sufficient conditions on the initial conditions for the final consensus value to be at either extreme of the opinion interval. Simulations are provided to show the effects of the susceptibility function when compared to the DeGroot model.

cs.SI↗

arXiv2017

Stability Structures of Conjunctive Boolean Networks

A Boolean network is a finite dynamical system, whose variables take values from a binary set. The value update rule for each variable is a Boolean function, depending on a selected subset of variables. Boolean networks have been widely used in modeling gene regulatory networks. We focus in this paper on a special class of Boolean networks, termed as conjunctive Boolean networks. A Boolean network is conjunctive if the associated value update rule is comprised of only AND operations. It is known that any trajectory of a finite dynamical system will enter a periodic orbit. We characterize in this paper all periodic orbits of a conjunctive Boolean network whose underlying graph is strongly connected. In particular, we establish a bijection between the set of periodic orbits and the set of binary necklaces of a certain length. We further investigate the stability of a periodic orbit. Specifically, we perturb a state in the periodic orbit by changing the value of a single entry of the state. The trajectory, with the perturbed state being the initial condition, will enter another (possibly the same) periodic orbit in finite time steps. We then provide a complete characterization of all such transitions from one periodic orbit to another. In particular, we construct a digraph, with the vertices being the periodic orbits, and the (directed) edges representing the transitions among the orbits. We call such a digraph the stability structure of the conjunctive Boolean network.

math.DS↗