arXiv subjects

Amir G. Aghdam

Publications and source records attributed to Amir G. Aghdam.

35 records · Page 2Linked to original sources

arXiv2021

Receding Horizon Control in Deep Structured Teams: A Provably Tractable Large-Scale Approach with Application to Swarm Robotics

In this paper, a deep structured tracking problem is introduced for a large number of decision-makers. The problem is formulated as a linear quadratic deep structured team, where the decision-makers wish to track a global target cooperatively while considering their local targets. For the unconstrained setup, the gauge transformation technique is used to decompose the resultant optimization problem in order to obtain a low-dimensional optimal control strategy in terms of the local and global Riccati equations. For the constrained case, however, the feasible set is not necessarily decomposable by the gauge transformation. To overcome this hurdle, we propose a family of local and global receding horizon control problems, where a carefully constructed linear combination of their solutions provides a feasible solution for the original constrained problem. The salient property of the above solutions is that they are tractable with respect to the number of decision-makers and can be implemented in a distributed manner. In addition, the main results are generalized to cases with multiple sub-populations and multiple features, including leader-follower setup, cohesive cost function and soft structural constraint. Furthermore, a class of cyber-physical attacks is proposed in terms of perturbed influence factors. A numerical example is presented to demonstrate the efficacy of the results.

eess.SY↗

arXiv2021

Data Collection versus Data Estimation: A Fundamental Trade-off in Dynamic Networks

An important question that often arises in the operation of networked systems is whether to collect the real-time data or to estimate them based on the previously collected data. Various factors should be taken into account such as how informative the data are at each time instant for state estimation, how costly and credible the collected data are, and how rapidly the data vary with time. The above question can be formulated as a dynamic decision making problem with imperfect information structure, where a decision maker wishes to find an efficient way to switch between data collection and data estimation while the quality of the estimation depends on the previously collected data (i.e., duality effect). In this paper, the evolution of the state of each node is modeled as an exchangeable Markov process for discrete features and equivariant linear system for continuous features, where the data of interest are defined in the former case as the empirical distribution of the states, and in the latter case as the weighted average of the states. When the data are collected, they may or may not be credible, according to a Bernoulli distribution. Based on a novel planning space, a Bellman equation is proposed to identify a near-optimal strategy. A reinforcement learning algorithm is developed for the case when the model is not known exactly, and its convergence to the near-optimal solution is shown subsequently. In addition, a certainty threshold is introduced that determines when data estimation is more desirable than data collection, as the number of nodes increases. For the special case of linear dynamics, a separation principle is constructed wherein the optimal estimate is computed by a Kalman-like filter, irrespective of the probability distribution of random variables...

math.OC↗

arXiv2021

Reinforcement Learning in Deep Structured Teams: Initial Results with Finite and Infinite Valued Features

In this paper, we consider Markov chain and linear quadratic models for deep structured teams with discounted and time-average cost functions under two non-classical information structures, namely, deep state sharing and no sharing. In deep structured teams, agents are coupled in dynamics and cost functions through deep state, where deep state refers to a set of orthogonal linear regressions of the states. In this article, we consider a homogeneous linear regression for Markov chain models (i.e., empirical distribution of states) and a few orthonormal linear regressions for linear quadratic models (i.e., weighted average of states). Some planning algorithms are developed for the case when the model is known, and some reinforcement learning algorithms are proposed for the case when the model is not known completely. The convergence of two model-free (reinforcement learning) algorithms, one for Markov chain models and one for linear quadratic models, is established. The results are then applied to a smart grid.

cs.MA↗

arXiv2020

Reinforcement Learning in Linear Quadratic Deep Structured Teams: Global Convergence of Policy Gradient Methods

In this paper, we study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms for linear quadratic deep structured teams. In such systems, agents are partitioned into a few sub-populations wherein the agents in each sub-population are coupled in the dynamics and cost function through a set of linear regressions of the states and actions of all agents. Every agent observes its local state and the linear regressions of states, called deep states. For a sufficiently small risk factor and/or sufficiently large population, we prove that model-based policy gradient methods globally converge to the optimal solution. Given an arbitrary number of agents, we develop model-free policy gradient and natural policy gradient algorithms for the special case of risk-neutral cost function. The proposed algorithms are scalable with respect to the number of agents due to the fact that the dimension of their policy space is independent of the number of agents in each sub-population. Simulations are provided to verify the theoretical results.

cs.MA↗

arXiv2020

Reinforcement Learning in Nonzero-sum Linear Quadratic Deep Structured Games: Global Convergence of Policy Optimization

We study model-based and model-free policy optimization in a class of nonzero-sum stochastic dynamic games called linear quadratic (LQ) deep structured games. In such games, players interact with each other through a set of weighted averages (linear regressions) of the states and actions. In this paper, we focus our attention to homogeneous weights; however, for the special case of infinite population, the obtained results extend to asymptotically vanishing weights wherein the players learn the sequential weighted mean-field equilibrium. Despite the non-convexity of the optimization in policy space and the fact that policy optimization does not generally converge in game setting, we prove that the proposed model-based and model-free policy gradient descent and natural policy gradient descent algorithms globally converge to the sub-game perfect Nash equilibrium. To the best of our knowledge, this is the first result that provides a global convergence proof of policy optimization in a nonzero-sum LQ game. One of the salient features of the proposed algorithms is that their parameter space is independent of the number of players, and when the dimension of state space is significantly larger than that of the action space, they provide a more efficient way of computation compared to those algorithms that plan and learn in the action space. Finally, some simulations are provided to numerically verify the obtained theoretical results.

cs.GT↗

arXiv2020

MinMax Mean-Field Team Approach for a Leader-Follower Network: A Saddle-Point Strategy

This paper investigates a soft-constrained MinMax control problem of a leader-follower network. The network consists of one leader and an arbitrary number of followers that wish to reach consensus with minimum energy consumption in the presence of external disturbances. The leader and followers are coupled in the dynamics and cost function. Two non-classical information structures are considered: mean-field sharing and intermittent mean-field sharing, where the mean-field refers to the aggregate state of the followers. In mean-field sharing, every follower observes its local state, the state of the leader and the mean field while in the intermittent mean-field sharing, the mean-field is only observed at some (possibly no) time instants. A social welfare cost function is defined, and it is shown that a unique saddle-point strategy exists which minimizes the worst-case value of the cost function under mean-field sharing information structure. The solution is obtained by two scalable Riccati equations, which depend on a prescribed attenuation parameter, serving as a robustness factor. For the intermittent mean-field sharing information structure, an approximate saddle-point strategy is proposed, and its converges to the saddle-point is analyzed. Two numerical examples are provided to demonstrate the efficacy of the obtained results.

math.OC↗

arXiv2020

Near-Optimal Design for Fault-Tolerant Systems with Homogeneous Components under Incomplete Information

In this paper, we study a fault-tolerant control for systems consisting of multiple homogeneous components such as parallel processing machines. This type of system is often more robust to uncertainty compared to those with a single component. The state of each component is either in the operating mode or faulty. At any time instant, each component may independently become faulty according to a Bernoulli probability distribution. If a component is faulty, it remains so until it is fixed. The objective is to design a fault-tolerant system by sequentially choosing one of the following three options: (a) do nothing at zero cost; b) detect the number of faulty components at the cost of inspection, and c) fix the system at the cost of repairing faulty components. A Bellman equation is developed to identify a near-optimal solution for the problem. The efficacy of the proposed solution is verified by numerical simulations.

math.OC↗

arXiv2020

Optimal Dynamic Pricing for Binary Demands in Smart Grids: A Fair and Privacy-Preserving Strategy

Motivated by demand-side management in smart grids, a decentralized controlled Markov chain formulation is proposed to model a homogeneous population of users with binary demands (i.e., off or on). The binary demands often arise in scheduling applications such as plug-in hybrid vehicles. Normally, an independent service operator (ISO) has a finite number of options when it comes to providing the users with electricity. The options represent various incentive means, generation resources, and price profiles. The objective of the ISO is to find optimal options in order to keep the distribution of demands close to a desired level (which varies with time, in general) by imposing the minimum price on the users. A Bellman equation is developed here to identify the globally team-optimal strategy. The proposed strategy is fair for all users and also protects the privacy of users. Moreover, its computational complexity increases linearly (rather than exponentially) with the number of users. A numerical example with 100 users is presented for peak-load management.

math.OC↗

arXiv2020

A Certainty Equivalence Result in Team-Optimal Control of Mean-Field Coupled Markov Chains

This paper studies a large number of homogeneous Markov decision processes where the transition probabilities and costs are coupled in the empirical distribution of states (also called mean-field). The state of each process is not known to others, which means that the information structure is fully decentralized. The objective is to minimize the average cost, defined as the empirical mean of individual costs, for which a sub-optimal solution is proposed. This solution does not depend on the number of processes, yet it converges to the optimal solution of the so-called mean-field sharing as the number of processes tends to infinity. Under some mild conditions, it is shown that the convergence rate of the proposed decentralized solution is proportional to the square root of the inverse of the number of processes. Finding this sub-optimal solution involves a non-smooth non-convex optimization problem over an uncountable set, in general. To overcome this drawback, a combinatorial optimization problem is introduced that achieves the same rate of convergence.

math.OC↗

arXiv2020

A Mean-Field Team Approach to Minimize the Spread of Infection in a Network

In this paper, a stochastic dynamic control strategy is presented to prevent the spread of an infection over a homogeneous network. The infectious process is persistent, i.e., it continues to contaminate the network once it is established. It is assumed that there is a finite set of network management options available such as degrees of nodes and promotional plans to minimize the number of infected nodes while taking the implementation cost into account. The network is modeled by an exchangeable controlled Markov chain, whose transition probability matrices depend on three parameters: the selected network management option, the state of the infectious process, and the empirical distribution of infected nodes (with not necessarily a linear dependence). Borrowing some techniques from mean-field team theory the optimal strategy is obtained for any finite number of nodes using dynamic programming decomposition and the convolution of some binomial probability mass functions. For infinite-population networks, the optimal solution is described by a Bellman equation. It is shown that the infinite-population strategy is a meaningful sub-optimal solution for finite-population networks if a certain condition holds. The theoretical results are verified by an example of rumor control in social networks.

eess.SY↗

arXiv2020

Optimal Distributed Control for Leader-Follower Networks: A Scalable Design

The focus of this paper is directed towards optimal control of multi-agent systems consisting of one leader and a number of followers in the presence of noise. The dynamics of every agent is assumed to be linear, and the performance index is a quadratic function of the states and actions of the leader and followers. The leader and followers are coupled in both dynamics and cost. The state of the leader and the average of the states of all followers (called mean-field) are common information and known to all agents; however, the local state of the followers are private information and unknown to other agents. It is shown that the optimal distributed control strategy is linear time-varying, and its computational complexity is independent of the number of followers. This strategy can be computed in a distributed manner, where the leader needs to solve one Riccati equation to determine its optimal strategy while each follower needs to solve two Riccati equations to obtain its optimal strategy. This result is subsequently extended to the case of the infinite horizon discounted and undiscounted cost functions, where the optimal distributed strategy is shown to be stationary. A numerical example with $100$ followers is provided to demonstrate the efficacy of the results.

math.OC↗

arXiv2020

Near-Optimal Control Strategy in Leader-Follower Networks: A Case Study for Linear Quadratic Mean-Field Teams

In this paper, a decentralized stochastic control system consisting of one leader and many homogeneous followers is studied. The leader and followers are coupled in both dynamics and cost, where the dynamics are linear and the cost function is quadratic in the states and actions of the leader and followers. The objective of the leader and followers is to reach consensus while minimizing their communication and energy costs. The leader knows its local state and each follower knows its local state and the state of the leader. The number of required links to implement this decentralized information structure is equal to the number of followers, which is the minimum number of links for a communication graph to be connected. In the special case of leaderless, no link is required among followers, i.e., the communication graph is not even connected. We propose a near-optimal control strategy that converges to the optimal solution as the number of followers increases. One of the salient features of the proposed solution is that it provides a design scheme, where the convergence rate \edit{as well as} the collective behavior of the followers can be designed by choosing appropriate cost functions. In addition, the computational complexity of the proposed solution does not depend on the number of followers. Furthermore, the proposed strategy can be computed in a distributed manner, where the leader solves one Riccati equation and each follower solves two Riccati equations to calculate their strategies. Two numerical examples are provided to demonstrate the effectiveness of the results in the control of multi-agent systems.

math.OC↗

arXiv2020

Deep Structured Teams with Linear Quadratic Model: Partial Equivariance and Gauge Transformation

Motivated by the recent developments in artificial intelligence, we introduce linear quadratic deep structured teams in this paper. Two notions of equivariant and partially equivariant systems are defined, and it is shown that such systems can be partitioned into a few sub-populations of decision makers, where every decision maker in each sub-population is coupled in both dynamics and cost function through a set of linear regressions of the states and actions of all decision makers. Two non-classical information structures are considered: deep-state sharing and partial deep-state sharing, where deep state refers to the linear regression of the states of the decision makers in each sub-population. For a risk-sensitive cost function with deep-state sharing structure, a closed-form low-complexity representation of the globally optimal strategy is obtained, whose computational complexity is independent of the number of decision makers in each sub-population. In addition, it is shown that the risk-sensitive solution converges to the risk-neutral one as the number of decision makers increases to infinity. Moreover, two sub-optimal sequential strategies under partial deep-state sharing information structure are proposed by introducing two Kalman-like filters, one based on the finite-population model and the other one based on the infinite-population model. It is proved that the prices of information associated with the above sub-optimal solutions converge to zero as the number of decision makers goes to infinity. Furthermore, a class of feed-forward deep neural networks with multiple layers of weighted sums and products is introduced wherein the optimal weights and biases are explicitly obtained. A supply-chain management example is presented to demonstrate the efficacy of the obtained results.

math.OC↗

arXiv2020

Deep Teams: Decentralized Decision Making with Finite and Infinite Number of Agents

Inspired by the concepts of deep learning in artificial intelligence and fairness in behavioural economics, we introduce deep teams in this paper. In such systems, agents are partitioned into a few sub-populations so that the dynamics and cost of agents in each sub-population is invariant to the indexing of agents. The goal of agents is to minimize a common cost function in such a manner that the agents in each sub-population are not discriminated or privileged by the way they are indexed. Two non-classical information structures are studied. In the first one, each agent observes its local state as well as the empirical distribution of the states of agents in each sub-population, called deep state, whereas in the second one, the deep states of a subset (possibly all) of sub-populations are not observed. Novel dynamic programs are developed to identify globally optimal and sub-optimal solutions for the first and second information structures, respectively. The computational complexity of finding the optimal solution in both space and time is polynomial (rather than exponential) with respect to the number of agents in each sub-population and is linear (rather than exponential) with respect to the control horizon. This complexity is further reduced in time by introducing a forward equation, that we call deep Chapman-Kolmogorov equation, described by multiple convolutional layers of Binomial probability distributions. Two different prices are defined for computation and communication, and it is shown that under mild assumptions they converge to zero as the quantization level and the number of agents tend to infinity. In addition, the main results are extended to the infinite-horizon discounted model and arbitrarily asymmetric cost function. Finally, a service-management example with 200 users is presented.

math.OC↗

arXiv2019

Deep Nash and Sequential Mean-Field Equilibria in Cooperative and Non-cooperative Games with Imperfect Information Structures

A class of nonzero-sum stochastic dynamic games with imperfect information structure is investigated. The game involves an arbitrary number of players, modeled as homogeneous Markov decision processes, aiming to find a sequential Nash equilibrium. The players are coupled in both dynamics and cost functions through the empirical distribution of states and actions of players. Two non-classical information structures are considered: deep state sharing and no-sharing, where deep state refers to the empirical distribution of the states of players. In the former, each player observes its local state as well as the deep state while in the latter each player observes only its local state. For both finite- and infinite-horizon cost functions, a sequential equilibrium, called deep Nash equilibrium, is identified, where the dynamics of deep state resembles a convolutional neural network. In addition, an approximate sequential equilibrium, called sequential mean-field equilibrium, under no-sharing information structure is proposed, whose performance converges to that of the deep Nash equilibrium despite the fact that the strategy is not necessarily continuous with respect to the deep state. The proposed strategies are robust to trembling-hand imperfection in both microscopic and macroscopic levels. Finally, the extension to multiple sub-populations and arbitrarily-coupled (asymmetric) cost functions are demonstrated.

math.OC↗

arXiv2016

Digraphs with Distinguishable Dynamics under the Multi-Agent Agreement Protocol

In this work, the ability to distinguish digraphs from the output response of some observing agents in a multi-agent network under the agreement protocol has been studied. Given a fixed observation point, it is desired to find sufficient graphical conditions under which the failure of a set of edges in the network information flow digraph is distinguishable from another set. When the latter is empty, this corresponds to the detectability of the former link set given the response of the observing agent. In developing the results, a powerful extension of the all-minors matrix tree theorem in algebraic graph theory is proved which relates the minors of the transformed Laplacian of a directed graph to the number and length of the shortest paths between its vertices. The results reveal an intricate relationship between the ability to distinguish the responses of a healthy and a faulty multi-agent network and the inter-nodal paths in their information flow digraphs. The results have direct implications for the operation and design of multi-agent systems subject to multiple link losses. Simulations and examples are presented to illustrate the analytic findings.

eess.SY↗

arXiv2016

Structural Controllability of Multi-Agent Networks: Robustness against Simultaneous Failures

In this paper, structural controllability of a leader-follower multi-agent system with multiple leaders is studied from a graph-theoretic point of view. The problem of preservation of structural controllability under simultaneous failures in both the communication links and the agents is investigated. The effects of the loss of agents and communication links on the controllability of an information flow graph are previously studied. In this work, the corresponding results are exploited to introduce some useful indices and importance measures that help characterize and quantify the role of individual links and agents in the controllability of the overall network. Existing results are then extended by considering the effects of losses in both links and agents at the same time. To this end, the concepts of joint (r,s)-controllability and joint t-controllability are introduced as quantitative measures of reliability for a multi-agent system, and their important properties are investigated. Lastly, the class of jointly critical digraphs is introduced and it is stated that if a digraph is jointly critical, then joint t-controllability is a necessary and sufficient condition for remaining controllable following the failure of any set of links and agents, with cardinality less than t. Various examples are exploited throughout the paper to elaborate on the analytical findings.

eess.SY↗