arXiv subjects

Jalal Arabneydi

Publications and source records attributed to Jalal Arabneydi.

At least 19 recordsLinked to original sources

arXiv2026

Explicit Sequential Equilibria in LQ Deep Structured Games and Weighted Mean-Field Games

We investigate a class of nonzero-sum dynamic stochastic games where players have linear dynamics and quadratic costs, coupled in both dynamics and cost through a linear regression (weighted average) and a quadratic regression (weighted covariance matrix) of states and actions; the linear regression of states is called the deep state. We study collaborative and non-collaborative games under three information structures: perfect sharing, deep state sharing, and no sharing, for three weight types: positive, homogeneous, and asymptotically vanishing. For perfect and deep state sharing structures, we propose a new technique using gauge transformation to solve the players' best-response equations and identify sufficient conditions for a unique subgame perfect Nash equilibrium. The equilibrium is linear in the local state and deep state, with gains obtained from a novel non-standard Riccati equation whose dimension is independent of the number of players, making the solution scalable. Under no sharing with an asymptotically large number of players, we propose an asymptotic population-size-dependent equilibrium and an asymptotic population-size-independent equilibrium (the sequential weighted mean-field equilibrium), and establish their convergence to the infinite-population limits. The main results are extended to infinite-horizon costs and generalized to multiple linear regressions and heterogeneous sub-populations. A numerical example illustrates the difference between the two approximate equilibria. To our knowledge, this is the first paper to propose a unified framework yielding the exact closed-form solution for an arbitrary number of players.

math.OC↗

arXiv2025

A Systematic Approach to Design Real-World Human-in-the-Loop Deep Reinforcement Learning: Salient Features, Challenges and Trade-offs

With the growing popularity of deep reinforcement learning (DRL), human-in-the-loop (HITL) approach has the potential to revolutionize the way we approach decision-making problems and create new opportunities for human-AI collaboration. In this article, we introduce a novel multi-layered hierarchical HITL DRL algorithm that comprises three types of learning: self learning, imitation learning and transfer learning. In addition, we consider three forms of human inputs: reward, action and demonstration. Furthermore, we discuss main challenges, trade-offs and advantages of HITL in solving complex problems and how human information can be integrated in the AI solution systematically. To verify our technical results, we present a real-world unmanned aerial vehicles (UAV) problem wherein a number of enemy drones attack a restricted area. The objective is to design a scalable HITL DRL algorithm for ally drones to neutralize the enemy drones before they reach the area. To this end, we first implement our solution using an award-winning open-source HITL software called Cogment. We then demonstrate several interesting results such as (a) HITL leads to faster training and higher performance, (b) advice acts as a guiding direction for gradient methods and lowers variance, and (c) the amount of advice should neither be too large nor too small to avoid over-training and under-training. Finally, we illustrate the role of human-AI cooperation in solving two real-world complex scenarios, i.e., overloaded and decoy attacks.

cs.AI↗

arXiv2023

Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning

Recent advances in reinforcement learning (RL) and Human-in-the-Loop (HitL) learning have made human-AI collaboration easier for humans to team with AI agents. Leveraging human expertise and experience with AI in intelligent systems can be efficient and beneficial. Still, it is unclear to what extent human-AI collaboration will be successful, and how such teaming performs compared to humans or AI agents only. In this work, we show that learning from humans is effective and that human-AI collaboration outperforms human-controlled and fully autonomous AI agents in a complex simulation environment. In addition, we have developed a new simulator for critical infrastructure protection, focusing on a scenario where AI-powered drones and human teams collaborate to defend an airport against enemy drone attacks. We develop a user interface to allow humans to assist AI agents effectively. We demonstrated that agents learn faster while learning from policy correction compared to learning from humans or agents. Furthermore, human-AI collaboration requires lower mental and temporal demands, reduces human effort, and yields higher performance than if humans directly controlled all agents. In conclusion, we show that humans can provide helpful advice to the RL agents, allowing them to improve learning in a multi-agent setting.

cs.AI↗

arXiv2021

Deep Structured Teams in Arbitrary-Size Linear Networks: Decentralized Estimation, Optimal Control and Separation Principle

In this article, we introduce decentralized Kalman filters for linear quadratic deep structured teams. The agents in deep structured teams are coupled in dynamics, costs and measurements through a set of linear regressions of the states and actions (also called deep states and deep actions). The information structure is decentralized, where every agent observes a noisy measurement of its local state and the global deep state. Since the number of agents is often very large in deep structured teams, any naive approach to finding an optimal Kalman filter suffers from the curse of dimensionality. Moreover, due to the decentralized nature of information structure, the resultant optimization problem is non-convex, in general, where non-linear strategies can outperform linear ones. However, we prove that the optimal strategy is linear in the local state estimate as well as the deep state estimate and can be efficiently computed by two scale-free Riccati equations and Kalman filters. We propose a bi-level orthogonal approach across both space and time levels based on a gauge transformation technique to achieve the above result. We also establish a separation principle between optimal control and optimal estimation. Furthermore, we show that as the number of agents goes to infinity, the Kalman gain associated with the deep state estimate converges to zero at a rate inversely proportional to the number of agents. This leads to a fully decentralized approximate strategy where every agent predicts the deep state by its conditional and unconditional expected value, also known as the certainty equivalence approximation and (weighted) mean-field approximation, respectively.

cs.MA↗

arXiv2021

Receding Horizon Control in Deep Structured Teams: A Provably Tractable Large-Scale Approach with Application to Swarm Robotics

In this paper, a deep structured tracking problem is introduced for a large number of decision-makers. The problem is formulated as a linear quadratic deep structured team, where the decision-makers wish to track a global target cooperatively while considering their local targets. For the unconstrained setup, the gauge transformation technique is used to decompose the resultant optimization problem in order to obtain a low-dimensional optimal control strategy in terms of the local and global Riccati equations. For the constrained case, however, the feasible set is not necessarily decomposable by the gauge transformation. To overcome this hurdle, we propose a family of local and global receding horizon control problems, where a carefully constructed linear combination of their solutions provides a feasible solution for the original constrained problem. The salient property of the above solutions is that they are tractable with respect to the number of decision-makers and can be implemented in a distributed manner. In addition, the main results are generalized to cases with multiple sub-populations and multiple features, including leader-follower setup, cohesive cost function and soft structural constraint. Furthermore, a class of cyber-physical attacks is proposed in terms of perturbed influence factors. A numerical example is presented to demonstrate the efficacy of the results.

eess.SY↗

arXiv2021

Data Collection versus Data Estimation: A Fundamental Trade-off in Dynamic Networks

An important question that often arises in the operation of networked systems is whether to collect the real-time data or to estimate them based on the previously collected data. Various factors should be taken into account such as how informative the data are at each time instant for state estimation, how costly and credible the collected data are, and how rapidly the data vary with time. The above question can be formulated as a dynamic decision making problem with imperfect information structure, where a decision maker wishes to find an efficient way to switch between data collection and data estimation while the quality of the estimation depends on the previously collected data (i.e., duality effect). In this paper, the evolution of the state of each node is modeled as an exchangeable Markov process for discrete features and equivariant linear system for continuous features, where the data of interest are defined in the former case as the empirical distribution of the states, and in the latter case as the weighted average of the states. When the data are collected, they may or may not be credible, according to a Bernoulli distribution. Based on a novel planning space, a Bellman equation is proposed to identify a near-optimal strategy. A reinforcement learning algorithm is developed for the case when the model is not known exactly, and its convergence to the near-optimal solution is shown subsequently. In addition, a certainty threshold is introduced that determines when data estimation is more desirable than data collection, as the number of nodes increases. For the special case of linear dynamics, a separation principle is constructed wherein the optimal estimate is computed by a Kalman-like filter, irrespective of the probability distribution of random variables...

math.OC↗

arXiv2021

Reinforcement Learning in Deep Structured Teams: Initial Results with Finite and Infinite Valued Features

In this paper, we consider Markov chain and linear quadratic models for deep structured teams with discounted and time-average cost functions under two non-classical information structures, namely, deep state sharing and no sharing. In deep structured teams, agents are coupled in dynamics and cost functions through deep state, where deep state refers to a set of orthogonal linear regressions of the states. In this article, we consider a homogeneous linear regression for Markov chain models (i.e., empirical distribution of states) and a few orthonormal linear regressions for linear quadratic models (i.e., weighted average of states). Some planning algorithms are developed for the case when the model is known, and some reinforcement learning algorithms are proposed for the case when the model is not known completely. The convergence of two model-free (reinforcement learning) algorithms, one for Markov chain models and one for linear quadratic models, is established. The results are then applied to a smart grid.

cs.MA↗

arXiv2020

Reinforcement Learning in Linear Quadratic Deep Structured Teams: Global Convergence of Policy Gradient Methods

In this paper, we study the global convergence of model-based and model-free policy gradient descent and natural policy gradient descent algorithms for linear quadratic deep structured teams. In such systems, agents are partitioned into a few sub-populations wherein the agents in each sub-population are coupled in the dynamics and cost function through a set of linear regressions of the states and actions of all agents. Every agent observes its local state and the linear regressions of states, called deep states. For a sufficiently small risk factor and/or sufficiently large population, we prove that model-based policy gradient methods globally converge to the optimal solution. Given an arbitrary number of agents, we develop model-free policy gradient and natural policy gradient algorithms for the special case of risk-neutral cost function. The proposed algorithms are scalable with respect to the number of agents due to the fact that the dimension of their policy space is independent of the number of agents in each sub-population. Simulations are provided to verify the theoretical results.

cs.MA↗

arXiv2020

Reinforcement Learning in Nonzero-sum Linear Quadratic Deep Structured Games: Global Convergence of Policy Optimization

We study model-based and model-free policy optimization in a class of nonzero-sum stochastic dynamic games called linear quadratic (LQ) deep structured games. In such games, players interact with each other through a set of weighted averages (linear regressions) of the states and actions. In this paper, we focus our attention to homogeneous weights; however, for the special case of infinite population, the obtained results extend to asymptotically vanishing weights wherein the players learn the sequential weighted mean-field equilibrium. Despite the non-convexity of the optimization in policy space and the fact that policy optimization does not generally converge in game setting, we prove that the proposed model-based and model-free policy gradient descent and natural policy gradient descent algorithms globally converge to the sub-game perfect Nash equilibrium. To the best of our knowledge, this is the first result that provides a global convergence proof of policy optimization in a nonzero-sum LQ game. One of the salient features of the proposed algorithms is that their parameter space is independent of the number of players, and when the dimension of state space is significantly larger than that of the action space, they provide a more efficient way of computation compared to those algorithms that plan and learn in the action space. Finally, some simulations are provided to numerically verify the obtained theoretical results.

cs.GT↗

arXiv2020

Team Optimal Control of Coupled Major-Minor Subsystems with Mean-Field Sharing

In this paper, we investigate team optimal control of coupled major-minor subsystems with mean-field sharing. In such a model, there is one major subsystem that directly influences the dynamics of $n$ homogeneous minor subsystems; however, the minor subsystems influence the dynamics of the major subsystem and each other only through their mean behaviour (indirectly). In this model, the major and the minor subsystems are arbitrarily coupled in the cost and have mean-field sharing information structure. We propose a two-step approach. In the first step, we describe a mean-field sharing model with multiple types of minor subsystems as a generalization of the mean-field sharing model of Arabneydi and Mahajan, CDC 2014. In the second step, we use the results obtained in the first step to construct a dynamic programming decomposition to identify optimal strategies for the major and the minor subsystems. We present an example with numerical results to illustrate the approach.

math.OC↗

arXiv2020

MinMax Mean-Field Team Approach for a Leader-Follower Network: A Saddle-Point Strategy

This paper investigates a soft-constrained MinMax control problem of a leader-follower network. The network consists of one leader and an arbitrary number of followers that wish to reach consensus with minimum energy consumption in the presence of external disturbances. The leader and followers are coupled in the dynamics and cost function. Two non-classical information structures are considered: mean-field sharing and intermittent mean-field sharing, where the mean-field refers to the aggregate state of the followers. In mean-field sharing, every follower observes its local state, the state of the leader and the mean field while in the intermittent mean-field sharing, the mean-field is only observed at some (possibly no) time instants. A social welfare cost function is defined, and it is shown that a unique saddle-point strategy exists which minimizes the worst-case value of the cost function under mean-field sharing information structure. The solution is obtained by two scalable Riccati equations, which depend on a prescribed attenuation parameter, serving as a robustness factor. For the intermittent mean-field sharing information structure, an approximate saddle-point strategy is proposed, and its converges to the saddle-point is analyzed. Two numerical examples are provided to demonstrate the efficacy of the obtained results.

math.OC↗

arXiv2020

Near-Optimal Design for Fault-Tolerant Systems with Homogeneous Components under Incomplete Information

In this paper, we study a fault-tolerant control for systems consisting of multiple homogeneous components such as parallel processing machines. This type of system is often more robust to uncertainty compared to those with a single component. The state of each component is either in the operating mode or faulty. At any time instant, each component may independently become faulty according to a Bernoulli probability distribution. If a component is faulty, it remains so until it is fixed. The objective is to design a fault-tolerant system by sequentially choosing one of the following three options: (a) do nothing at zero cost; b) detect the number of faulty components at the cost of inspection, and c) fix the system at the cost of repairing faulty components. A Bellman equation is developed to identify a near-optimal solution for the problem. The efficacy of the proposed solution is verified by numerical simulations.

math.OC↗

arXiv2020

Optimal Dynamic Pricing for Binary Demands in Smart Grids: A Fair and Privacy-Preserving Strategy

Motivated by demand-side management in smart grids, a decentralized controlled Markov chain formulation is proposed to model a homogeneous population of users with binary demands (i.e., off or on). The binary demands often arise in scheduling applications such as plug-in hybrid vehicles. Normally, an independent service operator (ISO) has a finite number of options when it comes to providing the users with electricity. The options represent various incentive means, generation resources, and price profiles. The objective of the ISO is to find optimal options in order to keep the distribution of demands close to a desired level (which varies with time, in general) by imposing the minimum price on the users. A Bellman equation is developed here to identify the globally team-optimal strategy. The proposed strategy is fair for all users and also protects the privacy of users. Moreover, its computational complexity increases linearly (rather than exponentially) with the number of users. A numerical example with 100 users is presented for peak-load management.

math.OC↗

arXiv2020

A Certainty Equivalence Result in Team-Optimal Control of Mean-Field Coupled Markov Chains

This paper studies a large number of homogeneous Markov decision processes where the transition probabilities and costs are coupled in the empirical distribution of states (also called mean-field). The state of each process is not known to others, which means that the information structure is fully decentralized. The objective is to minimize the average cost, defined as the empirical mean of individual costs, for which a sub-optimal solution is proposed. This solution does not depend on the number of processes, yet it converges to the optimal solution of the so-called mean-field sharing as the number of processes tends to infinity. Under some mild conditions, it is shown that the convergence rate of the proposed decentralized solution is proportional to the square root of the inverse of the number of processes. Finding this sub-optimal solution involves a non-smooth non-convex optimization problem over an uncountable set, in general. To overcome this drawback, a combinatorial optimization problem is introduced that achieves the same rate of convergence.

math.OC↗

arXiv2020

Team Optimal Control of Coupled Subsystems with Mean-Field Sharing

We investigate team optimal control of stochastic subsystems that are weakly coupled in dynamics (through the mean-field of the system) and are arbitrary coupled in the cost. The controller of each subsystem observes its local state and the mean-field of the state of all subsystems. The system has a non-classical information structure. Exploiting the symmetry of the problem, we identify an information state and use that to obtain a dynamic programming decomposition. This dynamic program determines a globally optimal strategy for all controllers. Our solution approach works for arbitrary number of controllers and generalizes to the setup when the mean-field is observed with noise. The size of the information state is time-invariant; thus, the results generalize to the infinite-horizon control setups as well. In addition, when the mean-field is observed without noise, the size of the corresponding information state increases polynomially (rather than exponentially) with the number of controllers which allows us to solve problems with moderate number of controllers. We illustrate our approach by an example motivated by smart grids that consists of $100$ coupled subsystems.

math.OC↗

arXiv2020

Reinforcement Learning in Decentralized Stochastic Control Systems with Partial History Sharing

In this paper, we are interested in systems with multiple agents that wish to collaborate in order to accomplish a common task while a) agents have different information (decentralized information) and b) agents do not know the model of the system completely i.e., they may know the model partially or may not know it at all. The agents must learn the optimal strategies by interacting with their environment i.e., by decentralized Reinforcement Learning (RL). The presence of multiple agents with different information makes decentralized reinforcement learning conceptually more difficult than centralized reinforcement learning. In this paper, we develop a decentralized reinforcement learning algorithm that learns $ε$-team-optimal solution for partial history sharing information structure, which encompasses a large class of decentralized control systems including delayed sharing, control sharing, mean field sharing, etc. Our approach consists of two main steps. In the first step, we convert the decentralized control system to an equivalent centralized POMDP (Partially Observable Markov Decision Process) using an existing approach called common information approach. However, the resultant POMDP requires the complete knowledge of system model. To circumvent this requirement, in the second step, we introduce a new concept called "Incrementally Expanding Representation" using which we construct a finite-state RL algorithm whose approximation error converges to zero exponentially fast. We illustrate the proposed approach and verify it numerically by obtaining a decentralized Q-learning algorithm for two-user Multi Access Broadcast Channel (MABC) which is a benchmark example for decentralized control systems.

math.OC↗

arXiv2020

Team-Optimal Solution of Finite Number of Mean-Field Coupled LQG Subsystems

A decentralized control system with linear dynamics, quadratic cost, and Gaussian disturbances is considered. The system consists of a finite number of subsystems whose dynamics and per-step cost function are coupled through their mean-field (empirical average). The system has mean-field sharing information structure, i.e., each controller observes the state of its local subsystem (either perfectly or with noise) and the mean-field. It is shown that the optimal control law is unique, linear, and identical across all subsystems. Moreover, the optimal gains are computed by solving two decoupled Riccati equations in the full observation model and by solving an additional filter Riccati equation in the noisy observation model. These Riccati equations do not depend on the number of subsystems. It is also shown that the optimal decentralized performance is the same as the optimal centralized performance. An example, motivated by smart grids, is presented to illustrate the result.

math.OC↗

arXiv2020

A Mean-Field Team Approach to Minimize the Spread of Infection in a Network

In this paper, a stochastic dynamic control strategy is presented to prevent the spread of an infection over a homogeneous network. The infectious process is persistent, i.e., it continues to contaminate the network once it is established. It is assumed that there is a finite set of network management options available such as degrees of nodes and promotional plans to minimize the number of infected nodes while taking the implementation cost into account. The network is modeled by an exchangeable controlled Markov chain, whose transition probability matrices depend on three parameters: the selected network management option, the state of the infectious process, and the empirical distribution of infected nodes (with not necessarily a linear dependence). Borrowing some techniques from mean-field team theory the optimal strategy is obtained for any finite number of nodes using dynamic programming decomposition and the convolution of some binomial probability mass functions. For infinite-population networks, the optimal solution is described by a Bellman equation. It is shown that the infinite-population strategy is a meaningful sub-optimal solution for finite-population networks if a certain condition holds. The theoretical results are verified by an example of rumor control in social networks.

eess.SY↗