arXiv subjects

Naci Saldi

Publications and source records attributed to Naci Saldi.

At least 37 records · Page 2Linked to original sources

arXiv2023

Linear Mean-Field Games with Discounted Cost

In this paper, we introduce discrete-time linear mean-field games subject to an infinite-horizon discounted-cost optimality criterion. The state space of a generic agent is a compact Borel space. At every time, each agent is randomly coupled with another agent via their dynamics and one-stage cost function, where this randomization is generated via the empirical distribution of their states (i.e., the mean-field term). Therefore, the transition probability and the one-stage cost function of each agent depend linearly on the mean-field term, which is the key distinction between classical mean-field games and linear mean-field games. Under mild assumptions, we show that the policy obtained from infinite population equilibrium is $\varepsilon(N)$-Nash when the number of agents $N$ is sufficiently large, where $\varepsilon(N)$ is an explicit function of $N$. Then, using the linear programming formulation of MDPs and the linearity of the transition probability in mean-field term, we formulate the game in the infinite population limit as a generalized Nash equilibrium problem (GNEP) and establish an algorithm for computing equilibrium with a convergence guarantee.

eess.SY↗

arXiv2022

Learning in Discounted-cost and Average-cost Mean-field Games

We consider learning approximate Nash equilibria for discrete-time mean-field games with nonlinear stochastic state dynamics subject to both average and discounted costs. To this end, we introduce a mean-field equilibrium (MFE) operator, whose fixed point is a mean-field equilibrium (i.e. equilibrium in the infinite population limit). We first prove that this operator is a contraction, and propose a learning algorithm to compute an approximate mean-field equilibrium by approximating the MFE operator with a random one. Moreover, using the contraction property of the MFE operator, we establish the error analysis of the proposed learning algorithm. We then show that the learned mean-field equilibrium constitutes an approximate Nash equilibrium for finite-agent games.

eess.SY↗

arXiv2022

Partially Observed Discrete-Time Risk-Sensitive Mean Field Games

In this paper, we consider discrete-time partially observed mean-field games with the risk-sensitive optimality criterion. We introduce risk-sensitivity behaviour for each agent via an exponential utility function. In the game model, each agent is weakly coupled with the rest of the population through its individual cost and state dynamics via the empirical distribution of states. We establish the mean-field equilibrium in the infinite-population limit using the technique of converting the underlying original partially observed stochastic control problem to a fully observed one on the belief space and the dynamic programming principle. Then, we show that the mean-field equilibrium policy, when adopted by each agent, forms an approximate Nash equilibrium for games with sufficiently many agents. We first consider finite-horizon cost function, and then, discuss extension of the result to infinite-horizon cost in the next-to-last section of the paper.

eess.SY↗

arXiv2022

Q-Learning in Regularized Mean-field Games

In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. Regularization is introduced by adding a strongly concave regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. The regularization term in general makes reinforcement learning algorithm more robust to the system components. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

math.OC↗

arXiv2022

Nash Equilibria for Exchangeable Team against Team Games, their Mean Field Limit, and Role of Common Randomness

We study stochastic mean-field games among finite number of teams with large finite as well as infinite number of decision makers. For this class of games within static and dynamic settings, we establish the existence of a Nash equilibrium, and show that a Nash equilibrium exhibits exchangeability in the finite decision maker regime and symmetry in the infinite one. To arrive at these existence and structural theorems, we endow the set of randomized policies with a suitable topology under various decentralized information structures, which leads to the desired convexity and compactness of the set of randomized policies. Then, we establish the existence of a randomized Nash equilibrium that is exchangeable (not necessarily symmetric) among decision makers within each team for a general class of exchangeable stochastic games. As the number of decision makers within each team goes to infinity (that is for the mean-field game among teams), using a de Finetti representation theorem, we show existence of a randomized Nash equilibrium that is symmetric (i.e., identical) among decision makers within each team and also independently randomized. Finally, we establish that a Nash equilibrium for a class of mean-field games among teams (which is symmetric) constitutes an approximate Nash equilibrium for the corresponding pre-limit (exchangeable) game among teams with large but finite number of decision makers. We thus show that common randomness is not necessary for large team-against-team games, unlike the case with small sized teams.

math.OC↗

arXiv2021

Large Deviations Principle for Discrete-time Mean-field Games

In this paper, we establish a large deviations principle (LDP) for interacting particle systems that arise from state and action dynamics of discrete-time mean-field games under the equilibrium policy of the infinite-population limit. The LDP is proved under weak Feller continuity of state and action dynamics. The proof is based on transferring LDP for empirical measures of initial states and noise variables under setwise topology to the original game model via contraction principle, which was first suggested by Delarue, Lacker, and Ramanan to establish LDP for continuous-time mean-field games under common noise. We also compare our work with LDP results established in prior literature for interacting particle systems, which are in a sense uncontrolled versions of mean-field games.

eess.SY↗

arXiv2021

Optimality of Independently Randomized Symmetric Policies for Exchangeable Stochastic Teams with Infinitely Many Decision Makers

We study stochastic team (known also as decentralized stochastic control or identical interest stochastic dynamic game) problems with large or countably infinite number of decision makers, and characterize existence and structural properties for (globally) optimal policies. We consider both static and dynamic non-convex team problems where the cost function and dynamics satisfy an exchangeability condition. To arrive at existence and structural results on optimal policies, we first introduce a topology on control policies, which involves various relaxations given the decentralized information structure. This is then utilized to arrive at a de Finetti type representation theorem for exchangeable policies. This leads to a representation theorem for policies which admit an infinite exchangeability condition. For a general setup of stochastic team problems with $N$ decision makers, under exchangeability of observations of decision makers and the cost function, we show that without loss of global optimality, the search for optimal policies can be restricted to those that are $N$-exchangeable. Then, by extending $N$-exchangeable policies to infinitely-exchangeable ones, establishing a convergence argument for the induced costs, and using the presented de Finetti type theorem, we establish the existence of an optimal decentralized policy for static and dynamic teams with countably infinite number of decision makers, which turns out to be symmetric (i.e., identical) and randomized. In particular, unlike prior work, convexity of the cost in policies is not assumed. Finally, we show near optimality of symmetric independently randomized policies for finite $N$-decision maker team problems and thus establish approximation results for $N$-decision maker weakly coupled stochastic teams.

math.OC↗

arXiv2020

Regularized Stochastic Team Problems

In this paper, we introduce regularized stochastic team problems. Under mild assumptions, we prove that there exists an unique fixed point of the best response operator, where this unique fixed point is the optimal regularized team decision rule. Then, we establish an asynchronous distributed algorithm to compute this optimal strategy. We also provide a bound that shows how the optimal regularized team decision rule performs in the original stochastic team problem.

math.OC↗

arXiv2020

Geometry of Information Structures, Strategic Measures and associated Control Topologies

In many areas of applied mathematics, engineering, and social and natural sciences, decentralization of information is a key aspect determining how to approach a problem. In this review article, we study information structures in a probability theoretic and geometric context. We define information structures, place various topologies on them, and study closedness, compactness and convexity properties on the strategic measures induced by information structures and decentralized control/decision policies under varying degree of relaxations with regard to access to private or common randomness. Ultimately, we present existence and tight approximation results for optimal decision/control policies. We discuss various lower bounding techniques, through relaxations and convex programs ranging from classically realizable and classically non-realizable (such as quantum and non-signaling) relaxations. For each of these, we establish closedness and convexity properties and also a hierarchy of correlation structures. As a second main theme, we review and introduce various topologies on decision/control strategies defined independent of information structures, but for which information structures determine whether the topologies entail utility in arriving at existence, compactness, convexification or approximation results. These approaches, which we term as the strategic measures approach and the control topology approach, lead to complementary results on existence, approximations and upper and lower bounds in optimal decentralized decision and control.

math.OC↗

arXiv2020

Non-signaling Approximations of Stochastic Team Problems

In this paper, we consider non-signaling approximation of finite stochastic teams. We first introduce a hierarchy of team decision rules that can be classified in an increasing order as randomized policies, quantum-correlated policies, and non-signaling policies. Then, we establish an approximation of team-optimal policies for sequential teams via extendible non-signaling policies. We prove that the distance between extendible non-signaling policies and decentralized policies is small if the extension is sufficiently large. Using this result, we establish a linear programming (LP) approximation of sequential teams. Finally, we state an open problem regarding computation of optimal value of quantum-correlated policies.

eess.SY↗

arXiv2020

Value Iteration Algorithm for Mean-field Games

In the literature, existence of mean-field equilibria has been established for discrete-time mean field games under both the discounted cost and the average cost optimality criteria. In this paper, we provide a value iteration algorithm to compute mean-field equilibrium for both the discounted cost and the average cost criteria, whose existence proved previously. We establish that the value iteration algorithm converges to the fixed point of a mean-field equilibrium operator. Then, using this fixed point, we construct a mean-field equilibrium. In our value iteration algorithm, we use $Q$-functions instead of value functions.

eess.SY↗

arXiv2019

Discrete-time average-cost mean-field games on Polish spaces

In stochastic dynamic games, when the number of players is sufficiently large and the interactions between agents depend on empirical state distribution, one way to approximate the original game is to introduce infinite-population limit of the problem. In the infinite population limit, a generic agent is faced with a \emph{so-called} mean-field game. In this paper, we study discrete-time mean-field games with average-cost criteria. Using average cost optimality equation and Kakutani's fixed point theorem, we establish the existence of Nash equilibria for mean-field games under drift and minorization conditions on the dynamics of each agent. Then, we show that the equilibrium policy in the mean-field game, when adopted by each agent, is an approximate Nash equilibrium for the corresponding finite-agent game with sufficiently many agents.

math.OC↗

arXiv2019

Weak Feller Property of Non-linear Filters

Weak Feller property of controlled and control-free Markov chains lead to many desirable properties. In control-free setups this leads to the existence of invariant probability measures for compact spaces and applicability of numerical approximation methods. For controlled setups, this leads to existence and approximation results for optimal control policies. We know from stochastic control theory that partially observed systems can be converted to fully observed systems by replacing the original state space with a probability measure-valued state space, with the corresponding kernel acting on probability measures known as the non-linear filter (belief) process. Establishing sufficient conditions for the weak Feller property for such processes is a significant problem, studied under various assumptions and setups in the literature. In this paper, we prove the weak Feller property of the non-linear filter process (i) first under weak continuity of the transition probability of controlled Markov chain and total variation continuity of its observation channel, and then, (ii) under total variation continuity of the transition probability of controlled Markov chain. The former result (i) has first appeared in Feinberg et. al. [Math. Oper. Res. 41(2) (2016) 656-681]. Here, we present a concise and easy to follow alternative proof for this existing result. The latter result (ii) establishes weak Feller property of non-linear filter process under conditions, which have not been previously reported in the literature.

math.OC↗

arXiv2018

Discrete-time Risk-sensitive Mean-field Games

In this paper, we study a class of discrete-time mean-field games under the infinite-horizon risk-sensitive discounted-cost optimality criterion. Risk-sensitivity is introduced for each agent (player) via an exponential utility function. In this game model, each agent is coupled with the rest of the population through the empirical distribution of the states, which affects both the agent's individual cost and its state dynamics. Under mild assumptions, we establish the existence of a mean-field equilibrium in the infinite-population limit as the number of agents ($N$) goes to infinity, and then show that the policy obtained from the mean-field equilibrium constitutes an approximate Nash equilibrium when $N$ is sufficiently large.

math.OC↗

arXiv2018

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

In this paper, we consider the finite-state approximation of a discrete-time constrained Markov decision process (MDP) under the discounted and average cost criteria. Using the linear programming formulation of the constrained discounted cost problem, we prove the asymptotic convergence of the optimal value of the finite-state model to the optimal value of the original model. With further continuity condition on the transition probability, we also establish a method to compute approximately optimal policies. For the average cost, instead of using the finite-state linear programming approximation method, we use the original problem definition to establish the finite-state asymptotic approximation of the constrained problem and compute approximately optimal policies. Under Lipschitz type regularity conditions on the components of the MDP, we also obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases.

math.OC↗

arXiv2018

Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions

Establishing the existence of Nash equilibria for partially observed stochastic dynamic games is known to be quite challenging, with the difficulties stemming from the noisy nature of the measurements available to individual players (agents) and the decentralized nature of this information. When the number of players is sufficiently large and the interactions among agents is of the mean-field type, one way to overcome this challenge is to investigate the infinite-population limit of the problem, which leads to a mean-field game. In this paper, we consider discrete-time partially observed mean-field games with infinite-horizon discounted cost criteria. Using the technique of converting the original partially observed stochastic control problem to a fully observed one on the belief space and the dynamic programming principle, we establish the existence of Nash equilibria for these game models under very mild technical conditions. Then, we show that the mean-field equilibrium policy, when adopted by each agent, forms an approximate Nash equilibrium for games with sufficiently many agents.

eess.SY↗

arXiv2017

Finite Model Approximations for Partially Observed Markov Decision Processes with Discounted Cost

We consider finite model approximations of discrete-time partially observed Markov decision processes (POMDPs) under the discounted cost criterion. After converting the original partially observed stochastic control problem to a fully observed one on the belief space, the finite models are obtained through the uniform quantization of the state and action spaces of the belief space Markov decision process (MDP). Under mild assumptions on the components of the original model, it is established that the policies obtained from these finite models are nearly optimal for the belief space MDP, and so, for the original partially observed problem. The assumptions essentially require that the belief space MDP satisfies a mild weak continuity condition. We provide examples and introduce explicit approximation procedures for the quantization of the set of probability measures on the state space of POMDP (i.e., belief space).

eess.SY↗

arXiv2017

Markov-Nash Equilibria in Mean-Field Games with Discounted Cost

In this paper, we consider discrete-time dynamic games of the mean-field type with a finite number $N$ of agents subject to an infinite-horizon discounted-cost optimality criterion. The state space of each agent is a locally compact Polish space. At each time, the agents are coupled through the empirical distribution of their states, which affects both the agents' individual costs and their state transition probabilities. We introduce a new solution concept of the Markov-Nash equilibrium, under which a policy is player-by-player optimal in the class of all Markov policies. Under mild assumptions, we demonstrate the existence of a mean-field equilibrium in the infinite-population limit $N \to \infty$, and then show that the policy obtained from the mean-field equilibrium is approximately Markov-Nash when the number of agents $N$ is sufficiently large.

eess.SY↗