arXiv subjects

David Mguni

Publications and source records attributed to David Mguni.

27 records · Page 2Linked to original sources

arXiv2021

Stochastic Potential Games

Computing the Nash equilibrium (NE) for N-player non-zerosum stochastic games is a formidable challenge. Currently, algorithmic methods in stochastic game theory are unable to compute NE for stochastic games (SGs) for settings in all but extreme cases in which the players either play as a team or have diametrically opposed objectives in a two-player setting. This greatly impedes the application of the SG framework to numerous problems within economics and practical systems of interest. In this paper, we provide a method of computing Nash equilibria in nonzero-sum settings and for populations of players more than two. In particular, we identify a subset of SGs known as stochastic potential games (SPGs) for which the (Markov perfect) Nash equilibrium can be computed tractably and in polynomial time. Unlike SGs for which, in general, computing the NE is PSPACE-hard, we show that SGs with the potential property are P-Complete. We further demonstrate that for each SPG there is a dual Markov decision process whose solution coincides with the MP-NE of the SPG. We lastly provide algorithms that tractably compute the MP-NE for SGs with more than two players.

math.OC↗

arXiv2020

Multi-Agent Determinantal Q-Learning

Centralized training with decentralized execution has become an important paradigm in multi-agent learning. Though practical, current methods rely on restrictive assumptions to decompose the centralized value function across agents for execution. In this paper, we eliminate this restriction by proposing multi-agent determinantal Q-learning. Our method is established on Q-DPP, an extension of determinantal point process (DPP) with partition-matroid constraint to multi-agent setting. Q-DPP promotes agents to acquire diverse behavioral models; this allows a natural factorization of the joint Q-functions with no need for \emph{a priori} structural constraints on the value function or special network architectures. We demonstrate that Q-DPP generalizes major solutions including VDN, QMIX, and QTRAN on decentralizable cooperative tasks. To efficiently draw samples from Q-DPP, we adopt an existing sample-by-projection sampler with theoretical approximation guarantee. The sampler also benefits exploration by coordinating agents to cover orthogonal directions in the state space during multi-agent training. We evaluate our algorithm on various cooperative benchmarks; its effectiveness has been demonstrated when compared with the state-of-the-art.

cs.LG↗

arXiv2019

Optimal Selection of Transaction Costs in a Dynamic Principal-Agent Problem

Environments with fixed adjustment costs such as transaction costs or \lq menu costs\rq$ $ are widespread within economic systems. The presence of fixed minimal adjustment costs produces adjustment stickiness so that agents must choose a sequence of points at which time to perform their actions. This paper performs an analysis of the effect of transaction costs on agent behaviour within a dynamic optimisation problem by way of introducing the theory of incentive design to optimal stochastic impulse control. The setup consists of an agent that maximises their utility by performing a sequence of purchases of some consumable good over some time horizon whilst facing transaction costs and a Principal that chooses the agent's transaction costs. This results in a dynamic Principal-Agent model in which the agent uses impulse controls to perform adjustments to their cash-flow process. We address the question of which fixed value of the transaction cost the Principal must choose to induce a desired behaviour from the agent. We study the effect of changes to the transaction cost and show that with an appropriate choice of transaction cost, the agent's preferences can be sufficiently distorted so that the agent finds it optimal to maximise the Principal's objective even when the agent's cash-flow is unobserved by the Principal.

math.OC↗

arXiv2019

Cutting Your Losses: Learning Fault-Tolerant Control and Optimal Stopping under Adverse Risk

Recently, there has been a surge in interest in safe and robust techniques within reinforcement learning (RL). Current notions of risk in RL fail to capture the potential for systemic failures such as abrupt stoppages from system failures or surpassing of safety thresholds and the appropriate responsive controls in such instances. We propose a novel approach to risk minimisation within RL in which, in addition to taking actions that maximise its expected return, the controller learns a policy that is robust against stoppages due to an adverse event such as an abrupt failure. The results of the paper cover fault-tolerant control in \textit{worst-case scenarios} under random stopping and optimal stopping, all in unknown environments. By demonstrating that the class of problems is represented by a variant of stochastic games, we prove the existence of a solution which is a unique fixed point equilibrium of the game and characterise the optimal controller behaviour. We then introduce a value function approximation algorithm that converges to the solution through simulation in unknown environments.

eess.SY↗

arXiv2019

A Viscosity Approach to Stochastic Differential Games of Control and Stopping Involving Impulsive Control

This paper analyses a stochastic differential game of control and stopping in which one of the players modifies a diffusion process using impulse controls, an adversary then chooses a stopping time to end the game. The paper firstly establishes the regularity and boundedness of the upper and lower value functions from which an appropriate variant of the dynamic programming principle (DPP) is derived. It is then proven that the upper and lower value functions coincide so that the game admits a value and that the value of the game is a unique viscosity solution to a HJBI equation described by a double obstacle quasi-integro-variational inequality.

math.OC↗

arXiv2019

Optimal Capital Injections with the Risk of Ruin: A Stochastic Differential Game of Impulse Control and Stopping Approach

We consider an investment problem in which an investor performs capital injections to increase the liquidity of a firm for it to maximise profit from market operations. Each time the investor performs an injection, the investor incurs a fixed transaction cost. In addition to maximising their terminal reward, the investor seeks to minimise the risk of loss of their investment (from a possible firm ruin) by exiting the market at some point in time. We show that the problem can be reformulated in terms of a new stochastic differential game of control and stopping in which one of the players modifies a (jump-)diffusion process using impulse controls and an adversary chooses a stopping time to end the game. We show that the value of this game can be computed by solving a double obstacle problem described by a quasi-variational inequality. We then characterise the value of the game via a set of HJBI equations, considering both games with zero-sum and non-zero-sum payoff structures. Our last result demonstrates that the solution to the investment problem is recoverable from the Nash equilibrium strategies of the game.

math.OC↗

arXiv2019

Duopoly Investment Problems with Minimally Bounded Adjustment Costs

In this paper, we study two-player investment problems with investment costs that are bounded below by some fixed positive constant. We seek a description of optimal investment strategies for a duopoly problem in which two firms invest in advertising projects to abstract market share from the rival firm. We show that the problem can be formulated as a stochastic differential game in which players modify a jump-diffusion process using impulse controls. We prove that the value of the game may be represented as a solution to a double obstacle quasi-variational inequality and derive a PDE characterisation (HJBI equation) of the value of the game. We characterise both the saddle point equilibrium and a Nash equilibrium for the zero-sum and non-zero-sum payoff games.

math.OC↗

arXiv2019

Coordinating the Crowd: Inducing Desirable Equilibria in Non-Cooperative Systems

Many real-world systems such as taxi systems, traffic networks and smart grids involve self-interested actors that perform individual tasks in a shared environment. However, in such systems, the self-interested behaviour of agents produces welfare inefficient and globally suboptimal outcomes that are detrimental to all - some common examples are congestion in traffic networks, demand spikes for resources in electricity grids and over-extraction of environmental resources such as fisheries. We propose an incentive-design method which modifies agents' rewards in non-cooperative multi-agent systems that results in independent, self-interested agents choosing actions that produce optimal system outcomes in strategic settings. Our framework combines multi-agent reinforcement learning to simulate (real-world) agent behaviour and black-box optimisation to determine the optimal modifications to the agents' rewards or incentives given some fixed budget that results in optimal system performance. By modifying the reward functions and generating agents' equilibrium responses within a sequence of offline Markov games, our method enables optimal incentive structures to be determined offline through iterative updates of the reward functions of a simulated game. Our theoretical results show that our method converges to reward modifications that induce system optimality. We demonstrate the applications of our framework by tackling a challenging problem within economics that involves thousands of selfish agents and tackle a traffic congestion problem.

cs.MA↗

arXiv2018

Decentralised Learning in Systems with Many, Many Strategic Agents

Although multi-agent reinforcement learning can tackle systems of strategically interacting entities, it currently fails in scalability and lacks rigorous convergence guarantees. Crucially, learning in multi-agent systems can become intractable due to the explosion in the size of the state-action space as the number of agents increases. In this paper, we propose a method for computing closed-loop optimal policies in multi-agent systems that scales independently of the number of agents. This allows us to show, for the first time, successful convergence to optimal behaviour in systems with an unbounded number of interacting adaptive learners. Studying the asymptotic regime of N-player stochastic games, we devise a learning protocol that is guaranteed to converge to equilibrium policies even when the number of agents is extremely large. Our method is model-free and completely decentralised so that each agent need only observe its local state information and its realised rewards. We validate these theoretical results by showing convergence to Nash-equilibrium policies in applications from economics and control theory with thousands of strategically interacting agents.

cs.MA↗