arXiv subjects

Mathieu Lauriere

Publications and source records attributed to Mathieu Lauriere.

At least 19 recordsLinked to original sources

arXiv2026

Computational Trade-Offs Between Newton and Picard Solvers for Mean Field Game PDE Systems

We study computational trade-offs between two solvers for the same semi-implicit finite-difference discretization of forward-backward partial differential equation (PDE) systems arising in mean field games (MFGs). The Picard method uses an outer fixed-point iteration that alternates a forward Fokker-Planck solve and a backward Hamilton-Jacobi-Bellman solve. The Newton method instead applies Newton's method directly to the coupled nonlinear space-time system. Across one- and two-dimensional MFG benchmarks, we find that the Picard method has much lower wall-clock cost when it converges, but may fail at sufficiently low viscosity and may require strong damping under temporal shocks. With parameter continuation, the Newton method is more robust in these regimes, at the cost of larger coupled linear systems. We relate these trade-offs to the residuals, Jacobian blocks, and sparsity structures produced by separable, local nonseparable, and nonlocal Hamiltonians. We also test a hybrid continuation strategy on a two-dimensional double-well MFG benchmark, using Picard iterations for the inexpensive part of the viscosity descent before switching to Newton continuation.

math.NA↗

arXiv2025

Population-aware Online Mirror Descent for Mean-Field Games with Common Noise by Deep Reinforcement Learning

Mean Field Games (MFGs) offer a powerful framework for studying large-scale multi-agent systems. Yet, learning Nash equilibria in MFGs remains a challenging problem, particularly when the initial distribution is unknown or when the population is subject to common noise. In this paper, we introduce an efficient deep reinforcement learning (DRL) algorithm designed to achieve population-dependent Nash equilibria without relying on averaging or historical sampling, inspired by Munchausen RL and Online Mirror Descent. The resulting policy is adaptable to various initial distributions and sources of common noise. Through numerical experiments on seven canonical examples, we demonstrate that our algorithm exhibits superior convergence properties compared to state-of-the-art algorithms, particularly a DRL version of Fictitious Play for population-dependent policies. The performance in the presence of common noise underscores the robustness and adaptability of our approach.

cs.LG↗

arXiv2025

A Game-Theoretic Framework for Network Formation in Large Populations

In this paper, we study a model of network formation in large populations. Each agent can choose the strength of interaction (i.e. connection) with other agents to find a Nash equilibrium. Different from the recently-developed theory of graphon games, here each agent's control depends not only on her own index but also on the index of other agents. After defining the general model of the game, we focus on a special case with piecewise constant graphs and we provide optimality conditions through a system of forward-backward stochastic differential equations. Furthermore, we show the uniqueness and existence results. Finally, we provide numerical experiments to discuss the effects of different model settings.

math.OC↗

arXiv2025

Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control

Reinforcement learning (RL) is currently one of the most prominent methods for optimizing dynamical systems, with breakthrough results across various fields. The framework is based on the concept of a Markov decision process (MDP), leading to a discrete-time optimal control problem. In the RL literature, such problems are typically formulated and solved using mixed policies, from which random actions are sampled at each time step. Recently, part of the optimal control community has begun investigating continuous-time versions of RL algorithms, replacing MDPs with continuous-time stochastic processes governed by relaxed controls, and asserting a full analogy between the two formulations. In this work, we examine the limitations of this analogy and rigorously establish a connection between the two problems in the case where only the drift term of the continuous-time model is controlled. We prove strong convergence of the RL implementation of mixed strategies as the time discretization mesh tends to zero. We also discuss the technical challenges posed by the possible presence of control in the diffusion component of the state.

math.OC↗

arXiv2025

Cooperation, Competition, and Common Pool Resources in Mean Field Games

Mean field games (MFGs) have been introduced to study Nash equilibria in very large population of self-interested agents. However, when applied to common pool resource (CPR) games, MFG equilibria lead to the so-called tragedy of the commons (TOTC). Empirical studies have shown that in many situations, TOTC does not materialize which hints at the fact that standard MFG models cannot explain the behavior of agents in CPR games. In this work, we study two models which incorporate a mix of cooperative and non-cooperative behaviors, either at the individual level or the population level. After defining these models, we study optimality conditions in the form of forward-backward stochastic differential equations and we prove that the mean field models provide approximate equilibria controls for corresponding finite-agent games. We then show an application to a model of fish stock management, for which the solution can be computed by solving systems of ordinary differential equations, which we prove to have a unique solution. Numerical results illustrate the impact of the level of cooperation at the individual and the population levels on the CPR.

math.OC↗

arXiv2024

How can the tragedy of the commons be prevented?: Introducing Linear Quadratic Mixed Mean Field Games

In a regular mean field game (MFG), the agents are assumed to be insignificant, they do not realize their effect on the population level and this may result in a phenomenon coined as the Tragedy of the Commons by the economists. However, in real life this phenomenon is often avoided thanks to the underlying altruistic behavior of (all or some of the) agents. Motivated by this observation, we introduce and analyze two different mean field models to include altruism in the decision making of agents. In the first model, mixed individual MFGs, there are infinitely many agents who are partially altruistic (i.e., they behave partially cooperatively) and partially non-cooperative. In the second model, mixed population MFGs, one part of the population behaves cooperatively and the remaining agents behave non-cooperatively. Both models are introduced in a general linear quadratic framework for which we characterize the equilibrium via forward backward stochastic differential equations. Furthermore, we give explicit solutions in terms of ordinary differential equations, and prove the existence and uniqueness results.

math.OC↗

arXiv2024

Machine Learning Methods for Large Population Games with Applications in Operations Research

In this tutorial, we provide an introduction to machine learning methods for finding Nash equilibria in games with large number of agents. These types of problems are important for the operations research community because of their applicability to real life situations such as control of epidemics, optimal decisions in financial markets, electricity grid management, or traffic control for self-driving cars. We start the tutorial by introducing stochastic optimal control problems for a single agent, in discrete time and in continuous time. Then, we present the framework of dynamic games with finite number of agents. To tackle games with a very large number of agents, we discuss the paradigm of mean field games, which provides an efficient way to compute approximate Nash equilibria. Based on this approach, we discuss machine learning algorithms for such problems. First in the context of discrete time games, we introduce fixed point based methods and related methods based on reinforcement learning. Second, we discuss machine learning methods that are specific to continuous time problems, by building on optimality conditions phrased in terms of stochastic or partial differential equations. Several examples and numerical illustrations of problems arising in operations research are provided along the way.

math.OC↗

arXiv2024

Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning

Mean Field Games (MFGs) have the ability to handle large-scale multi-agent systems, but learning Nash equilibria in MFGs remains a challenging task. In this paper, we propose a deep reinforcement learning (DRL) algorithm that achieves population-dependent Nash equilibrium without the need for averaging or sampling from history, inspired by Munchausen RL and Online Mirror Descent. Through the design of an additional inner-loop replay buffer, the agents can effectively learn to achieve Nash equilibrium from any distribution, mitigating catastrophic forgetting. The resulting policy can be applied to various initial distributions. Numerical experiments on four canonical examples demonstrate our algorithm has better convergence properties than SOTA algorithms, in particular a DRL version of Fictitious Play for population-dependent policies.

cs.GT↗

arXiv2023

From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspective

Mean field games (MFG) and mean field control (MFC) problems have been introduced to study large populations of strategic players. They correspond respectively to non-cooperative or cooperative scenarios, where the aim is to find the Nash equilibrium and social optimum. These frameworks provide approximate solutions to situations with a finite number of players and have found a wide range of applications, from economics to biology and machine learning. In this paper, we study how the players can pass from a non-cooperative to a cooperative regime, and vice versa. The first direction is reminiscent of mechanism design, in which the game's definition is modified so that non-cooperative players reach an outcome similar to a cooperative scenario. The second direction studies how players that are initially cooperative gradually deviate from a social optimum to reach a Nash equilibrium when they decide to optimize their individual cost similar to the free rider phenomenon. To formalize these connections, we introduce two new classes of games which lie between MFG and MFC: $\lambda$-interpolated mean field games, in which the cost of an individual player is a $\lambda$-interpolation of the MFG and the MFC costs, and $p$-partial mean field games, in which a proportion $p$ of the population deviates from the social optimum by playing the game non-cooperatively. We conclude the paper by providing an algorithm for myopic players to learn a $p$-partial mean field equilibrium, and we illustrate it on a stylized model.

cs.GT↗

arXiv2023

Non-standard Stochastic Control with Nonlinear Feynman-Kac Costs

We consider the conditional control problem introduced by P.L. Lions in his lectures at the Coll\`ege de France in November 2016. In his lectures, Lions emphasized some of the major differences with the analysis of classical stochastic optimal control problems, and in so doing, raised the question of the possible differences between the value functions resulting from optimization over the class of Markovian controls as opposed to the general family of open loop controls. The goal of the paper is to elucidate this quandary and provide elements of response to Lions' original conjecture. First, we justify the mathematical formulation of the conditional control problem by the description of practical model from evolutionary biology. Next, we relax the original formulation by the introduction of \emph{soft} as opposed to hard killing, and using a \emph{mimicking} argument, we reduce the open loop optimization problem to an optimization over a specific class of feedback controls. After proving existence of optimal feedback control functions, we prove a superposition principle allowing us to recast the original stochastic control problems as deterministic control problems for dynamical systems of probability Gibbs measures. Next, we characterize the solutions by forward-backward systems of coupled non-linear Partial Differential Equations (PDEs) very much in the spirit of the Mean Field Game (MFG) systems. From there, we identify a common optimizer, proving the conjecture of equality of the value functions. Finally we illustrate the results by convincing numerical experiments.

math.OC↗

arXiv2023

Multi-population Mean Field Games with Multiple Major Players: Application to Carbon Emission Regulations

In this paper, we propose and study a mean field game model with multiple populations of minor players and multiple major players, motivated by applications to the regulation of carbon emissions. Each population of minor players represent a large group of electricity producers and each major player represents a regulator. We first characterize the minor players equilibrium controls using forward-backward differential equations, and show existence and uniqueness of the minor players equilibrium. We then express the major players' equilibrium controls through analytical formulas given the other players' controls. Finally, we then provide a method to solve the Nash equilibrium between all the players, and we illustrate numerically the sensitivity of the model to its parameters.

math.OC↗

arXiv2023

Deep Learning for Population-Dependent Controls in Mean Field Control Problems with Common Noise

In this paper, we propose several approaches to learn the optimal population-dependent controls in order to solve mean field control problems (MFC). Such policies enable us to solve MFC problems with forms of common noises at a level of generality that was not covered by existing methods. We analyze rigorously the theoretical convergence of the proposed approximation algorithms. Of particular interest for its simplicity of implementation is the $N$-particle approximation. The effectiveness and the flexibility of our algorithms is supported by numerical experiments comparing several combinations of distribution approximation techniques and neural network architectures. We use three different benchmark problems from the literature: a systemic risk model, a price impact model, and a crowd motion model. We first show that our proposed algorithms converge to the correct solution in an explicitly solvable MFC problem. Then, we show that population-dependent controls outperform state-dependent controls. Along the way, we show that specific neural network architectures can improve the learning further.

math.OC↗

arXiv2023

The communication complexity of functions with large outputs

We study the two-party communication complexity of functions with large outputs, and show that the communication complexity can greatly vary depending on what output model is considered. We study a variety of output models, ranging from the open model, in which an external observer can compute the outcome, to the XOR model, in which the outcome of the protocol should be the bitwise XOR of the players' local outputs. This model is inspired by XOR games, which are widely studied two-player quantum games. We focus on the question of error-reduction in these new output models. For functions of output size k, applying standard error reduction techniques in the XOR model would introduce an additional cost linear in k. We show that no dependency on k is necessary. Similarly, standard randomness removal techniques, incur a multiplicative cost of $2^k$ in the XOR model. We show how to reduce this factor to O(k). In addition, we prove analogous error reduction and randomness removal results in the other models, separate all models from each other, and show that some natural problems, including Set Intersection and Find the First Difference, separate the models when the Hamming weights of their inputs is bounded. Finally, we show how to use the rank lower bound technique for our weak output models.

cs.CC↗

arXiv2023

A Machine Learning Method for Stackelberg Mean Field Games

We propose a single-level numerical approach to solve Stackelberg mean field game (MFG) problems. In Stackelberg MFG, an infinite population of agents play a non-cooperative game and choose their controls to optimize their individual objectives while interacting with the principal and other agents through the population distribution. The principal can influence the mean field Nash equilibrium at the population level through policies, and she optimizes her own objective, which depends on the population distribution. This leads to a bi-level problem between the principal and mean field of agents that cannot be solved using traditional methods for MFGs. We propose a reformulation of this problem as a single-level mean field optimal control problem through a penalization approach. We prove convergence of the reformulated problem to the original problem. We propose a machine learning method based on (feed-forward and recurrent) neural networks and illustrate it on several examples from the literature.

math.OC↗

arXiv2022

Learning Correlated Equilibria in Mean-Field Games

The designs of many large-scale systems today, from traffic routing environments to smart grids, rely on game-theoretic equilibrium concepts. However, as the size of an $N$-player game typically grows exponentially with $N$, standard game theoretic analysis becomes effectively infeasible beyond a low number of players. Recent approaches have gone around this limitation by instead considering Mean-Field games, an approximation of anonymous $N$-player games, where the number of players is infinite and the population's state distribution, instead of every individual player's state, is the object of interest. The practical computability of Mean-Field Nash equilibria, the most studied Mean-Field equilibrium to date, however, typically depends on beneficial non-generic structural properties such as monotonicity or contraction properties, which are required for known algorithms to converge. In this work, we provide an alternative route for studying Mean-Field games, by developing the concepts of Mean-Field correlated and coarse-correlated equilibria. We show that they can be efficiently learnt in \emph{all games}, without requiring any additional assumption on the structure of the game, using three classical algorithms. Furthermore, we establish correspondences between our notions and those already present in the literature, derive optimality bounds for the Mean-Field - $N$-player transition, and empirically demonstrate the convergence of these algorithms on simple games.

cs.GT↗

arXiv2022

Reinforcement Learning Algorithm for Mixed Mean Field Control Games

We present a new combined \textit{mean field control game} (MFCG) problem which can be interpreted as a competitive game between collaborating groups and its solution as a Nash equilibrium between groups. Players coordinate their strategies within each group. An example is a modification of the classical trader's problem. Groups of traders maximize their wealth. They face cost for their transactions, for their own terminal positions, and for the average holding within their group. The asset price is impacted by the trades of all agents. We propose a three-timescale reinforcement learning algorithm to approximate the solution of such MFCG problems. We test the algorithm on benchmark linear-quadratic specifications for which we provide analytic solutions.

math.OC↗

arXiv2021

Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO

Recent advances in multiagent learning have seen the introduction ofa family of algorithms that revolve around the population-based trainingmethod PSRO, showing convergence to Nash, correlated and coarse corre-lated equilibria. Notably, when the number of agents increases, learningbest-responses becomes exponentially more difficult, and as such ham-pers PSRO training methods. The paradigm of mean-field games pro-vides an asymptotic solution to this problem when the considered gamesare anonymous-symmetric. Unfortunately, the mean-field approximationintroduces non-linearities which prevent a straightforward adaptation ofPSRO. Building upon optimization and adversarial regret minimization,this paper sidesteps this issue and introduces mean-field PSRO, an adap-tation of PSRO which learns Nash, coarse correlated and correlated equi-libria in mean-field games. The key is to replace the exact distributioncomputation step by newly-defined mean-field no-adversarial-regret learn-ers, or by black-box optimization. We compare the asymptotic complexityof the approach to standard PSRO, greatly improve empirical bandit con-vergence speed by compressing temporal mixture weights, and ensure itis theoretically robust to payoff noise. Finally, we illustrate the speed andaccuracy of mean-field PSRO on several mean-field games, demonstratingconvergence to strong and weak equilibria.

cs.GT↗

arXiv2021

Solving N-player dynamic routing games with congestion: a mean field approach

The recent emergence of navigational tools has changed traffic patterns and has now enabled new types of congestion-aware routing control like dynamic road pricing. Using the fundamental diagram of traffic flows - applied in macroscopic and mesoscopic traffic modeling - the article introduces a new N-player dynamic routing game with explicit congestion dynamics. The model is well-posed and can reproduce heterogeneous departure times and congestion spill back phenomena. However, as Nash equilibrium computations are PPAD-complete, solving the game becomes intractable for large but realistic numbers of vehicles N. Therefore, the corresponding mean field game is also introduced. Experiments were performed on several classical benchmark networks of the traffic community: the Pigou, Braess, and Sioux Falls networks with heterogeneous origin, destination and departure time tuples. The Pigou and the Braess examples reveal that the mean field approximation is generally very accurate and computationally efficient as soon as the number of vehicles exceeds a few dozen. On the Sioux Falls network (76 links, 100 time steps), this approach enables learning traffic dynamics with more than 14,000 vehicles.

math.DS↗