arXiv subjects

Mathieu Laurière

Publications and source records attributed to Mathieu Laurière.

At least 37 records · Page 2Linked to original sources

arXiv2026

Dynamic Programming Principle and Stabilization for Mean-Field Quantum Filtering Systems

Working within the quantum filtering framework, we establish a dynamic programming principle in an infinite-dimensional setting by embedding the state space into the Hilbert-Schmidt space. We then study a stabilization problem for continuously monitored Ising-coupled qubits and, in the mean-field limit, demonstrate quantum state reduction together with exponential convergence toward prescribed eigenstates under suitable feedback laws.

quant-ph↗

arXiv2026

Clustering in Deep Stochastic Transformers

Transformers have revolutionized deep learning across various domains but understanding the precise token dynamics remains a theoretical challenge. Existing theories of deep Transformers with layer normalization typically predict that tokens cluster to a single point; however, these results rely on deterministic weight assumptions, which fail to capture the standard initialization scheme in Transformers. In this work, we show that accounting for the intrinsic stochasticity of random initialization alters this picture. More precisely, we analyze deep Transformers where noise arises from the random initialization of value matrices. Under diffusion scaling and token-wise RMS normalization, we prove that, as the number of Transformer layers goes to infinity, the discrete token dynamics converge to an interacting-particle system on the sphere where tokens are driven by a \emph{common} matrix-valued Brownian noise. In this limit, we show that initialization noise prevents the collapse to a single cluster predicted by deterministic models. For two tokens, we prove a phase transition governed by the interaction strength and the token dimension: unlike deterministic attention flows, antipodal configurations become attracting with positive probability. Numerical experiments confirm the predicted transition, reveal that antipodal formations persist for more than two tokens, and demonstrate that suppressing the intrinsic noise degrades accuracy.

stat.ML↗

arXiv2025

Discrete-Time Mean Field Type Games: Probabilistic Setup

We introduce a general probabilistic framework for discrete-time, infinite-horizon discounted Mean Field Type Games (MFTGs) with both global common noise and team-specific common noises. In our model, agents are allowed to use randomized actions, both at the individual level and at the team level. We formalize the concept of Mean Field Markov Games (MFMGs) and establish a connection between closed-loop policies in MFTGs and Markov policies in MFMGs through different layers of randomization. By leveraging recent results on infinite-horizon discounted games with infinite compact state-action spaces, we prove the existence of an optimal closed-loop policy for the original MFTG when the state spaces are at most countable and the action spaces are general Polish spaces. We also present an example satisfying our assumptions, called Mean Field Drift of Intentions, where the dynamics are strongly randomized, and we establish the existence of a Nash equilibrium using our theoretical results.

math.OC↗

arXiv2025

Deep Learning for the Multiple Optimal Stopping Problem

This paper presents a novel deep learning framework for solving multiple optimal stopping problems in high dimensions. While deep learning has recently shown promise for single stopping problems, the multiple exercise case involves complex recursive dependencies that remain challenging. We address this by combining the Dynamic Programming Principle with neural network approximation of the value function. Unlike policy-search methods, our algorithm explicitly learns the value surface. We first consider the discrete-time problem and analyze neural network training error. We then turn to continuous problems and analyze the additional error due to the discretization of the underlying stochastic processes. Numerical experiments on high-dimensional American basket options and nonlinear utility maximization demonstrate that our method provides an efficient and scalable method for the multiple optimal stopping problem.

math.OC↗

arXiv2025

Probabilistic Analysis of Graphon Mean Field Control

Motivated by recent interest in graphon mean field games and their applications, this paper provides a comprehensive probabilistic analysis of graphon mean field control (GMFC) problems, where the controlled dynamics are governed by a graphon mean field stochastic differential equation with heterogeneous mean field interactions. We formulate the GMFC problem with general graphon mean field dependence and establish the existence and uniqueness of the associated graphon mean field forward-backward stochastic differential equations (FBSDEs). We then derive a version of the Pontryagin stochastic maximum principle tailored to GMFC problems. Furthermore, we analyze the solvability of the GMFC problem for linear dynamics and study the continuity and stability of the graphon mean field FBSDEs under the optimal control profile. Finally, we show that the solution to the GMFC problem provides an approximately optimal solution for large systems with heterogeneous mean field interactions, based on a propagation of chaos result.

math.OC↗

arXiv2025

Deep Signature Approach for McKean-Vlasov FBSDEs in a Random Environment

Mean-field games with common noise provide a powerful framework for modeling the collective behavior of large populations subject to shared randomness, such as systemic risk in finance or environmental shocks in economics. These problems can be reformulated as McKean-Vlasov forward-backward stochastic differential equations (MV-FBSDEs) in a random environment, where the coefficients depend on the conditional law of the state given the common noise. Existing numerical methods, however, are largely limited to cases where interactions depend only on expectations or low-order moments, and therefore cannot address the general setting of full distributional dependence. In this work, we introduce a deep learning-based algorithm for solving MV-FBSDEs with common noise and general mean-field interactions. Building on fictitious play, our method iteratively solves conditional FBSDEs with fixed distributions, where the conditional law is efficiently represented using signatures, and then updates the distribution through supervised learning. Deep neural networks are employed both to solve the conditional FBSDEs and to approximate the distribution-dependent coefficients, enabling scalability to high-dimensional problems. Under suitable assumptions, we establish convergence in terms of the fictitious play iterations, with error controlled by the supervised learning step. Numerical experiments, including a distribution-dependent mean-field game with common noise, demonstrate the effectiveness of the proposed approach.

math.OC↗

arXiv2025

Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit solutions are available.

math.OC↗

arXiv2025

An Overview of Some Extensions of Mean Field Games beyond Perfect Homogeneity and Anonymity

The mean field games (MFG) paradigm was introduced to provide tractable approximations of games involving very large populations. The theory typically rests on two key assumptions: homogeneity, meaning that all players share the same dynamics and cost functions, and anonymity, meaning that each player interacts with others only through their empirical distribution. While these assumptions simplify the analysis, they can be restrictive for many applications. Fortunately, several extensions of the standard MFG framework that relax these assumptions have been developed in the literature. The purpose of these notes is to offer a pedagogical introduction to such models. In particular, we discuss multi-population MFGs, graphon MFGs, major-minor MFGs, and Stackelberg MFGs, as well as variants involving cooperative players.

math.OC↗

arXiv2025

Robust mean-field control under common noise uncertainty

We propose and analyze a framework for discrete-time robust mean-field control problems under common noise uncertainty. In this framework, the mean-field interaction describes the collective behavior of infinitely many cooperative agents' state and action, while the common noise -- a random disturbance affecting all agents' state dynamics -- is uncertain. A social planner optimizes over open-loop controls on an infinite horizon to maximize the representative agent's worst-case expected reward, where worst-case corresponds to the most adverse probability measure among all candidates inducing the unknown true law of the common noise process. We refer to this optimization as a robust mean-field control problem under common noise uncertainty. We first show that this problem arises as the asymptotic limit of a cooperative $N$-agent robust optimization problem, commonly known as propagation of chaos. We then prove the existence of an optimal open-loop control by linking the robust mean field control problem to a lifted robust Markov decision problem on the space of probability measures and by establishing the dynamic programming principle and Bellman--Isaac fixed point theorem for the lifted robust Markov decision problem. Finally, we complement our theoretical results with numerical experiments motivated by distribution planning and systemic risk in finance, highlighting the advantages of accounting for common noise uncertainty.

math.OC↗

arXiv2025

Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics

Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi-agent systems. Despite recent advancements in reinforcement learning (RL) for MFGs, existing methods are typically limited to finite spaces or stationary models, hindering their applicability to real-world problems. This paper introduces a novel deep reinforcement learning (DRL) algorithm specifically designed for non-stationary continuous MFGs. The proposed approach builds upon a Fictitious Play (FP) methodology, leveraging DRL for best-response computation and supervised learning for average policy representation. Furthermore, it learns a representation of the time-dependent population distribution using a Conditional Normalizing Flow. To validate the effectiveness of our method, we evaluate it on three different examples of increasing complexity. By addressing critical limitations in scalability and density approximation, this work represents a significant advancement in applying DRL techniques to complex MFG problems, bringing the field closer to real-world multi-agent systems.

cs.LG↗

arXiv2025

Iterative Schemes for Markov Perfect Equilibria

We study Markov perfect equilibria in continuous-time dynamic games with finitely many symmetric players. The corresponding Nash system reduces to the Nash-Lasry-Lions equation for the common value function, also known as the master equation in the mean-field setting. In the finite-state space problems we consider, this equation becomes a nonlinear ordinary differential equation admitting a unique classical solution. Leveraging this uniqueness, we prove the convergence of both Picard and weighted Picard iterations, yielding efficient computational methods. Numerical experiments confirm the effectiveness of algorithms based on this approach.

math.OC↗

arXiv2025

Reinforcement Learning for Finite Space Mean-Field Type Games

Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that the MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on the quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 4 environments with mean-field distributions of dimension up to $200$ show the scalability and efficiency of the proposed method.

cs.GT↗

arXiv2025

Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective

In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of \emph{stochastic} and \emph{non-stochastic} uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem in a worst-case (minimax) framework, which is is intractable in general. Thus, we focus on the Linear Quadratic setting to derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.

cs.MA↗

arXiv2025

Learning to Stop: Deep Learning for Mean Field Optimal Stopping

Optimal stopping is a fundamental problem in optimization with applications in risk management, finance, robotics, and machine learning. We extend the standard framework to a multi-agent setting, named multi-agent optimal stopping (MAOS), where agents cooperate to make optimal stopping decisions in a finite-space, discrete-time environment. Since solving MAOS becomes computationally prohibitive as the number of agents is very large, we study the mean-field optimal stopping (MFOS) problem, obtained as the number of agents tends to infinity. We establish that MFOS provides a good approximation to MAOS and prove a dynamic programming principle (DPP) based on mean-field control theory. We then propose two deep learning approaches: one that learns optimal stopping decisions by simulating full trajectories and another that leverages the DPP to compute the value function and to learn the optimal stopping rule using backward induction. Both methods train neural networks to approximate optimal stopping policies. We demonstrate the effectiveness and the scalability of our work through numerical experiments on 6 different problems in spatial dimension up to 300. To the best of our knowledge, this is the first work to formalize and computationally solve MFOS in discrete time and finite space, opening new directions for scalable MAOS methods.

math.OC↗

arXiv2025

Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well-established performance of TRPO in the reinforcement learning (RL) setting, we extend its methodology to the MFG framework, leveraging its stability and robustness in policy optimization. Under standard assumptions in the MFG literature, we provide a rigorous analysis of MF-TRPO, establishing theoretical guarantees on its convergence. Our results cover both the exact formulation of the algorithm and its sample-based counterpart, where we derive high-probability guarantees and finite sample complexity. This work advances MFG optimization by bridging RL techniques with mean-field decision-making, offering a theoretically grounded approach to solving complex multi-agent problems.

stat.ML↗

arXiv2025

An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control

We present a novel on-policy algorithm for solving stochastic optimal control (SOC) problems. By leveraging the Girsanov theorem, our method directly computes on-policy gradients of the SOC objective without expensive backpropagation through stochastic differential equations or adjoint problem solutions. This approach significantly accelerates the optimization of neural network control policies while scaling efficiently to high-dimensional problems and long time horizons. We evaluate our method on classical SOC benchmarks as well as applications to sampling from unnormalized distributions via Schrödinger-Föllmer processes and fine-tuning pre-trained diffusion models. Experimental results demonstrate substantial improvements in both computational speed and memory efficiency compared to existing approaches.

cs.LG↗

arXiv2025

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

We investigate reinforcement learning in the setting of Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Applications include, for example, the control of a large number of robots communicating through a central unit dispatching the optimal policy computed by maximizing an aggregate reward. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states and actions of the other agents. We first provide a full analysis this discrete-time mean field control problem. We then rigorously prove the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting and establish bounds on the rates of convergence. We also provide graphical evidence of the convergence based on implementations of our algorithms.

math.OC↗

arXiv2025

Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective

We address in this paper Reinforcement Learning (RL) among agents that are grouped into teams such that there is cooperation within each team but general-sum (non-zero sum) competition across different teams. To develop an RL method that provably achieves a Nash equilibrium, we focus on a linear-quadratic structure. Moreover, to tackle the non-stationarity induced by multi-agent interactions in the finite population setting, we consider the case where the number of agents within each team is infinite, i.e., the mean-field setting. This results in a General-Sum LQ Mean-Field Type Game (GS-MFTG). We characterize the Nash equilibrium (NE) of the GS-MFTG, under a standard invertibility condition. This MFTG NE is then shown to be $O(1/M)$-NE for the finite population game where $M$ is a lower bound on the number of agents in each team. These structural results motivate an algorithm called Multi-player Receding-horizon Natural Policy Gradient (MRNPG), where each team minimizes its cumulative cost \emph{independently} in a receding-horizon manner. Despite the non-convexity of the problem, we establish that the resulting algorithm converges to a global NE through a novel problem decomposition into sub-problems using backward recursive discrete-time Hamilton-Jacobi-Isaacs (HJI) equations, in which \emph{independent natural policy gradient} is shown to exhibit linear convergence under time-independent diagonal dominance. Numerical studies included corroborate the theoretical results.

cs.LG↗