arXiv subjects

Johannes Forkel

Publications and source records attributed to Johannes Forkel.

12 recordsLinked to original sources

arXiv2026

Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

AI agents deployed in real-world settings must be capable of coordinating with humans and other AI agents they have not encountered before. Zero-shot coordination (ZSC) algorithms aim to achieve this by specifying high-level learning rules such that independently engineered agents can coordinate with each other at test time. Rigorous evaluation of ZSC algorithms remains difficult: ideally, multiple independent implementations of each proposed algorithm must be used, reflecting the variation that arises when independent parties interpret and implement the same specification. In practice, however, ZSC algorithms have almost exclusively been evaluated using a single implementation trained across different random seeds, with only a handful of works additionally varying the neural network architecture. This leaves open questions about robustness to specification ambiguities and implementation details. In this work, we provide the first systematic evaluation of this robustness. We introduce a new evaluation scheme, cross-implementation cross-play, varying implementation details that prior work has shown to affect the performance of multi-agent reinforcement learning (MARL) algorithms, and we evaluate Other-Play, a popular ZSC algorithm, with this scheme. Our findings are encouraging and suggest that, for Other-Play, the standard ZSC evaluation is, in fact, a reasonable proxy for this more thorough cross-implementation evaluation.

cs.AI↗

arXiv2026

Expected Return Symmetries

Symmetry is an important inductive bias that can improve model robustness and generalization across many deep learning domains. In multi-agent settings, a priori known symmetries have been shown to address a fundamental coordination failure mode known as mutually incompatible symmetry breaking; e.g. in a game where two independent agents can choose to move "left'' or "right'', and where a reward of +1 or -1 is received when the agents choose the same action or different actions, respectively. However, the efficient and automatic discovery of environment symmetries, in particular for decentralized partially observable Markov decision processes, remains an open problem. Furthermore, environmental symmetry breaking constitutes only one type of coordination failure, which motivates the search for a more accessible and broader symmetry class. In this paper, we introduce such a broader group of previously unexplored symmetries, which we call expected return symmetries, which contains environment symmetries as a subgroup. We show that agents trained to be compatible under the group of expected return symmetries achieve better zero-shot coordination results than those using environment symmetries. As an additional benefit, our method makes minimal a priori assumptions about the structure of their environment and does not require access to ground truth symmetries.

cs.MA↗

arXiv2026

Recurrent Structural Policy Gradient for Partially Observable Mean Field Games

Mean Field Games (MFGs) provide a principled framework for modelling interactions in large population systems. However, algorithmic progress has been limited since model-free methods are high variance and exact methods scale poorly. Recent Hybrid Structural Methods (HSMs) reduce variance while maintaining tractability by leveraging low-dimensional individual state and action spaces and known transition dynamics to compute the exact expected return conditioned on Monte Carlo rollouts of common noise. However, HSMs have not been extended to partially observable settings. We propose Recurrent Structural Policy Gradient (RSPG), the first history-aware HSM for MFGs with public partial information. RSPG achieves an order-of-magnitude faster convergence than model-free RL methods while learning history-aware behaviour, unlike current HSMs. To facilitate research into MFGs, we also introduce MFAX, our JAX-based framework for MFGs that supports both analytic and sample-based mean-field updates. MFAX and usage examples can be found at https://clarisse-wibault.github.io/rspg/.

cs.AI↗

arXiv2026

Learning to Reason at the Frontier of Learnability

Reinforcement learning is now widely adopted as the final stage of large language model training, especially for reasoning-style tasks such as maths problems. Typically, models attempt each question many times during a single training step and attempt to learn from their successes and failures. However, we demonstrate that throughout training with two popular algorithms (PPO and VinePPO) on two widely used datasets, many questions are either solved by all attempts - meaning they are already learned - or by none - providing no meaningful training signal. To address this, we adapt a method from the reinforcement learning literature - sampling for learnability - and apply it to the reinforcement learning stage of LLM training. Our curriculum prioritises questions with high variance of success, i.e. those where the agent sometimes succeeds, but not always. Our findings demonstrate that this curriculum consistently boosts training performance across multiple algorithms and datasets, paving the way for more efficient and effective reinforcement learning with LLMs.

cs.LG↗

arXiv2025

High entropy leads to symmetry-equivariant policies in Dec-POMDPs

We prove that in any Dec-POMDP, sufficiently high entropy regularization ensures that the policy gradient flow with tabular softmax parametrization always converges, for any initialization, to the same joint policy, and that this joint policy is equivariant w.r.t. all symmetries of the Dec-POMDP. In particular, policies coming from different initializations will be fully compatible, in that their cross-play returns are equal to their self-play returns. Through extensive evaluation of independent PPO, arguably the standard baseline deep multi-agent policy gradient algorithm, in the Hanabi, Overcooked and Yokai environments, we find that the entropy coefficient has a massive influence on the cross-play returns between independently trained policies, and that the decrease in self-play returns coming from increased entropy regularization can often be counteracted by greedifying the learned policies after training. In Hanabi in particular we achieve a new SOTA in inter-seed cross-play this way. While we give examples of Dec-POMDPs in which one cannot learn the optimal symmetry-equivariant policy this way, both our theoretical and empirical results suggest that one should consider far higher entropy coefficients during hyperparameter sweeps in Dec-POMDPs than is typically done. Code for our experiments can be found at https://github.com/jforkel/JAX-OBL

cs.LG↗

arXiv2025

The Yokai Learning Environment: Tracking Beliefs Over Space and Time

The ability to cooperate with unknown partners is a central challenge in cooperative AI and widely studied in the form of zero-shot coordination (ZSC), which evaluates an algorithm by measuring the performance of independently trained agents when paired. The Hanabi Learning Environment (HLE) has become the dominant benchmark for ZSC, but recent work has achieved near-perfect inter-seed cross-play performance, limiting its ability to track algorithmic progress. We introduce the Yokai Learning Environment (YLE) - an open-source multi-agent RL benchmark in which effective collaboration requires building common ground by tracking and updating beliefs over moving cards, reasoning under ambiguous hints, and deciding when to terminate the game based on inferred shared knowledge - features absent in the HLE, where beliefs are tied to hand slots and hints are truthful by rule. We evaluate the leading ZSC methods, including High-Entropy IPPO, Other-Play, and Off-Belief Learning, which achieve near-perfect inter-seed cross-play in the HLE, and show that in the YLE they exhibit persistent SP-XP gaps, degraded early-ending calibration, and weaker belief representations in cross-play, indicating failure to maintain consistent internal models with unseen partners. Methods that perform best in the HLE do not perform best in the YLE, indicating that progress measured on a single benchmark may not generalise. Together, these results establish YLE as a challenging new ZSC benchmark.

cs.AI↗

arXiv2025

Ad-Hoc Human-AI Coordination Challenge

Achieving seamless coordination between AI agents and humans is crucial for real-world applications, yet it remains a significant open challenge. Hanabi is a cooperative card game featuring imperfect information, constrained communication, theory of mind requirements, and coordinated action -- making it an ideal testbed for human-AI coordination. However, its use for human-AI interaction has been limited by the challenges of human evaluation. In this work, we introduce the Ad-Hoc Human-AI Coordination Challenge (AH2AC2) to overcome the constraints of costly and difficult-to-reproduce human evaluations. We develop \textit{human proxy agents} on a large-scale human dataset that serve as robust, cheap, and reproducible human-like evaluation partners in AH2AC2. To encourage the development of data-efficient methods, we open-source a dataset of 3,079 games, deliberately limiting the amount of available human gameplay data. We present baseline results for both two- and three- player Hanabi scenarios. To ensure fair evaluation, we host the proxy agents through a controlled evaluation system rather than releasing them publicly. The code is available at \href{https://github.com/FLAIROx/ah2ac2}{https://github.com/FLAIROx/ah2ac2}.

cs.AI↗

arXiv2025

Fully Offline Reinforcement Learning

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

cs.LG↗

arXiv2023

Fisher-Hartwig Asymptotics and Log-Correlated Fields in Random Matrix Theory

This thesis is based on joint work with Jon Keating [FK21], Tom Claeys and Jon Keating [CFK23], and Isao Sauzedde [FS22], and is concerned with establishing and studying connections between random matrices and log-correlated fields. This is done with the help of formulae, including some newly established ones, for the asymptotics of Toeplitz, and Toeplitz+Hankel determinants with Fisher-Hartwig singularities. In Chapter 1, we give an introduction to the mathematical objects that we are interested in. In particular we explain the relations between the characteristic polynomial of random matrices, log-correlated fields, Gaussian multiplicative chaos, the moments of moments, and Toeplitz and Toeplitz+Hankel determinants with Fisher-Hartwig singularities. In Chapter 2 we use Riemann-Hilbert techniques to establish formulae for the asymptotics of Toeplitz, and Toeplitz+Hankel determinants with two complex conjugate pairs of merging Fisher-Hartwig singularities. In Chapter 3 we complete the connection between the classical compact groups and Gaussian multiplicative chaos, by showing that analogously to the case of the unitary group first established in [Web15], the characteristic polynomial of random orthogonal and symplectic matrices, when properly normalized, converges to a Gaussian multiplicative chaos measure on the unit circle. In Chapter 4 we compute the asymptotics of the moments of moments of random orthogonal and symplectic matrices, which can be expressed in terms of integrals over Toeplitz+Hankel determinants. The phase transitions we observe are in stark contrast to the ones proven for the unitary group in [Fah21]. In Chapter 5 we establish convergence in Sobolev spaces, of the logarithm of the characteristic polynomial of unitary Brownian motion to the Gaussian free field on the cylinder, thus proving the dynamical analogue of the classical stationary result in [HKO01].

math-ph↗

arXiv2023

Moments of Moments of the Characteristic Polynomials of Random Orthogonal and Symplectic Matrices

Using asymptotics of Toeplitz+Hankel determinants, we establish formulae for the asymptotics of the moments of the moments of the characteristic polynomials of random orthogonal and symplectic matrices, as the matrix-size tends to infinity. Our results are analogous to those that Fahs obtained for random unitary matrices in [14]. A key feature of the formulae we derive is that the phase transitions in the moments of moments are seen to depend on the symmetry group in question in a significant way.

math-ph↗

arXiv2022

Convergence of the logarithm of the characteristic polynomial of unitary Brownian motion in Sobolev space

We prove that the convergence of the real and imaginary parts of the logarithm of the characteristic polynomial of unitary Brownian motion toward Gaussian free fields on the cylinder, as the matrix dimension goes to infinity, holds in certain suitable Sobolev spaces, which we believe to be optimal. This is the natural dynamical analogue of the result for a fixed time by Hughes, Keating and O'Connell [1]. A weak kind of convergence is known since the work of Spohn [2], which was widely improved recently by Bourgade and Falconet [3]. In the course of this research we also proved a Wick-type identity, which we include in this paper, as it might be of independent interest.

math.PR↗

arXiv2022

The Classical Compact Groups and Gaussian Multiplicative Chaos

We consider powers of the absolute value of the characteristic polynomial of Haar distributed random orthogonal or symplectic matrices, as well as powers of the exponential of its argument, as a random measure on the unit circle minus small neighborhoods around $\pm 1$. We show that for small enough powers and under suitable normalization, as the matrix size goes to infinity, these random measures converge in distribution to a Gaussian multiplicative chaos measure. Our result is analogous to one on unitary matrices previously established by Christian Webb in [31]. We thus complete the connection between the classical compact groups and Gaussian multiplicative chaos. To prove this we establish appropriate asymptotic formulae for Toeplitz and Toeplitz+Hankel determinants with merging singularities. Using a recent formula communicated to us by Claeys et al., we are able to extend our result to the whole of the unit circle.

math-ph↗