arXiv subjects

Wolfram Barfuss

Publications and source records attributed to Wolfram Barfuss.

13 recordsLinked to original sources

arXiv2026

Dynamics of Multi-Agent Actor-Critic Learning in Stochastic Games: from Multistability and Chaos to Stable Cooperation

Achieving robust coordination and cooperation is a central challenge in multi-agent reinforcement learning (MARL). Uncovering the mechanisms underlying such emergent behaviors calls for a dynamical understanding of learn processes. In this work, we investigate the dynamics of actor-critic agents in stochastic games, focusing on the impact of entropy regularization. By leveraging time-scale separation, we derive the system's evolution equations, which are then formally analyzed using dynamical systems theory. We find that in the constant-sum game of Matching Pennies, the system exhibits chaotic behavior. Entropy regularization mitigates this chaos and drives the dynamics toward convergence to fair cooperation. In contrast, in the general-sum game of the Prisoner's Dilemma, the system displays multistability. Interestingly, the three stable equilibria of the system correspond to the well-known ALLC (Always Cooperate), ALLD (Always Defect), and GRIM (Grim Trigger) strategies from evolutionary game theory (EGT). Entropy regularization strengthens system resilience by enlarging the basin of attraction of the cooperative equilibrium. Our findings reveal a close link between the mechanism of direct reciprocity in EGT and how cooperation emerges in MARL, offering insights for designing more robust and collaborative multi-agent systems.

physics.soc-ph↗

arXiv2025

AI for a Planet Under Pressure

Artificial intelligence (AI) is already driving scientific breakthroughs in a variety of research fields, ranging from the life sciences to mathematics. This raises a critical question: can AI be applied both responsibly and effectively to address complex and interconnected sustainability challenges? This report is the result of a collaboration between the Stockholm resilience Centre (Stockholm University), the Potsdam Institute for Climate Impact Research (PIK), and Google DeepMind. Our work explores the potential and limitations of using AI as a research method to help tackle eight broad sustainability challenges. The results build on iterated expert dialogues and assessments, a systematic AI-supported literature overview including over 8,500 academic publications, and expert deep-dives into eight specific issue areas. The report also includes recommendations to sustainability scientists, research funders, the private sector, and philanthropies.

cs.CY↗

arXiv2025

Multi-Agent Risks from Advanced AI

The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.

cs.MA↗

arXiv2024

Deterministic Model of Incremental Multi-Agent Boltzmann Q-Learning: Transient Cooperation, Metastability, and Oscillations

Multi-Agent Reinforcement Learning involves agents that learn together in a shared environment, leading to emergent dynamics sensitive to initial conditions and parameter variations. A Dynamical Systems approach, which studies the evolution of multi-component systems over time, has uncovered some of the underlying dynamics by constructing deterministic approximation models of stochastic algorithms. In this work, we demonstrate that even in the simplest case of independent Q-learning with a Boltzmann exploration policy, significant discrepancies arise between the actual algorithm and previous approximations. We elaborate why these models actually approximate interesting variants rather than the original incremental algorithm. To explain the discrepancies, we introduce a new discrete-time approximation model that explicitly accounts for agents' update frequencies within the learning process and show that its dynamics fundamentally differ from the simplified dynamics of prior models. We illustrate the usefulness of our approach by applying it to the question of spontaneous cooperation in social dilemmas, specifically the Prisoner's Dilemma as the simplest case study. We identify conditions under which the learning behaviour appears as long-term stable cooperation from an external perspective. However, our model shows that this behaviour is merely a metastable transient phase and not a true equilibrium, making it exploitable. We further exemplify how specific parameter settings can significantly exacerbate the moving target problem in independent learning. Through a systematic analysis of our model, we show that increasing the discount factor induces oscillations, preventing convergence to a joint policy. These oscillations arise from a supercritical Neimark-Sacker bifurcation, which transforms the unique stable fixed point into an unstable focus surrounded by a stable limit cycle.

cs.MA↗

arXiv2023

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems like the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges, and give perspectives on future research directions, looking to inspire interdisciplinary approaches.

nlin.AO↗

arXiv2023

Intrinsic fluctuations of reinforcement learning promote cooperation

In this work, we ask for and answer what makes classical temporal-difference reinforcement learning with epsilon-greedy strategies cooperative. Cooperating in social dilemma situations is vital for animals, humans, and machines. While evolutionary theory revealed a range of mechanisms promoting cooperation, the conditions under which agents learn to cooperate are contested. Here, we demonstrate which and how individual elements of the multi-agent learning setting lead to cooperation. We use the iterated Prisoner's dilemma with one-period memory as a testbed. Each of the two learning agents learns a strategy that conditions the following action choices on both agents' action choices of the last round. We find that next to a high caring for future rewards, a low exploration rate, and a small learning rate, it is primarily intrinsic stochastic fluctuations of the reinforcement learning process which double the final rate of cooperation to up to 80%. Thus, inherent noise is not a necessary evil of the iterative learning process. It is a critical asset for the learning of cooperation. However, we also point out the trade-off between a high likelihood of cooperative behavior and achieving this in a reasonable amount of time. Our findings are relevant for purposefully designing cooperative algorithms and regulating undesired collusive effects.

cs.LG↗

arXiv2022

Modeling the effects of environmental and perceptual uncertainty using deterministic reinforcement learning dynamics with partial observability

Assessing the systemic effects of uncertainty that arises from agents' partial observation of the true states of the world is critical for understanding a wide range of scenarios. Yet, previous modeling work on agent learning and decision-making either lacks a systematic way to describe this source of uncertainty or puts the focus on obtaining optimal policies using complex models of the world that would impose an unrealistically high cognitive demand on real agents. In this work we aim to efficiently describe the emergent behavior of biologically plausible and parsimonious learning agents faced with partially observable worlds. Therefore we derive and present deterministic reinforcement learning dynamics where the agents observe the true state of the environment only partially. We showcase the broad applicability of our dynamics across different classes of partially observable agent-environment systems. We find that partial observability creates unintuitive benefits in a number of specific contexts, pointing the way to further research on a general understanding of such effects. For instance, partially observant agents can learn better outcomes faster, in a more stable way and even overcome social dilemmas. Furthermore, our method allows the application of dynamical systems theory to partially observable multiagent leaning. In this regard we find the emergence of catastrophic limit cycles, a critical slowing down of the learning processes between reward regimes and the separation of the learning dynamics into fast and slow directions, all caused by partial observability. Therefore, the presented dynamics have the potential to become a formal, yet practical, lightweight and robust tool for researchers in biology, social science and machine learning to systematically investigate the effects of interacting partially observant agents.

nlin.AO↗

arXiv2022

Conceptualizing World-Earth System resilience: Exploring transformation pathways towards a safe and just operating space for humanity

We develop a framework within which to conceptualize World-Earth System resilience. Our notion of World-Earth System resilience emphasizes the need to move beyond the basin of attraction notion of resilience as we are not in a basin we can stay in. We are on a trajectory to a new basin and we have to avoid falling into undesirable basins. We thus focus on `pathway resilience', i.e. the relative number of paths that allow us to move from the transitional operating space we occupy now as we leave the Holocene basin to a safe and just operating space in the Anthropocene. We develop a mathematical model to formalize this conceptualization and demonstrate how interactions between earth system resilience (biophysical processes) and world system resilience (social processes) impact pathway resilience. Our findings show that building earth system resilience is probably our only chance to reach a safe and just operating space. We also illustrate the importance of world system dynamics by showing how the notion of fairness coupled with regional inequality affects pathway resilience.

physics.soc-ph↗

arXiv2019

The physics of governance networks: critical transitions in contagion dynamics on multilayer adaptive networks with application to the sustainable use of renewable resources

Adaptive networks are a versatile approach to model phenomena such as contagion and spreading dynamics, critical transitions and structure formation that emerge from the dynamic coevolution of complex network structure and node states. Here, we study critical transitions in contagion dynamics on multilayer adaptive networks with dynamic node states and present an application to the governance of sustainable resource use. We focus on a three layer adaptive network model, where a polycentric governance network interacts with a social network of resource users which in turn interacts with an ecological network of renewable resources. We uncover that sustainability is favored for slow interaction timescales, large homophilic network adaptation rate (as long it is below the fragmentation threshold) and high taxation rates. Interestingly, we also observe a trade-off between an eco-dictatorship (reduced model with a single governance actor that always taxes unsustainable resource use) and the polycentric governance network of multiple actors. In the latter setup, sustainability is enhanced for low but hindered for high tax rates compared to the eco-dictatorship case. These results highlight mechanisms generating emergent critical transitions in contagion dynamics on multilayer adaptive network and show how these can be understood and approximated analytically, relevant for understanding complex adaptive systems from various disciplines ranging from physics and epidemiology to sociology and global sustainability science. The paper also provides insights into potential critical intervention points for policy in the form of taxes in the governance of sustainable renewable resource use that can inform more process-detailed social-ecological modeling.

physics.soc-ph↗

arXiv2019

Earth system modeling with endogenous and dynamic human societies: the copan:CORE open World-Earth modeling framework

Analysis of Earth system dynamics in the Anthropocene requires to explicitly take into account the increasing magnitude of processes operating in human societies, their cultures, economies and technosphere and their growing feedback entanglement with those in the physical, chemical and biological systems of the planet. However, current state-of-the-art Earth System Models do not represent dynamic human societies and their feedback interactions with the biogeophysical Earth system and macroeconomic Integrated Assessment Models typically do so only with limited scope. This paper (i) proposes design principles for constructing World-Earth Models (WEM) for Earth system analysis of the Anthropocene, i.e., models of social (World) - ecological (Earth) co-evolution on up to planetary scales, and (ii) presents the copan:CORE open simulation modeling framework for developing, composing and analyzing such WEMs based on the proposed principles. The framework provides a modular structure to flexibly construct and study WEMs. These can contain biophysical (e.g. carbon cycle dynamics), socio-metabolic/economic (e.g. economic growth) and socio-cultural processes (e.g. voting on climate policies or changing social norms) and their feedback interactions, and are based on elementary entity types, e.g., grid cells and social systems. Thereby, copan:CORE enables the epistemic flexibility needed for contributions towards Earth system analysis of the Anthropocene given the large diversity of competing theories and methodologies used for describing socio-metabolic/economic and socio-cultural processes in the Earth system by various fields and schools of thought. To illustrate the capabilities of the framework, we present an exemplary and highly stylized WEM implemented in copan:CORE that illustrates how endogenizing socio-cultural processes and feedbacks could fundamentally change macroscopic model outcomes.

physics.soc-ph↗

arXiv2019

Deep reinforcement learning in World-Earth system models to discover sustainable management strategies

Increasingly complex, non-linear World-Earth system models are used for describing the dynamics of the biophysical Earth system and the socio-economic and socio-cultural World of human societies and their interactions. Identifying pathways towards a sustainable future in these models for informing policy makers and the wider public, e.g. pathways leading to a robust mitigation of dangerous anthropogenic climate change, is a challenging and widely investigated task in the field of climate research and broader Earth system science. This problem is particularly difficult when constraints on avoiding transgressions of planetary boundaries and social foundations need to be taken into account. In this work, we propose to combine recently developed machine learning techniques, namely deep reinforcement learning (DRL), with classical analysis of trajectories in the World-Earth system. Based on the concept of the agent-environment interface, we develop an agent that is generally able to act and learn in variable manageable environment models of the Earth system. We demonstrate the potential of our framework by applying DRL algorithms to two stylized World-Earth system models. Conceptually, we explore thereby the feasibility of finding novel global governance policies leading into a safe and just operating space constrained by certain planetary and socio-economic boundaries. The artificially intelligent agent learns that the timing of a specific mix of taxing carbon emissions and subsidies on renewables is of crucial relevance for finding World-Earth system trajectories that are sustainable on the long term.

physics.soc-ph↗

arXiv2019

Deterministic limit of temporal difference reinforcement learning for stochastic games

Reinforcement learning in multiagent systems has been studied in the fields of economic game theory, artificial intelligence and statistical physics by developing an analytical understanding of the learning dynamics (often in relation to the replicator dynamics of evolutionary game theory). However, the majority of these analytical studies focuses on repeated normal form games, which only have a single environmental state. Environmental dynamics, i.e., changes in the state of an environment affecting the agents' payoffs has received less attention, lacking a universal method to obtain deterministic equations from established multistate reinforcement learning algorithms. In this work we present a novel methodological extension, separating the interaction from the adaptation time scale, to derive the deterministic limit of a general class of reinforcement learning algorithms, called temporal difference learning. This form of learning is equipped to function in more realistic multistate environments by using the estimated value of future environmental states to adapt the agent's behavior. We demonstrate the potential of our method with the three well established learning algorithms Q learning, SARSA learning and Actor-Critic learning. Illustrations of their dynamics on two multiagent, multistate environments reveal a wide range of different dynamical regimes, such as convergence to fixed points, limit cycles, and even deterministic chaos.

cs.MA↗

arXiv2016

Parsimonious modeling with Information Filtering Networks

We introduce a methodology to construct parsimonious probabilistic models. This method makes use of Information Filtering Networks to produce a robust estimate of the global sparse inverse covariance from a simple sum of local inverse covariances computed on small sub-parts of the network. Being based on local and low-dimensional inversions, this method is computationally very efficient and statistically robust even for the estimation of inverse covariance of high-dimensional, noisy and short time-series. Applied to financial data our method results computationally more efficient than state-of-the-art methodologies such as Glasso producing, in a fraction of the computation time, models that can have equivalent or better performances but with a sparser inference structure. We also discuss performances with sparse factor models where we notice that relative performances decrease with the number of factors. The local nature of this approach allows us to perform computations in parallel and provides a tool for dynamical adaptation by partial updating when the properties of some variables change without the need of recomputing the whole model. This makes this approach particularly suitable to handle big datasets with large numbers of variables. Examples of practical application for forecasting, stress testing and risk allocation in financial systems are also provided.

cs.IT↗