arXiv subjects

Edward Hughes

Publications and source records attributed to Edward Hughes.

43 records · Page 3Linked to original sources

arXiv2019

Evolving intrinsic motivations for altruistic behavior

Multi-agent cooperation is an important feature of the natural world. Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate. Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory. Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp. By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way. To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection. We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.

cs.MA↗

arXiv2019

Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research

Evolution has produced a multi-scale mosaic of interacting adaptive units. Innovations arise when perturbations push parts of the system away from stable equilibria into new regimes where previously well-adapted solutions no longer work. Here we explore the hypothesis that multi-agent systems sometimes display intrinsic dynamics arising from competition and cooperation that provide a naturally emergent curriculum, which we term an autocurriculum. The solution of one social task often begets new social tasks, continually generating novel challenges, and thereby promoting innovation. Under certain conditions these challenges may become increasingly complex over time, demanding that agents accumulate ever more innovations.

cs.AI↗

arXiv2019

Malthusian Reinforcement Learning

Here we explore a new algorithmic framework for multi-agent reinforcement learning, called Malthusian reinforcement learning, which extends self-play to include fitness-linked population size dynamics that drive ongoing innovation. In Malthusian RL, increases in a subpopulation's average return drive subsequent increases in its size, just as Thomas Malthus argued in 1798 was the relationship between preindustrial income levels and population growth. Malthusian reinforcement learning harnesses the competitive pressures arising from growing and shrinking population size to drive agents to explore regions of state and policy spaces that they could not otherwise reach. Furthermore, in environments where there are potential gains from specialization and division of labor, we show that Malthusian reinforcement learning is better positioned to take advantage of such synergies than algorithms based on self-play.

cs.NE↗

arXiv2019

Causal Reasoning from Meta-reinforcement Learning

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal causal reasoning algorithms also exist, in this paper we show that such reasoning can arise from model-free reinforcement learning, and suggest that causal reasoning in complex settings may benefit from the more end-to-end learning-based approaches presented here. This work also offers new strategies for structured exploration in reinforcement learning, by providing agents with the ability to perform -- and interpret -- experiments.

cs.LG↗

arXiv2018

Inequity aversion improves cooperation in intertemporal social dilemmas

Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.

cs.NE↗

arXiv2016

The connected prescription for form factors in twistor space

We propose a connected prescription formula in twistor space for all tree-level form factors of the stress tensor multiplet operator in $\mathcal{N}=4$ super Yang-Mills, which is a generalisation of the expression of Roiban, Spradlin and Volovich for superamplitudes. By introducing link variables, we show that our formula is identical to the recently proposed four-dimensional scattering equations for form factors. Similarly to the case of amplitudes, the link representation of form factors is shown to be directly related to BCFW recursion relations, and is considerably more tractable than the scattering equations. We also discuss how our results are related to a recent Grassmannian formulation of form factors, and comment on a possible derivation of our formula from ambitwistor strings.

hep-th↗

arXiv2015

One-Loop Soft Theorems via Dual Superconformal Symmetry

We study soft theorems at one loop in planar N=4 super Yang-Mills theory through finite order in the infrared regulator and to subleading order in the soft parameter δ. In particular, we derive a universal constraint from dual superconformal symmetry, which we use to bootstrap subleading log δ behaviour. Moreover, we determine the complete infrared-finite subleading soft contribution of n-point MHV amplitudes using momentum twistors. Finally, we compute the subleading log δ behaviour of one-loop NMHV ratio functions at six and seven points, finding that universality holds within but not between helicity sectors.

hep-th↗