arXiv subjects

Benjamin Black

Publications and source records attributed to Benjamin Black.

7 recordsLinked to original sources

arXiv2022

Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments

Visualizing optimization landscapes has led to many fundamental insights in numeric optimization, and novel improvements to optimization techniques. However, visualizations of the objective that reinforcement learning optimizes (the "reward surface") have only ever been generated for a small number of narrow contexts. This work presents reward surfaces and related visualizations of 27 of the most widely used reinforcement learning environments in Gym for the first time. We also explore reward surfaces in the policy gradient direction and show for the first time that many popular reinforcement learning environments have frequent "cliffs" (sudden large drops in expected return). We demonstrate that A2C often "dives off" these cliffs into low reward regions of the parameter space while PPO avoids them, confirming a popular intuition for PPO's improved performance over previous methods. We additionally introduce a highly extensible library that allows researchers to easily generate these visualizations in the future. Our findings provide new intuition to explain the successes and failures of modern RL methods, and our visualizations concretely characterize several failure modes of reinforcement learning agents in novel ways.

cs.LG↗

arXiv2021

A revised lower estimate of ozone columns during Earth's oxygenated history

The history of molecular oxygen (O$_2$) in Earth's atmosphere is still debated; however, geological evidence supports at least two major episodes where O$_2$ increased by an order of magnitude or more: the Great Oxidation Event (GOE) and the Neoproterozoic Oxidation Event. O$_2$ concentrations have likely fluctuated (between $10^{-3}$ and $1.5$ times the present atmospheric level) since the GOE $\sim 2.4$ Gyr ago, resulting in a time-varying ozone (O$_3$) layer. Using a three-dimensional chemistry-climate model, we simulate changes in O$_3$ in Earth's atmosphere since the GOE and consider the implications for surface habitability, and glaciation during the Mesoproterozoic. We find lower O$_3$ columns (reduced by up to $4.68$ times for a given O$_2$ level) compared to previous work; hence, higher fluxes of biologically harmful UV radiation would have reached the surface. Reduced O$_3$ leads to enhanced tropospheric production of the hydroxyl radical (OH) which then substantially reduces the lifetime of methane (CH$_4$). We show that a CH$_4$ supported greenhouse effect during the Mesoproterozoic is highly unlikely. The reduced O$_3$ columns we simulate have important implications for astrobiological and terrestrial habitability, demonstrating the relevance of three-dimensional chemistry-climate simulations when assessing paleoclimates and the habitability of faraway worlds.

astro-ph.EP↗

arXiv2020

PettingZoo: Gym for Multi-Agent Reinforcement Learning

This paper introduces the PettingZoo library and the accompanying Agent Environment Cycle ("AEC") games model. PettingZoo is a library of diverse sets of multi-agent environments with a universal, elegant Python API. PettingZoo was developed with the goal of accelerating research in Multi-Agent Reinforcement Learning ("MARL"), by making work more interchangeable, accessible and reproducible akin to what OpenAI's Gym library did for single-agent reinforcement learning. PettingZoo's API, while inheriting many features of Gym, is unique amongst MARL APIs in that it's based around the novel AEC games model. We argue, in part through case studies on major problems in popular MARL environments, that the popular game models are poor conceptual models of games commonly used in MARL and accordingly can promote confusing bugs that are hard to detect, and that the AEC games model addresses these problems.

cs.LG↗

arXiv2020

Agent Environment Cycle Games

Partially Observable Stochastic Games (POSGs) are the most general and common model of games used in Multi-Agent Reinforcement Learning (MARL). We argue that the POSG model is conceptually ill suited to software MARL environments, and offer case studies from the literature where this mismatch has led to severely unexpected behavior. In response to this, we introduce the Agent Environment Cycle Games (AEC Games) model, which is more representative of software implementation. We then prove it's as an equivalent model to POSGs. The AEC games model is also uniquely useful in that it can elegantly represent both all forms of MARL environments, whereas for example POSGs cannot elegantly represent strictly turn based games like chess.

cs.LG↗

arXiv2020

Multiplayer Support for the Arcade Learning Environment

The Arcade Learning Environment ("ALE") is a widely used library in the reinforcement learning community that allows easy programmatic interfacing with Atari 2600 games, via the Stella emulator. We introduce a publicly available extension to the ALE that extends its support to multiplayer games and game modes. This interface is additionally integrated with PettingZoo to allow for a simple Gym-like interface in Python to interact with these games. We additionally introduce experimental baselines for all environments included.

cs.LG↗

arXiv2020

SuperSuit: Simple Microwrappers for Reinforcement Learning Environments

In reinforcement learning, wrappers are universally used to transform the information that passes between a model and an environment. Despite their ubiquity, no library exists with reasonable implementations of all popular preprocessing methods. This leads to unnecessary bugs, code inefficiencies, and wasted developer time. Accordingly we introduce SuperSuit, a Python library that includes all popular wrappers, and wrappers that can easily apply lambda functions to the observations/actions/reward. It's compatible with the standard Gym environment specification, as well as the PettingZoo specification for multi-agent environments. The library is available at https://github.com/PettingZoo-Team/SuperSuit,and can be installed via pip.

cs.LG↗

arXiv2020

Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning

Parameter sharing, where each agent independently learns a policy with fully shared parameters between all policies, is a popular baseline method for multi-agent deep reinforcement learning. Unfortunately, since all agents share the same policy network, they cannot learn different policies or tasks. This issue has been circumvented experimentally by adding an agent-specific indicator signal to observations, which we term "agent indication". Agent indication is limited, however, in that without modification it does not allow parameter sharing to be applied to environments where the action spaces and/or observation spaces are heterogeneous. This work formalizes the notion of agent indication and proves that it enables convergence to optimal policies for the first time. Next, we formally introduce methods to extend parameter sharing to learning in heterogeneous observation and action spaces, and prove that these methods allow for convergence to optimal policies. Finally, we experimentally confirm that the methods we introduce function empirically, and conduct a wide array of experiments studying the empirical efficacy of many different agent indication schemes for image based observation spaces.

cs.LG↗