arXiv subjects

Alex Davey

Publications and source records attributed to Alex Davey.

7 recordsLinked to original sources

arXiv2025

PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities,particularly in environments with complex reward landscapes.

cs.AI↗

arXiv2025

StaQ: a Finite Memory Approach to Discrete Action Policy Mirror Descent

In Reinforcement Learning (RL), regularization with a Kullback-Leibler divergence that penalizes large deviations between successive policies has emerged as a popular tool both in theory and practice. This family of algorithms, often referred to as Policy Mirror Descent (PMD), has the property of averaging out policy evaluation errors which are bound to occur when using function approximators. However, exact PMD has remained a mostly theoretical framework, as its closed-form solution involves the sum of all past Q-functions which is generally intractable. A common practical approximation of PMD is to follow the natural policy gradient or use actor-critic approaches, but this potentially introduces errors in the policy update. In this paper, we propose and analyze PMD-like algorithms for discrete action spaces that only keep the last $M$ Q-functions in memory. We show theoretically that for a finite and large enough $M$, an RL algorithm can be derived that introduces no errors from the policy update, yet keeps the desirable PMD property of averaging out policy evaluation errors. Using an efficient GPU implementation, we then show empirically on medium-scale RL benchmarks such as MinAtar that increasing $M$ improves performance up to a certain threshold after which the performance becomes close to that of exact PMD, reinforcing the theoretical findings that using an infinite sum might be unnecessary and that keeping in memory the last M Q-functions is a practical and theoretically grounded implementation of PMD.

cs.LG↗

arXiv2024

Strong Cosmic Censorship in Kerr-Newman-de Sitter

Christodoulou's formulation of Strong Cosmic Censorship (SCC) holds true for Kerr-de Sitter black holes. On the other hand, Reissner-Nordstr\"om-de Sitter black holes violate SCC. We do a detailed scan of the parameter space of Kerr-Newman-de Sitter black holes between these two limiting families, to identify the boundary that marks the transition between solutions that respect and violate SCC. We focus our attention on linear scalar field perturbations. SCC is violated inside a (roughly) `spherical' shell of the parameter space of Kerr-Newman-de Sitter, centred at the corner that describes arbitrarily small extremal Reissner-Nordstr\"om-de Sitter solutions. Outside of this region, including the Kerr-de Sitter limit, we identify perturbation modes that decay slow enough to enforce SCC. Additionally, we do a necessary study of the quasinormal mode spectra of Kerr-Newman-de Sitter in some detail. As established in the literature, in the Kerr-de Sitter and Reissner-Nordstr\"om-de Sitter limits, we find three families of modes: de Sitter, photon sphere and near-horizon modes. These interact non-trivially away from the Reissner-Nordstr\"om-de Sitter limit and display eigenvalue repulsions like in Kerr-Newman black holes.

gr-qc↗

arXiv2023

Scalar QNM spectra of Kerr and Reissner-Nordstr\"om revealed by eigenvalue repulsions in Kerr-Newman

Recent studies of the gravito-electromagnetic frequency spectra of Kerr-Newman (KN) black holes have revealed two families of quasinormal modes (QNMs), namely photon sphere modes and near-horizon modes. However, they can only be unambiguously distinguished in the Reissner-Nordstr\"om (RN) limit, due to a phenomenon called eigenvalue repulsion (also known as level repulsion, avoided crossing or the Wigner-Teller effect), whereby the two families can interact strongly near extremality. We find that these features are also present in the QNM spectra of a scalar field in KN, where the perturbation modes are described by ODEs and thus easier to explore. Starting from the RN limit, we study how the scalar QNM spectra of KN dramatically changes as we vary the ratio of charge to angular momentum, all the way until the Kerr limit, while staying at a fixed distance from extremality. This scalar field case clarifies the (so far puzzling) relationship between the QNM spectra of RN and Kerr black holes and the nature of the eigenvalue repulsions in KN, that ultimately settle the fate of the QNM spectra in Kerr. We study not just the slowest-decaying QNMs (both for $\ell=m=0$ and $\ell=m=2$), but several sub-dominant overtones as well, as these turn out to play a crucial role understanding the KN QNM spectra. We also give a new high-order WKB expansion of KN QNMs that typically describes the photon sphere modes beyond the eikonal limit, and use a matched asymptotic expansion to get a very good approximation of the near-horizon modes near extremality.

gr-qc↗

arXiv2022

Strong Cosmic Censorship and Eigenvalue Repulsions for rotating de Sitter black holes in higher-dimensions

It has been established that Christodoulou's formulation of Strong Cosmic Censorship (SCC) is violated by Reissner-Nordstr\"om-de Sitter black holes, but holds in four-dimensional Kerr-de Sitter black holes. We show that SCC is also respected by equal angular momenta (cohomogeneity-1) Myers-Perry-de Sitter (MP-dS) in odd $d \ge 5$ spacetime dimensions. This suggests that the preservation of SCC in rotating backgrounds might be a universal property of Einstein gravity and not limited to the d = 4 Kerr-dS background. As required to discuss SCC in de Sitter spacetimes, we also study important aspects of the scalar field quasinormal mode (QNM) spectra of MP-dS. In particular, we find eigenvalue repulsions similar to those recently observed in the QNM spectra of asymptotically flat Kerr-Newman black holes. For axisymmetric modes (i.e. with azimuthal quantum number m = 0) there are three distinct families of QNM (de Sitter, photon sphere and near-horizon). However, typically, for non-axisymmetric ($m \ne 0$) QNMs, we find that the entire spectra can be described by just two families of QNM (since several overtone sections of the photon sphere and near-horizon families merge). For completeness, we also study the full scalar field QNM spectra of higher-dimensional Schwarzschild-de Sitter black holes.

gr-qc↗

arXiv2021

Phase diagram of the charged black hole bomb system

We find the phase diagram of solutions of the charged black hole bomb system. In particular, we find the static hairy black holes of Einstein-Maxwell-Scalar theory confined in a Minkowski box. We impose boundary conditions such that the scalar field vanishes at and outside a cavity of constant radius. These hairy black holes are asymptotically flat with a scalar condensate floating above the horizon. We identify four critical scalar charges which mark significant changes in the qualitative features of the phase diagram. When they coexist, hairy black holes always have higher entropy than the Reissner-Nordstr\"om black hole with the same quasilocal mass and charge. So hairy black holes are natural candidates for the endpoint of the superradiant/near-horizon instabilities of the black hole bomb system. We also relate hairy black holes to the boson stars of the theory. When it has a zero horizon radius limit, the hairy black hole family terminates on the boson star family. Finally, we find the Israel surface tensor of the box required to confine the scalar condensate and that it can obey suitable energy conditions.

gr-qc↗

arXiv2021

Fully constrained, high-resolution shock-capturing, formulation of the Einstein-fluid equations in $2+1$ dimensions

Four components of the axisymmetric Einstein equations in 2+1 dimensions with negative cosmological constant can be written as $\nabla_aM=\dots$ and $\nabla_aJ=\dots$, where the dots stand for stress-energy terms, and $M$ and $J$ are scalars. In vacuum, they reduce to the constant mass and angular momentum parameters of the BTZ solution of the same name. The integrability conditions for the Einstein equations give rise to two conserved stress-energy currents $\nabla_aj^a_{(M)}=0$ and $\nabla_aj^a_{(J)}=0$. The angular momentum current is just the Noether current due to axisymmetry, but the mass current is unexpected in the presence of rotation. The conserved quantity $M$ exists in all dimensions in spherical symmetry, known as the Misner-Sharp, Hawking or Kodama mass, but in 2+1 dimensions $M$ exists also in axisymmetry, even with rotation. We use $M$ and $J$ to give a fully constrained formulation of the axisymmetric Einstein equations in 2+1 dimensions, where the Einstein equations are solved by explicit integration from the center along time slices. We use the two conserved matter currents in the construction of a high-resolution shock-capturing formulation of the Einstein-perfect fluid system, in which $M$ and $J$ momentum are then exactly conserved by construction. We demonstrate convergence of the code in the test cases of generic dispersion and collapse and stable and unstable rotating stars.

gr-qc↗