arXiv subjects

Andrew Patterson

Publications and source records attributed to Andrew Patterson.

At least 19 recordsLinked to original sources

arXiv2026

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.

cs.LG↗

arXiv2026

Forager: a lightweight testbed for continual learning with partial observability in RL

In continual reinforcement learning (CRL), good performance requires never-ending learning, acting, and exploration in a big, partially observable world. Most CRL experiments have focused on loss of plasticity -- the inability to keep learning -- in one-off experiments where some unobservable non-stationarity is added to classic fully observable MDPs. Further, these experiments rarely consider the role of partial observability and the importance of CRL agents that use memory or recurrence. One potential reason for this focus on mitigating loss of plasticity without considering partial observability is that many partially-observable CRL environments are prohibitively expensive. In this paper, we introduce Forager, a light-weight partially-observable CRL environment with a constant memory footprint. We provide a set of experiments and sample tasks demonstrating that Forager is challenging for current CRL agents and yet also allows for in-depth study of those agents. We demonstrate that agents exhibit loss of plasticity, proposed mitigations can help, but that most useful is to leverage state construction. We conclude with a variant of Forager that generates an unending stream of new tasks to learn that clearly highlights the limitations of current CRL agents.

cs.LG↗

arXiv2025

Real-Time Optimal Control via Transformer Networks and Bernstein Polynomials

In this paper, we propose a Transformer-based framework for approximating solutions to infinite-dimensional optimization problems: calculus of variations problems and optimal control problems. Our approach leverages offline training on data generated by solving a sample of infinite- dimensional optimization problems using composite Bernstein collocation. Once trained, the Transformer efficiently generates near-optimal, feasible trajectories, making it well-suited for real-time applications. In motion planning for autonomous vehicles, for instance, these trajectories can serve to warm- start optimal motion planners or undergo rigorous evaluation to ensure safety. We demonstrate the effectiveness of this method through numerical results on a classical control problem and an online obstacle avoidance task. This data-driven approach offers a promising solution for real-time optimal control of nonlinear, nonconvex systems.

math.OC↗

arXiv2025

Matrix inversion polynomials for the quantum singular value transformation

Quantum matrix inversion with the quantum singular value transformation (QSVT) requires a polynomial approximation to $1/x$. Several methods from the literature construct polynomials that achieve the known degree complexity $\mathcal{O}(\kappa\log(\kappa/\varepsilon))$ with condition number $\kappa$ and uniform error $\varepsilon$. However, the \emph{optimal} polynomial with lowest degree for fixed error $\varepsilon$ can only be approximated numerically with the resource-intensive Remez method, leading to impractical preprocessing runtimes. Here, we derive an analytic shortcut to the optimal polynomial. Comparisons with other polynomials from the literature, based on Taylor expansion, Chebyshev iteration, and convex optimization, confirm that our result is optimal. Furthermore, for large $\kappa\log(\kappa/\varepsilon)$, our polynomial has the smallest maximum value on $[-1,1]$ of all approaches considered, leading to reduced circuit depth due to the normalization condition of QSVT. With the Python code provided, this paper will also be useful for practitioners in the field.

quant-ph↗

arXiv2025

Simulating plasma wave propagation on a superconducting quantum chip

Quantum computers may one day enable the efficient simulation of strongly coupled plasmas that lie beyond the reach of classical computation in regimes where quantum effects are important and the scale separation is large. In this article, we take a first step toward efficient simulation of quantum plasmas by demonstrating linear plasma wave propagation on a superconducting quantum chip. Using high-fidelity and highly expressive device-native gates, combined with an error-mitigation technique, we simulate the scattering of laser pulses from inhomogeneous plasmas. Our approach is made feasible by the identification of a suitable local spin model whose excitations mimic plasma waves, and whose circuit implementation requires a lower gate count than other proposed approaches that would require a future fault-tolerant quantum computer. This work opens avenues to study more complicated phenomena that cannot be simulated efficiently on classical computers, such as nonlinear quantum dynamics when strongly coupled plasmas are driven out of equilibrium.

quant-ph↗

arXiv2025

Deep Reinforcement Learning with Gradient Eligibility Traces

Achieving fast and stable off-policy learning in deep reinforcement learning (RL) is challenging. Most existing methods rely on semi-gradient temporal-difference (TD) methods for their simplicity and efficiency, but are consequently susceptible to divergence. While more principled approaches like Gradient TD (GTD) methods have strong convergence guarantees, they have rarely been used in deep RL. Recent work introduced the generalized Projected Bellman Error ($\overline{\text{PBE}}$), enabling GTD methods to work efficiently with nonlinear function approximation. However, this work is limited to one-step methods, which are slow at credit assignment and require a large number of samples. In this paper, we extend the generalized $\overline{\text{PBE}}$ objective to support multistep credit assignment based on the $\lambda$-return and derive three gradient-based methods that optimize this new objective. We provide both a forward-view formulation compatible with experience replay and a backward-view formulation compatible with streaming algorithms. Finally, we evaluate the proposed algorithms and show that they outperform both PPO and StreamQ in MuJoCo and MinAtar environments, respectively. Code available at https://github.com/esraaelelimy/gtd\_algos

cs.LG↗

arXiv2025

The Cell Must Go On: Agar.io for Continual Reinforcement Learning

Continual reinforcement learning (RL) concerns agents that are expected to learn continually, rather than converge to a policy that is then fixed for evaluation. This setting is well-suited to environments that the agent perceives as changing over time, rendering any static policy ineffective. In continual RL, researchers often simulate such changes either by modifying episodic environments to incorporate task shifts during interaction or by designing simulators that explicitly model continual dynamics. However, transforming episodic problems into continual ones primarily captures scenarios involving abrupt changes in the data stream and still relies on episodic structure. Meanwhile, the few simulators explicitly designed for empirical continual RL research are often limited in scope or complexity. In this paper, we introduce AgarCL, a research platform for continual RL that enables agents to progress toward increasingly sophisticated behaviour. AgarCL is based on the game Agar.io, a non-episodic, high-dimensional problem with stochastic, ever-evolving dynamics, continuous actions, and partial observability. We provide benchmark results for DQN, PPO, and SAC on the primary continual RL challenge, as well as across a suite of smaller tasks within AgarCL. These smaller tasks isolate aspects of the full environment and allow us to characterize the distinct challenges posed by different components of the game. We further evaluate three continual learning methods-Shrink and Perturb, ReDo, and Continual Backpropagation-and observe little improvement over standard RL algorithms, suggesting that the challenges posed by AgarCL extend beyond the stability-plasticity dilemma.

cs.LG↗

arXiv2024

Measurement Schemes for Quantum Linear Equation Solvers

Solving Computational Fluid Dynamics (CFD) problems requires the inversion of a linear system of equations, which can be done using a quantum algorithm for matrix inversion arxiv:1806.01838. However, the number of shots required to measure the output of the system can be prohibitive and remove any advantage obtained by quantum computing. In this work we propose a scheme for measuring the output of QSVT matrix inversion algorithms specifically for the CFD use case. We use a Quantum Signal Processing (QSP) based amplitude estimation algorithm arxiv:2207.08628 and show how it can be combined with the QSVT matrix inversion algorithm. We perform a detailed resource estimation of the amount of computational resources required for a single iteration of amplitude estimation, and compare the costs of amplitude estimation with the cost of not doing amplitude estimation and measuring the whole wavefunction. We also propose a measurement scheme to reduce the number of amplitudes measured in the CFD example by focusing on large amplitudes only. We simulate the whole CFD loop, finding that thus measuring only a small number of the total amplitudes in the output vector still results in an acceptable level of overall error.

quant-ph↗

arXiv2024

Quantum Computation of Electronic Structure with Projector Augmented-Wave Method and Plane Wave Basis Set

Quantum simulation of materials is a promising application area of quantum computers. To practically realize this promise, we must reduce quantum resources while maintaining accuracy. In electronic structure calculations on classical computers, resource reduction has been achieved by using the projector augmented-wave method (PAW) and plane wave basis sets. However, the PAW method generalized for many-body states introduces non-orthogonality effects which impede its direct application to quantum computing. In this work, we develop a unitary variant of the PAW (UPAW) that preserves the orthogonality constraints. We provide a linear-combination-of-unitaries decomposition of the UPAW Hamiltonian to enable ground state estimation using qubitized quantum phase estimation. Additionally, we further improve algorithmic efficiency by extending classical down-sampling techniques into the quantum setting. We then estimate quantum resources for crystalline solids to estimate the energy within chemical accuracy with respect to the full basis set limit, and also consider a supercell approach which is more suitable for calculations of defect states. We provide the quantum resources for energy estimation of a nitrogen-vacancy defect centre in diamond which is a challenging system for classical algorithms and a quintessential problem in the studies of quantum point defects.

quant-ph↗

arXiv2024

The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

This paper introduces a new empirical methodology, the Cross-environment Hyperparameter Setting Benchmark, that compares RL algorithms across environments using a single hyperparameter setting, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that this benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using few samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at low cost. We demonstrate two example instantiations of the CHS, on a set of six small control environments (SC-CHS) and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to illustrate the applicability of the CHS to modern RL algorithms on challenging environments, we conduct a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm on the DMC-CHS.

cs.LG↗

arXiv2024

Optimal Control using Composite Bernstein Approximants

In this work, we present composite Bernstein polynomials as a direct collocation method for approximating optimal control problems. An analysis of the convergence properties of composite Bernstein polynomials is provided, and beneficial properties of composite Bernstein polynomials for the solution of optimal control problems are discussed. The efficacy of the proposed approximation method is demonstrated through a bang-bang example. Lastly, we apply this method to a motion planning problem, offering a practical solution that emphasizes the ability of this method to solve complex optimal control problems.

math.OC↗

arXiv2024

Investigating the Interplay of Prioritized Replay and Generalization

Experience replay, the reuse of past data to improve sample efficiency, is ubiquitous in reinforcement learning. Though a variety of smart sampling schemes have been introduced to improve performance, uniform sampling by far remains the most common approach. One exception is Prioritized Experience Replay (PER), where sampling is done proportionally to TD errors, inspired by the success of prioritized sweeping in dynamic programming. The original work on PER showed improvements in Atari, but follow-up results were mixed. In this paper, we investigate several variations on PER, to attempt to understand where and when PER may be useful. Our findings in prediction tasks reveal that while PER can improve value propagation in tabular settings, behavior is significantly different when combined with neural networks. Certain mitigations $-$ like delaying target network updates to control generalization and using estimates of expected TD errors in PER to avoid chasing stochasticity $-$ can avoid large spikes in error with PER and neural networks but generally do not outperform uniform replay. In control tasks, none of the prioritized variants consistently outperform uniform replay. We present new insight into the interaction between prioritization, bootstrapping, and neural networks and propose several improvements for PER in tabular settings and noisy domains.

cs.LG↗

arXiv2023

When is Offline Policy Selection Sample Efficient for Reinforcement Learning?

Offline reinforcement learning algorithms often require careful hyperparameter tuning. Before deployment, we need to select amongst a set of candidate policies. However, there is limited understanding about the fundamental limits of this offline policy selection (OPS) problem. In this work we provide clarity on when sample efficient OPS is possible, primarily by connecting OPS to off-policy policy evaluation (OPE) and Bellman error (BE) estimation. We first show a hardness result, that in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to OPS. As a result, no OPS method can be more sample efficient than OPE in the worst case. We then connect BE estimation to the OPS problem, showing how BE can be used as a tool for OPS. While BE-based methods generally require stronger requirements than OPE, when those conditions are met they can be more sample efficient. Building on this insight, we propose a BE method for OPS, called Identifiable BE Selection (IBES), that has a straightforward method for selecting its own hyperparameters. We conclude with an empirical study comparing OPE and IBES, and by showing the difficulty of OPS on an offline Atari benchmark dataset.

cs.LG↗

arXiv2023

Empirical Design in Reinforcement Learning

Empirical design in reinforcement learning is no small task. Running good experiments requires attention to detail and at times significant computational resources. While compute resources available per dollar have continued to grow rapidly, so have the scale of typical experiments in reinforcement learning. It is now common to benchmark agents with millions of parameters against dozens of tasks, each using the equivalent of 30 days of experience. The scale of these experiments often conflict with the need for proper statistical evidence, especially when comparing algorithms. Recent studies have highlighted how popular algorithms are sensitive to hyper-parameter settings and implementation details, and that common empirical practice leads to weak statistical evidence (Machado et al., 2018; Henderson et al., 2018). Here we take this one step further. This manuscript represents both a call to action, and a comprehensive resource for how to do good experiments in reinforcement learning. In particular, we cover: the statistical assumptions underlying common performance measures, how to properly characterize performance variation and stability, hypothesis testing, special considerations for comparing multiple agents, baseline and illustrative example construction, and how to deal with hyper-parameters and experimenter bias. Throughout we highlight common mistakes found in the literature and the statistical consequences of those in example experiments. The objective of this document is to provide answers on how we can use our unprecedented compute to do good science in reinforcement learning, as well as stay alert to potential pitfalls in our empirical design.

cs.LG↗

arXiv2023

Development and Demonstration of an Efficient Readout Error Mitigation Technique for use in NISQ Algorithms

The approximate state estimation and the closely related classical shadows methods allow for the estimation of complicated observables with relatively few shots. As these methods make use of random measurements that can symmetrise the effect of readout errors, they have been shown to permit simplified approaches to readout error mitigation which require only a number of samples that scales as $\mathcal{O}(1)$ with increasing numbers of qubits. However, these techniques require executing a different circuit at each shot, adding a typically prohibitive amount of latency that prohibits their practical application. In this manuscript we consider the approximate state estimation of readout-mitigated expectation values, and how to best implement that procedure on the Rigetti quantum computing hardware. We discuss the theoretical aspects involved, providing an explicit computation of the effect of readout error on the estimated expectation values and how to mitigate that effect. Leveraging improvements to the Rigetti control systems, we then demonstrate an efficient implementation of this approach. Not only do we find that we can suppress the effect of correlated errors and accurately mitigate the readout errors, we find that we can do so quickly, collecting and processing $10^6$ samples in less than $1.5$ minutes. This development opens the way for practical uses of methods with this type of randomisation.

quant-ph↗

arXiv2022

Robust Losses for Learning Value Functions

Most value function learning algorithms in reinforcement learning are based on the mean squared (projected) Bellman error. However, squared errors are known to be sensitive to outliers, both skewing the solution of the objective and resulting in high-magnitude and high-variance gradients. To control these high-magnitude updates, typical strategies in RL involve clipping gradients, clipping rewards, rescaling rewards, or clipping errors. While these strategies appear to be related to robust losses -- like the Huber loss -- they are built on semi-gradient update rules which do not minimize a known loss. In this work, we build on recent insights reformulating squared Bellman errors as a saddlepoint optimization problem and propose a saddlepoint reformulation for a Huber Bellman error and Absolute Bellman error. We start from a formalization of robust losses, then derive sound gradient-based approaches to minimize these losses in both the online off-policy prediction and control settings. We characterize the solutions of the robust losses, providing insight into the problem settings where the robust losses define notably better solutions than the mean squared Bellman error. Finally, we show that the resulting gradient-based algorithms are more stable, for both prediction and control, with less sensitivity to meta-parameters.

cs.LG↗

arXiv2022

A Temporal-Difference Approach to Policy Gradient Estimation

The policy gradient theorem (Sutton et al., 2000) prescribes the usage of a cumulative discounted state distribution under the target policy to approximate the gradient. Most algorithms based on this theorem, in practice, break this assumption, introducing a distribution shift that can cause the convergence to poor solutions. In this paper, we propose a new approach of reconstructing the policy gradient from the start state without requiring a particular sampling strategy. The policy gradient calculation in this form can be simplified in terms of a gradient critic, which can be recursively estimated due to a new Bellman equation of gradients. By using temporal-difference updates of the gradient critic from an off-policy data stream, we develop the first estimator that sidesteps the distribution shift issue in a model-free way. We prove that, under certain realizability conditions, our estimator is unbiased regardless of the sampling strategy. We empirically show that our technique achieves a superior bias-variance trade-off and performance in presence of off-policy samples.

cs.LG↗

arXiv2021

A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learning

Many reinforcement learning algorithms rely on value estimation, however, the most widely used algorithms -- namely temporal difference algorithms -- can diverge under both off-policy sampling and nonlinear function approximation. Many algorithms have been developed for off-policy value estimation based on the linear mean squared projected Bellman error (MSPBE) and are sound under linear function approximation. Extending these methods to the nonlinear case has been largely unsuccessful. Recently, several methods have been introduced that approximate a different objective -- the mean-squared Bellman error (MSBE) -- which naturally facilitate nonlinear approximation. In this work, we build on these insights and introduce a new generalized MSPBE that extends the linear MSPBE to the nonlinear setting. We show how this generalized objective unifies previous work and obtain new bounds for the value error of the solutions of the generalized objective. We derive an easy-to-use, but sound, algorithm to minimize the generalized objective, and show that it is more stable across runs, is less sensitive to hyperparameters, and performs favorably across four control domains with neural network function approximation.

cs.LG↗