arXiv subjects

James Harrison

Publications and source records attributed to James Harrison.

At least 37 records · Page 2Linked to original sources

arXiv2023

Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies

Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally, we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.

cs.NE↗

arXiv2023

The role of plasma-atom and molecule interactions on power \& particle balance during detachment on the MAST Upgrade Super-X divertor

This paper shows first quantitative analysis of the detachment processes in the MAST Upgrade Super-X divertor (SXD). We identify an unprecedented impact of plasma-molecular interactions involving molecular ions (likely $D_2^+$), resulting in strong ion sinks (Molecular Activated Recombination - MAR), leading to a reduction of ion target flux. The MAR ion sinks exceed the divertor ion sources before electron-ion recombination (EIR) starts to occur, suggesting that significant ionisation occurs outside of the divertor chamber. In the EIR region, $T_e \ll 0.2$ eV is observed and MAR remains significant in these deep detached phases. The total ion sink strength demonstrates the capability for particle (ion) exhaust in the Super-X Configuration. Molecular Activated Dissociation (MAD) is the dominant volumetric neutral atom creation process can lead to an electron cooling of 20\% of $P_{SOL}$. The measured total radiative power losses \emph{in the divertor chamber} are consistent with inferred hydrogenic radiative power losses. This suggests that intrinsic divertor impurity radiation, despite the carbon walls, is minor in the divertor chamber. This contrasts previous TCV results, which may be associated with enhanced plasma-neutral interactions and reduced chemical erosion in the detached, tightly baffled SXD. The above observations have also been observed in higher heat flux (narrower SOL width) type I ELMy H-mode discharges. This provides evidence that the characterisation in this paper may be general.

physics.plasm-ph↗

arXiv2023

Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

Learning-based behavior prediction methods are increasingly being deployed in real-world autonomous systems, e.g., in fleets of self-driving vehicles, which are beginning to commercially operate in major cities across the world. Despite their advancements, however, the vast majority of prediction systems are specialized to a set of well-explored geographic regions or operational design domains, complicating deployment to additional cities, countries, or continents. Towards this end, we present a novel method for efficiently adapting behavior prediction models to new environments. Our approach leverages recent advances in meta-learning, specifically Bayesian regression, to augment existing behavior prediction models with an adaptive layer that enables efficient domain transfer via offline fine-tuning, online adaptation, or both. Experiments across multiple real-world datasets demonstrate that our method can efficiently adapt to a variety of unseen environments.

cs.LG↗

arXiv2023

Graph Reinforcement Learning for Network Control via Bi-Level Optimization

Optimization problems over dynamic networks have been extensively studied and widely used in the past decades to formulate numerous real-world problems. However, (1) traditional optimization-based approaches do not scale to large networks, and (2) the design of good heuristics or approximation algorithms often requires significant manual trial-and-error. In this work, we argue that data-driven strategies can automate this process and learn efficient algorithms without compromising optimality. To do so, we present network control problems through the lens of reinforcement learning and propose a graph network-based framework to handle a broad class of problems. Instead of naively computing actions over high-dimensional graph elements, e.g., edges, we propose a bi-level formulation where we (1) specify a desired next state via RL, and (2) solve a convex program to best achieve it, leading to drastically improved scalability and performance. We further highlight a collection of desirable features to system designers, investigate design decisions, and present experiments on real-world control problems showing the utility, scalability, and flexibility of our framework.

cs.LG↗

arXiv2023

Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systems

We consider the sequential decision-making problem of making proactive request assignment and rejection decisions for a profit-maximizing operator of an autonomous mobility on demand system. We formalize this problem as a Markov decision process and propose a novel combination of multi-agent Soft Actor-Critic and weighted bipartite matching to obtain an anticipative control policy. Thereby, we factorize the operator's otherwise intractable action space, but still obtain a globally coordinated decision. Experiments based on real-world taxi data show that our method outperforms state of the art benchmarks with respect to performance, stability, and computational tractability.

cs.LG↗

arXiv2022

Adaptive Robust Model Predictive Control via Uncertainty Cancellation

We propose a learning-based robust predictive control algorithm that compensates for significant uncertainty in the dynamics for a class of discrete-time systems that are nominally linear with an additive nonlinear component. Such systems commonly model the nonlinear effects of an unknown environment on a nominal system. We optimize over a class of nonlinear feedback policies inspired by certainty equivalent "estimate-and-cancel" control laws pioneered in classical adaptive control to achieve significant performance improvements in the presence of uncertainties of large magnitude, a setting in which existing learning-based predictive control algorithms often struggle to guarantee safety. In contrast to previous work in robust adaptive MPC, our approach allows us to take advantage of structure (i.e., the numerical predictions) in the a priori unknown dynamics learned online through function approximation. Our approach also extends typical nonlinear adaptive control methods to systems with state and input constraints even when we cannot directly cancel the additive uncertain function from the dynamics. We apply contemporary statistical estimation techniques to certify the system's safety through persistent constraint satisfaction with high probability. Moreover, we propose using Bayesian meta-learning algorithms that learn calibrated model priors to help satisfy the assumptions of the control design in challenging settings. Finally, we show in simulation that our method can accommodate more significant unknown dynamics terms than existing methods and that the use of Bayesian meta-learning allows us to adapt to the test environments more rapidly.

eess.SY↗

arXiv2022

VeLO: Training Versatile Learned Optimizers by Scaling Up

While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers. In this work, we leverage the same scaling approach behind the success of deep learning to learn versatile optimizers. We train an optimizer for deep learning which is itself a small neural network that ingests gradients and outputs parameter updates. Meta-trained with approximately four thousand TPU-months of compute on a wide variety of optimization tasks, our optimizer not only exhibits compelling performance, but optimizes in interesting and unexpected ways. It requires no hyperparameter tuning, instead automatically adapting to the specifics of the problem being optimized. We open source our learned optimizer, meta-training code, the associated train and test data, and an extensive optimizer benchmark suite with baselines at velo-code.github.io.

cs.LG↗

arXiv2022

Spectroscopic investigations of detachment on the MAST Upgrade Super-X divertor

We present the first analysis of the atomic and molecular processes at play during detachment in the MAST-U Super-X divertor using divertor spectroscopy data. Our analysis indicates detachment in the MAST-U Super-X divertor can be separated into four sequential phases: First, the ionisation region detaches from the target at detachment onset leaving a region of increased molecular densities downstream. The plasma interacts with these molecules, resulting in molecular ions ($D_2^+$ and/or $D_2^- \rightarrow D + D^-$) that further react with the plasma leading to Molecular Activated Recombination and Dissociation (MAR and MAD), which results in excited atoms and significant Balmer line emission. Second, the MAR region detaches from the target leaving a sub-eV temperature region downstream. Third, an onset of strong emission from electron-ion recombination (EIR) ensues. Finally, the electron density decays near the target, resulting in a density front moving upstream. The analysis in this paper indicates that plasma-molecule interactions have a larger impact than previously reported and play a critical role in the intensity and interpretation of hydrogen atomic line emission characteristics on MAST-U. Furthermore, we find that the Fulcher band emission profile in the divertor can be used as a proxy for the ionisation region and may also be employed as a plasma temperature diagnostic for improving the separation of hydrogenic emission arising from electron-impact excitation and that from plasma-molecular interactions. We provide evidences for the presence of low electron temperatures ($<0.5$ eV) during detachment phases III-IV based on quantitative spectroscopy analysis, a Boltzmann relation of the high-n Balmer line transitions together with an analysis of the brightness of high-n Balmer lines.

physics.plasm-ph↗

arXiv2022

Bayesian Embeddings for Few-Shot Open World Recognition

As autonomous decision-making agents move from narrow operating environments to unstructured worlds, learning systems must move from a closed-world formulation to an open-world and few-shot setting in which agents continuously learn new classes from small amounts of information. This stands in stark contrast to modern machine learning systems that are typically designed with a known set of classes and a large number of examples for each class. In this work we extend embedding-based few-shot learning algorithms to the open-world recognition setting. We combine Bayesian non-parametric class priors with an embedding-based pre-training scheme to yield a highly flexible framework which we refer to as few-shot learning for open world recognition (FLOWR). We benchmark our framework on open-world extensions of the common MiniImageNet and TieredImageNet few-shot learning datasets. Our results show, compared to prior methods, strong classification accuracy performance and up to a 12% improvement in H-measure (a measure of novel class detection) from our non-parametric open-world few-shot learning scheme.

cs.CV↗

arXiv2022

A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases

Learned optimizers -- neural networks that are trained to act as optimizers -- have the potential to dramatically accelerate training of machine learning models. However, even when meta-trained across thousands of tasks at huge computational expense, blackbox learned optimizers often struggle with stability and generalization when applied to tasks unlike those in their meta-training set. In this paper, we use tools from dynamical systems to investigate the inductive biases and stability properties of optimization algorithms, and apply the resulting insights to designing inductive biases for blackbox optimizers. Our investigation begins with a noisy quadratic model, where we characterize conditions in which optimization is stable, in terms of eigenvalues of the training dynamics. We then introduce simple modifications to a learned optimizer's architecture and meta-training procedure which lead to improved stability, and improve the optimizer's inductive bias. We apply the resulting learned optimizer to a variety of neural network training tasks, where it outperforms the current state of the art learned optimizer -- at matched optimizer computational overhead -- with regard to optimization performance and meta-training speed, and is capable of generalization to tasks far different from those it was meta-trained on.

cs.LG↗

arXiv2022

Practical tradeoffs between memory, compute, and performance in learned optimizers

Optimization plays a costly and crucial role in developing machine learning systems. In learned optimizers, the few hyperparameters of commonly used hand-designed optimizers, e.g. Adam or SGD, are replaced with flexible parametric functions. The parameters of these functions are then optimized so that the resulting learned optimizer minimizes a target loss on a chosen class of models. Learned optimizers can both reduce the number of required training steps and improve the final test loss. However, they can be expensive to train, and once trained can be expensive to use due to computational and memory overhead for the optimizer itself. In this work, we identify and quantify the design features governing the memory, compute, and performance trade-offs for many learned and hand-designed optimizers. We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work. Our model and training code are open source.

cs.LG↗

arXiv2022

Safe Active Dynamics Learning and Control: A Sequential Exploration-Exploitation Framework

Safe deployment of autonomous robots in diverse scenarios requires agents that are capable of efficiently adapting to new environments while satisfying constraints. In this work, we propose a practical and theoretically-justified approach to maintaining safety in the presence of dynamics uncertainty. Our approach leverages Bayesian meta-learning with last-layer adaptation. The expressiveness of neural-network features trained offline, paired with efficient last-layer online adaptation, enables the derivation of tight confidence sets which contract around the true dynamics as the model adapts online. We exploit these confidence sets to plan trajectories that guarantee the safety of the system. Our approach handles problems with high dynamics uncertainty, where reaching the goal safely is potentially initially infeasible, by first \textit{exploring} to gather data and reduce uncertainty, before autonomously \textit{exploiting} the acquired information to safely perform the task. Under reasonable assumptions, we prove that our framework guarantees the high-probability satisfaction of all constraints at all times jointly, i.e. over the total task duration. This theoretical analysis also motivates two regularizers of last-layer meta-learning models that improve online adaptation capabilities as well as performance by reducing the size of the confidence sets. We extensively demonstrate our approach in simulation and on hardware.

cs.RO↗

arXiv2022

Graph Meta-Reinforcement Learning for Transferable Autonomous Mobility-on-Demand

Autonomous Mobility-on-Demand (AMoD) systems represent an attractive alternative to existing transportation paradigms, currently challenged by urbanization and increasing travel needs. By centrally controlling a fleet of self-driving vehicles, these systems provide mobility service to customers and are currently starting to be deployed in a number of cities around the world. Current learning-based approaches for controlling AMoD systems are limited to the single-city scenario, whereby the service operator is allowed to take an unlimited amount of operational decisions within the same transportation system. However, real-world system operators can hardly afford to fully re-train AMoD controllers for every city they operate in, as this could result in a high number of poor-quality decisions during training, making the single-city strategy a potentially impractical solution. To address these limitations, we propose to formalize the multi-city AMoD problem through the lens of meta-reinforcement learning (meta-RL) and devise an actor-critic algorithm based on recurrent graph neural networks. In our approach, AMoD controllers are explicitly trained such that a small amount of experience within a new city will produce good system performance. Empirically, we show how control policies learned through meta-RL are able to achieve near-optimal performance on unseen cities by learning rapidly adaptable policies, thus making them more robust not only to novel environments, but also to distribution shifts common in real-world operations, such as special events, unexpected congestion, and dynamic pricing schemes.

eess.SY↗

arXiv2021

On the Problem of Reformulating Systems with Uncertain Dynamics as a Stochastic Differential Equation

We identify an issue in recent approaches to learning-based control that reformulate systems with uncertain dynamics using a stochastic differential equation. Specifically, we discuss the approximation that replaces a model with fixed but uncertain parameters (a source of epistemic uncertainty) with a model subject to external disturbances modeled as a Brownian motion (corresponding to aleatoric uncertainty).

eess.SY↗

arXiv2021

Adaptive Robust Model Predictive Control with Matched and Unmatched Uncertainty

We propose a learning-based robust predictive control algorithm that compensates for significant uncertainty in the dynamics for a class of discrete-time systems that are nominally linear with an additive nonlinear component. Such systems commonly model the nonlinear effects of an unknown environment on a nominal system. We optimize over a class of nonlinear feedback policies inspired by certainty equivalent "estimate-and-cancel" control laws pioneered in classical adaptive control to achieve significant performance improvements in the presence of uncertainties of large magnitude, a setting in which existing learning-based predictive control algorithms often struggle to guarantee safety. In contrast to previous work in robust adaptive MPC, our approach allows us to take advantage of structure (i.e., the numerical predictions) in the a priori unknown dynamics learned online through function approximation. Our approach also extends typical nonlinear adaptive control methods to systems with state and input constraints even when we cannot directly cancel the additive uncertain function from the dynamics. Moreover, we apply contemporary statistical estimation techniques to certify the system's safety through persistent constraint satisfaction with high probability. Finally, we show in simulation that our method can accommodate more significant unknown dynamics terms than existing methods.

eess.SY↗

arXiv2021

Particle MPC for Uncertain and Learning-Based Control

As robotic systems move from highly structured environments to open worlds, incorporating uncertainty from dynamics learning or state estimation into the control pipeline is essential for robust performance. In this paper we present a nonlinear particle model predictive control (PMPC) approach to control under uncertainty, which directly incorporates any particle-based uncertainty representation, such as those common in robotics. Our approach builds on scenario methods for MPC, but in contrast to existing approaches, which either constrain all or only the first timestep to share actions across scenarios, we investigate the impact of a \textit{partial consensus horizon}. Implementing this optimization for nonlinear dynamics by leveraging sequential convex optimization, our approach yields an efficient framework that can be tuned to the particular information gain dynamics of a system to mitigate both over-conservatism and over-optimism. We investigate our approach for two robotic systems across three problem settings: time-varying, partially observed dynamics; sensing uncertainty; and model-based reinforcement learning, and show that our approach improves performance over baselines in all settings.

eess.SY↗

arXiv2021

Graph Neural Network Reinforcement Learning for Autonomous Mobility-on-Demand Systems

Autonomous mobility-on-demand (AMoD) systems represent a rapidly developing mode of transportation wherein travel requests are dynamically handled by a coordinated fleet of robotic, self-driving vehicles. Given a graph representation of the transportation network - one where, for example, nodes represent areas of the city, and edges the connectivity between them - we argue that the AMoD control problem is naturally cast as a node-wise decision-making problem. In this paper, we propose a deep reinforcement learning framework to control the rebalancing of AMoD systems through graph neural networks. Crucially, we demonstrate that graph neural networks enable reinforcement learning agents to recover behavior policies that are significantly more transferable, generalizable, and scalable than policies learned through other approaches. Empirically, we show how the learned policies exhibit promising zero-shot transfer capabilities when faced with critical portability tasks such as inter-city generalization, service area expansion, and adaptation to potentially complex urban topologies.

eess.SY↗

arXiv2021

The role of plasma-molecule interactions on power and particle balance during detachment on the TCV tokamak

This paper shows experimental results from the TCV tokamak that indicate plasma-molecule interactions involving $D_2^+$ and possibly $D^-$ play an important role as sinks of energy (through hydrogenic radiation as well as dissociation) and particles during divertor detachment if low target temperatures ($< 3$ eV) are achieved. Both molecular activated recombination (MAR) and ion source reduction due to a power limitation effect are shown to be important in reducing the ion target flux during a density ramp. In contrast, the electron-ion recombination (EIR) ion sink is too small to play an important role in reducing the ion target flux. MAR or EIR do not occur during $N_2$ seeding induced detachment as the target temperatures are not sufficiently low. The impact of $D_2^+$ is shown to be underestimated in present (vibrationally unresolved) SOLPS-ITER simulations, which could result from an underestimated $D_2 + D^+ \rightarrow D_2^+ + D$ rate. The converged SOLPS-ITER simulations are post-processed with alternative reaction rates, resulting in considerable contributions of $D_2^+$ to particle and power losses as well as dissociation below the $D_2$ dissociation area. Those findings are in quantitative agreement with the experimental results.

physics.plasm-ph↗