arXiv subjects

Joe Watson

Publications and source records attributed to Joe Watson.

33 records · Page 2Linked to original sources

arXiv2022

Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes

Monte Carlo methods have become increasingly relevant for control of non-differentiable systems, approximate dynamics models and learning from data. These methods scale to high-dimensional spaces and are effective at the non-convex optimizations often seen in robot learning. We look at sample-based methods from the perspective of inference-based control, specifically posterior policy iteration. From this perspective, we highlight how Gaussian noise priors produce rough control actions that are unsuitable for physical robot deployment. Considering smoother Gaussian process priors, as used in episodic reinforcement learning and motion planning, we demonstrate how smoother model predictive control can be achieved using online sequential inference. This inference is realized through an efficient factorization of the action distribution and a novel means of optimizing the likelihood temperature to improve importance sampling accuracy. We evaluate this approach on several high-dimensional robot control tasks, matching the sample efficiency of prior heuristic methods while also ensuring smoothness. Simulation results can be seen at https://monte-carlo-ppi.github.io/.

cs.LG↗

arXiv2022

Real Robot Challenge: A Robotics Competition in the Cloud

Dexterous manipulation remains an open problem in robotics. To coordinate efforts of the research community towards tackling this problem, we propose a shared benchmark. We designed and built robotic platforms that are hosted at MPI for Intelligent Systems and can be accessed remotely. Each platform consists of three robotic fingers that are capable of dexterous object manipulation. Users are able to control the platforms remotely by submitting code that is executed automatically, akin to a computational cluster. Using this setup, i) we host robotics competitions, where teams from anywhere in the world access our platforms to tackle challenging tasks ii) we publish the datasets collected during these competitions (consisting of hundreds of robot hours), and iii) we give researchers access to these platforms for their own projects.

cs.RO↗

arXiv2022

Efficient Stochastic Optimal Control through Approximate Bayesian Input Inference

Optimal control under uncertainty is a prevailing challenge for many reasons. One of the critical difficulties lies in producing tractable solutions for the underlying stochastic optimization problem. We show how advanced approximate inference techniques can be used to handle the statistical approximations principled and practically by framing the control problem as a problem of input estimation. Analyzing the Gaussian setting, we present an inference-based solver that is effective in stochastic and deterministic settings and was found to be superior to popular baselines on nonlinear simulated tasks. We draw connections that relate this inference formulation to previous approaches for stochastic optimal control and outline several advantages that this inference view brings due to its statistical nature.

cs.LG↗

arXiv2021

Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation

Dexterous manipulation is a challenging and important problem in robotics. While data-driven methods are a promising approach, current benchmarks require simulation or extensive engineering support due to the sample inefficiency of popular methods. We present benchmarks for the TriFinger system, an open-source robotic platform for dexterous manipulation and the focus of the 2020 Real Robot Challenge. The benchmarked methods, which were successful in the challenge, can be generally described as structured policies, as they combine elements of classical robotics and modern policy optimization. This inclusion of inductive biases facilitates sample efficiency, interpretability, reliability and high performance. The key aspects of this benchmarking is validation of the baselines across both simulation and the real system, thorough ablation study over the core features of each solution, and a retrospective analysis of the challenge as a manipulation benchmark. The code and demo videos for this work can be found on our website (https://sites.google.com/view/benchmark-rrc).

cs.RO↗

arXiv2021

A Differentiable Newton-Euler Algorithm for Real-World Robotics

Obtaining dynamics models is essential for robotics to achieve accurate model-based controllers and simulators for planning. The dynamics models are typically obtained using model specification of the manufacturer or simple numerical methods such as linear regression. However, this approach does not guarantee physically plausible parameters and can only be applied to kinematic chains consisting of rigid bodies. In this article, we describe a differentiable simulator that can be used to identify the system parameters of real-world mechanical systems with complex friction models, holonomic as well as non-holonomic constraints. To guarantee physically consistent parameters, we utilize virtual parameters and gradient-based optimization. The described Differentiable Newton-Euler Algorithm (DiffNEA) can be applied to a class of dynamical systems and guarantees physically plausible predictions. The extensive experimental evaluation shows, that the proposed model learning approach learns accurate dynamics models of systems with complex friction and non-holonomic constraints. Especially in the offline reinforcement learning experiments, the identified DiffNEA models excel. For the challenging ball in a cup task, these models solve the task using model-based offline reinforcement learning on the physical system. The black-box baselines fail on this task in simulation and on the physical system despite using more data for learning the model.

cs.RO↗

arXiv2021

Advancing Trajectory Optimization with Approximate Inference: Exploration, Covariance Control and Adaptive Risk

Discrete-time stochastic optimal control remains a challenging problem for general, nonlinear systems under significant uncertainty, with practical solvers typically relying on the certainty equivalence assumption, replanning and/or extensive regularization. Control as inference is an approach that frames stochastic control as an equivalent inference problem, and has demonstrated desirable qualities over existing methods, namely in exploration and regularization. We look specifically at the input inference for control (i2c) algorithm, and derive three key characteristics that enable advanced trajectory optimization: An `expert' linear Gaussian controller that combines the benefits of open-loop optima and closed-loop variance reduction when optimizing for nonlinear systems, inherent adaptive risk sensitivity from the inference formulation, and covariance control functionality with only a minor algorithmic adjustment.

eess.SY↗

arXiv2020

Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

A limitation of model-based reinforcement learning (MBRL) is the exploitation of errors in the learned models. Black-box models can fit complex dynamics with high fidelity, but their behavior is undefined outside of the data distribution.Physics-based models are better at extrapolating, due to the general validity of their informed structure, but underfit in the real world due to the presence of unmodeled phenomena. In this work, we demonstrate experimentally that for the offline model-based reinforcement learning setting, physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known. Physics-based models can learn to perform the ball in a cup (BiC) task on a physical manipulator using only 4 minutes of sampled data using offline MBRL. We find that black-box models consistently produce unviable policies for BiC as all predicted trajectories diverge to physically impossible state, despite having access to more data than the physics-based model. In addition, we generalize the approach of physics parameter identification from modeling holonomic multi-body systems to systems with nonholonomic dynamics using end-to-end automatic differentiation. Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/

cs.RO↗

arXiv2020

A Differentiable Newton Euler Algorithm for Multi-body Model Learning

In this work, we examine a spectrum of hybrid model for the domain of multi-body robot dynamics. We motivate a computation graph architecture that embodies the Newton Euler equations, emphasizing the utility of the Lie Algebra form in translating the dynamical geometry into an efficient computational structure for learning. We describe the used virtual parameters that enable unconstrained physical plausible dynamics and the used actuator models. In the experiments, we define a family of 26 grey-box models and evaluate them for system identification of the simulated and physical Furuta Pendulum and Cartpole. The comparison shows that the kinematic parameters, required by previous white-box system identification methods, can be accurately inferred from data. Furthermore, we highlight that models with guaranteed bounded energy of the uncontrolled system generate non-divergent trajectories, while more general models have no such guarantee, so their performance strongly depends on the data distribution. Therefore, the main contributions of this work is the introduction of a white-box model that jointly learns dynamic and kinematics parameters and can be combined with black-box components. We then provide extensive empirical evaluation on challenging systems and different datasets that elucidates the comparative performance of our grey-box architecture with comparable white- and black-box models.

cs.RO↗

arXiv2020

Active Inference or Control as Inference? A Unifying View

Active inference (AI) is a persuasive theoretical framework from computational neuroscience that seeks to describe action and perception as inference-based computation. However, this framework has yet to provide practical sensorimotor control algorithms that are competitive with alternative approaches. In this work, we frame active inference through the lens of control as inference (CaI), a body of work that presents trajectory optimization as inference. From the wider view of `probabilistic numerics', CaI offers principled, numerically robust optimal control solvers that provide uncertainty quantification, and can scale to nonlinear problems with approximate inference. We show that AI may be framed as partially-observed CaI when the cost function is defined specifically in the observation states.

cs.LG↗

arXiv2020

Estimating animal utilization distributions from multiple data types: a joint spatio-temporal point process framework

Models of the spatial distribution of animals provide useful tools to help ecologists quantify species-environment relationships, and they are increasingly being used to help determine the impacts of climate and habitat changes on species. While high-quality survey-style data with known effort are sometimes available, often researchers have multiple datasets of varying quality and type. In particular, collections of sightings made by citizen scientists are becoming increasingly common, with no information typically provided on their observer effort. Many standard modelling approaches ignore observer effort completely, which can severely bias estimates of an animal's distribution. Combining sightings data from observers who followed different protocols is challenging. Any differences in observer skill, spatial effort, and the detectability of the animals across space all need to be accounted for. To achieve this, we build upon the recent advancements made in integrative species distribution models and present a novel marked spatio-temporal point process framework for estimating the utilization distribution (UD) of the individuals of a highly mobile species. We show that in certain settings, we can also use the framework to combine the UDs from the sampled individuals to estimate the species' distribution. We combine the empirical results from a simulation study with the implications outlined in a causal directed acyclic graph to identify the necessary assumptions required for our framework to control for observer effort when it is unknown. We then apply our framework to combine multiple datasets collected on the endangered Southern Resident Killer Whales, to estimate their monthly effort-corrected space-use.

stat.ME↗

arXiv2020

Stochastic Optimal Control as Approximate Input Inference

Optimal control of stochastic nonlinear dynamical systems is a major challenge in the domain of robot learning. Given the intractability of the global control problem, state-of-the-art algorithms focus on approximate sequential optimization techniques, that heavily rely on heuristics for regularization in order to achieve stable convergence. By building upon the duality between inference and control, we develop the view of Optimal Control as Input Estimation, devising a probabilistic stochastic optimal control formulation that iteratively infers the optimal input distributions by minimizing an upper bound of the control cost. Inference is performed through Expectation Maximization and message passing on a probabilistic graphical model of the dynamical system, and time-varying linear Gaussian feedback controllers are extracted from the joint state-action distribution. This perspective incorporates uncertainty quantification, effective initialization through priors, and the principled regularization inherent to the Bayesian treatment. Moreover, it can be shown that for deterministic linearized systems, our framework derives the maximum entropy linear quadratic optimal control law. We provide a complete and detailed derivation of our probabilistic approach and highlight its advantages in comparison to other deterministic and probabilistic solvers.

cs.LG↗

arXiv2020

A fast Monte Carlo test for preferential sampling

The preferential sampling of locations chosen to observe a spatio-temporal process has been identified as a major problem across multiple fields. Predictions of the process can be severely biased when standard statistical methodologies are applied to preferentially sampled data without adjustment. Currently, methods that can adjust for preferential sampling are rarely implemented in the software packages most popular with researchers. Furthermore, they are technically demanding to design and fit. This paper presents a fast and intuitive Monte Carlo test for detecting preferential sampling. The test can be applied across a wide range of data types. Importantly, the method can also help with the discovery of a set of informative covariates that can sufficiently control for the preferential sampling. The discovery of these covariates can justify continued use of standard methodologies. A thorough simulation study is presented to demonstrate both the power and validity of the test in various data settings. The test is shown to attain high power for non-Gaussian data with sample sizes as low as 50. Finally, two previously-published case studies are revisited and new insights into the nature of the informative sampling are gained. The test can be implemented with the R package PStestR

stat.ME↗

arXiv2019

A general theory for preferential sampling in environmental networks

This paper presents a general model framework for detecting the preferential sampling of environmental monitors recording an environmental process across space and/or time. This is achieved by considering the joint distribution of an environmental process with a site--selection process that considers where and when sites are placed to measure the process. The environmental process may be spatial, temporal or spatio--temporal in nature. By sharing random effects between the two processes, the joint model is able to establish whether site placement was stochastically dependent of the environmental process under study. The embedding into a spatio--temporal framework also allows for the modelling of the dynamic site---selection process itself. Real--world factors affecting both the size and location of the network can be easily modelled and quantified. Depending upon the choice of population of locations to consider for selection across space and time under the site--selection process, different insights about the precise nature of preferential sampling can be obtained. The general framework developed in the paper is designed to be easily and quickly fit using the R-INLA package. We apply this framework to a case study involving particulate air pollution over the UK where a major reduction in the size of a monitoring network through time occurred. It is demonstrated that a significant response--biased reduction in the air quality monitoring network occurred. We also show that the network was consistently unrepresentative of the levels of particulate matter seen across much of GB throughout the operating life of the network. Finally we show that this may have led to a severe over-reporting of the population--average exposure levels experienced across GB. This could have great impacts on estimates of the health effects of black smoke levels.

stat.ME↗

arXiv1996

Dynamic Critical Phenomena in Channel Flow

A simple model of the driven motion of interacting particles in a two dimensional random medium is analyzed, focusing on the critical behavior near to the threshold that separates a static phase from a flowing phase with a steady-state current. The critical behavior is found to be surprisingly robust, being independent of whether the driving force is increased suddenly or adiabatically. Just above threshold, the flow is concentrated on a sparse network of channels, but the time scale for convergence to this fixed network diverges with a larger exponent that that for convergence of the current density to its steady-state value. This is argued to be caused by the ``dangerous irrelevance'' of dynamic particle collisions at the critical point. Possible applications to vortex motion near to the critical current in dirty thin film superconductors are discussed briefly.

cond-mat.stat-mech↗

arXiv1995

Collective Particle Flow through Random Media

A simple model for the nonlinear collective transport of interacting particles in a random medium with strong disorder is introduced and analyzed. A finite threshold for the driving force divides the behavior into two regimes characterized by the presence or absence of a steady-state particle current. Below this threshold, transient motion is found in response to an increase in the force, while above threshold the flow approaches a steady state with motion only on a network of channels which is sparse near threshold. Some of the critical behavior near threshold is analyzed via mean field theory, and analytic results on the statistics of the moving phase are derived. Many of the results should apply, at least qualitatively, to the motion of magnetic bubble arrays and to the driven motion of vortices in thin film superconductors when the randomness is strong enough to destroy the tendencies to lattice order even on short length scales. Various history dependent phenomena are also discussed.

cond-mat↗