Searcharxiv⌕ Search

arXiv subjects

Tom Lefebvre

Publications and source records attributed to Tom Lefebvre.

25 records · Page 2Linked to original sources

Adaptive control of a mechatronic system using constrained residual reinforcement learning

We propose a simple, practical and intuitive approach to improve the performance of a conventional controller in uncertain environments using deep reinforcement learning while maintaining safe operation. Our approach is motivated by the observation that conventional controllers in industrial motion control value robustness over adaptivity to deal with different operating conditions and are suboptimal as a consequence. Reinforcement learning on the other hand can optimize a control signal directly from input-output data and thus adapt to operational conditions, but lacks safety guarantees, impeding its use in industrial environments. To realize adaptive control using reinforcement learning in such conditions, we follow a residual learning methodology, where a reinforcement learning algorithm learns corrective adaptations to a base controller's output to increase optimality. We investigate how constraining the residual agent's actions enables to leverage the base controller's robustness to guarantee safe operation. We detail the algorithmic design and propose to constrain the residual actions relative to the base controller to increase the method's robustness. Building on Lyapunov stability theory, we prove stability for a broad class of mechatronic closed-loop systems. We validate our method experimentally on a slider-crank setup and investigate how the constraints affect the safety during learning and optimality after convergence.

eess.SY↗

Entropy Regularised Deterministic Optimal Control: From Path Integral Solution to Sample-Based Trajectory Optimisation

Sample-based trajectory optimisers are a promising tool for the control of robotics with non-differentiable dynamics and cost functions. Contemporary approaches derive from a restricted subclass of stochastic optimal control where the optimal policy can be expressed in terms of an expectation over stochastic paths. By estimating the expectation with Monte Carlo sampling and reinterpreting the process as exploration noise, a stochastic search algorithm is obtained tailored to (deterministic) trajectory optimisation. For the purpose of future algorithmic development, it is essential to properly understand the underlying theoretical foundations that allow for a principled derivation of such methods. In this paper we make a connection between entropy regularisation in optimisation and deterministic optimal control. We then show that the optimal policy is given by a belief function rather than a deterministic function. The policy belief is governed by a Bayesian-type update where the likelihood can be expressed in terms of a conditional expectation over paths induced by a prior policy. Our theoretical investigation firmly roots sample based trajectory optimisation in the larger family of control as inference. It allows us to justify a number of heuristics that are common in the literature and motivate a number of new improvements that benefit convergence.

cs.RO↗

Risk Sensitive Path Integral Control for Infinite Horizon Problem Formulations

Path Integral Control methods were developed for stochastic optimal control covering a wide class of finite horizon formulations with control affine nonlinear dynamics. Characteristic for this class is that the HJB equation is linear and consequently the value function can be expressed as a conditional expectation of the exponentially weighted cost-to-go evaluated over trajectories with uncontrolled system dynamics, hence the name. Subsequently it was shown that under the same assumptions Path Integral Control generalises to finite horizon risk sensitive stochastic optimal control problems. Here we study whether the HJB of infinite horizon formulations can be made linear as well. Our interest in infinite horizon formulations is motivated by the stationarity of the associated value function and their inherent dynamic stability seeking nature. Technically a stationary value function may ease the solution of the associated linear HJB. Second we argue this may offer an interesting starting point for off-linear Reinforcement Learning applications. We show formally that the discounted and average cost formulations are respectively intractable and tractable.

math.OC↗

On Entropy Regularized Path Integral Control for Trajectory Optimization

In this article we present a generalised view on Path Integral Control (PIC) methods. PIC refers to a particular class of policy search methods that are closely tied to the setting of Linearly Solvable Optimal Control (LSOC), a restricted subclass of nonlinear Stochastic Optimal Control (SOC) problems. This class is unique in the sense that it can be solved explicitly to yield a formal optimal state trajectory distribution. In this contribution we first review the PIC theory and discuss related algorithms tailored to policy search in general. We are able to identify a generic design strategy that relies on the existence of an optimal state trajectory distribution and finds a parametric policy by minimizing the cross entropy between the optimal and a state trajectory distribution parametrized through its policy. Inspired by this observation we then aim to formulate a SOC problem that shares traits with the LSOC setting yet that covers a less restrictive class of problem formulations. We refer to this SOC problem as Entropy Regularized Trajectory Optimization. The problem is closely related to the Entropy Regularized Stochastic Optimal Control setting which is lately often addressed by the Reinforcement Learning (RL) community. We analyse the theoretical convergence behaviour of the theoretical state trajectory distribution sequence and draw connections with stochastic search methods tailored to classic optimization problems. Finally we derive explicit updates and compare the implied Entropy Regularized PIC with earlier work in the context of both PIC and RL for derivative-free trajectory optimization.

math.OC↗

Inverse Parametric Uncertain Identification using Polynomial Chaos and high-order Moment Matching benchmarked on a Wet Friction Clutch

A numerically efficient inverse method for parametric model uncertainty identification using maximum likelihood estimation is presented. The goal is to identify a probability model for a fixed number of model parameters based on a set of experiments. To perform maximum likelihood estimation, the output probability density function is required. Forward propagation of input uncertainty is established combining Polynomial Chaos and moment matching. High-order moments of the output distribution are estimated using the generalized Polynomial Chaos framework. Next, a maximum entropy parametric distribution is matched with the estimated moments. This method is numerically very attractive due to reduced forward sampling and deterministic nature of the propagation strategy. The methodology is applied on a wet clutch system for which certain model variables are considered as stochastic. The number of required model simulations to achieve the same accuracy as the brute force methodologies is decreased by one order of magnitude. The probability model identified with the high order estimates resulted into a true log-likelihood increase of about 4% since the accuracy of the estimated output probability density function could be improved up to 47%.

stat.AP↗

Polynomial Chaos reformulation in Nonlinear Stochastic Optimal Control with application on a drivetrain subject to bifurcation phenomena

This paper discusses a method enabling optimal control of nonlinear systems that are subject to parametric uncertainty. A stochastic optimal tracking problem is formulated that can be expressed in function of the first two stochastic moments of the state. The proposed formulation allows to penalize system performance and system robustness independently. The use of polynomial chaos expansions is investigated to arrive at a computationally tractable formulation expressing the stochastic moments in function of the polynomial expansion coefficients rigorously. It is then demonstrated how the stochastic optimal control problem can be reformulated as a deterministic optimal control problem in function of these coefficients. The proposed method is applied to find a robust control input for the start-up of an eccentrically loaded drive train that is inherently prone to bifurcation behaviour. A reference trajectory is chosen to deliberately provoke a bifurcation. The proposed framework is able to avoid the bifurcation behaviour regardlessly.

math.OC↗

A Radial Basis Function based Optimization Algorithm with Regular Simplex set geometry in Ellipsoidal Trust-Regions

We present a novel derivative-free interpolation based optimization algorithm. A trust-region method is used where a surrogate model is realized via an interpolation framework. The framework for interpolation is provided by Universal Kriging. A first contribution focuses on the development of an original sampling strategy. A valid model is guaranteed by maintaining a well-poised subset that exhibits the regular simplex geometry approximately. It follows that this strategy improves the scattering of points with respect to the state-of-the-art and, even importantly, assures that the surrogate model exhibits curvature. A second contribution focuses on the generalization of the spherical trust-region geometry to an ellipsoidal geometry, that to account for local anisotropy of the objective function and to improve the interpolation conditions as seen from the output space. The ensemble method is validated against its direct competitors on a set of multidimensional problems.

math.OC↗