arXiv subjects

Mario Zanon

Publications and source records attributed to Mario Zanon.

49 records · Page 3Linked to original sources

arXiv2020

Robust Control Invariance and Communication Scheduling in Lossy Wireless Networked Control Systems

In Networked Control Systems (NCS) impairments of the communication channel can be disruptive to stability and performance. In this paper, we consider the problem of scheduling the access to limited communication resources for a number of decoupled closed-loop systems subject to state and input constraint. The control objective is to preserve the invariance property of local state and input sets, such that constraint satisfaction can be guaranteed. Offline and online, state feedback scheduling policies are proposed and illustrated through numerical examples, also in case the network is subject to packet losses.

eess.SY↗

arXiv2020

Reinforcement Learning Based on Real-Time Iteration NMPC

Reinforcement Learning (RL) has proven a stunning ability to learn optimal policies from data without any prior knowledge on the process. The main drawback of RL is that it is typically very difficult to guarantee stability and safety. On the other hand, Nonlinear Model Predictive Control (NMPC) is an advanced model-based control technique which does guarantee safety and stability, but only yields optimality for the nominal model. Therefore, it has been recently proposed to use NMPC as a function approximator within RL. While the ability of this approach to yield good performance has been demonstrated, the main drawback hindering its applicability is related to the computational burden of NMPC, which has to be solved to full convergence. In practice, however, computationally efficient algorithms such as the Real-Time Iteration (RTI) scheme are deployed in order to return an approximate NMPC solution in very short time. In this paper we bridge this gap by extending the existing theoretical framework to also cover RL based on RTI NMPC. We demonstrate the effectiveness of this new RL approach with a nontrivial example modeling a challenging nonlinear system subject to stochastic perturbations with the objective of optimizing an economic cost.

eess.SY↗

arXiv2020

Reinforcement Learning for Mixed-Integer Problems Based on MPC

Model Predictive Control has been recently proposed as policy approximation for Reinforcement Learning, offering a path towards safe and explainable Reinforcement Learning. This approach has been investigated for Q-learning and actor-critic methods, both in the context of nominal Economic MPC and Robust (N)MPC, showing very promising results. In that context, actor-critic methods seem to be the most reliable approach. Many applications include a mixture of continuous and integer inputs, for which the classical actor-critic methods need to be adapted. In this paper, we present a policy approximation based on mixed-integer MPC schemes, and propose a computationally inexpensive technique to generate exploration in the mixed-integer input space that ensures a satisfaction of the constraints. We then propose a simple compatible advantage function approximation for the proposed policy, that allows one to build the gradient of the mixed-integer MPC-based policy.

eess.SY↗

arXiv2020

Safe Reinforcement Learning via Projection on a Safe Set: How to Achieve Optimality?

For all its successes, Reinforcement Learning (RL) still struggles to deliver formal guarantees on the closed-loop behavior of the learned policy. Among other things, guaranteeing the safety of RL with respect to safety-critical systems is a very active research topic. Some recent contributions propose to rely on projections of the inputs delivered by the learned policy into a safe set, ensuring that the system safety is never jeopardized. Unfortunately, it is unclear whether this operation can be performed without disrupting the learning process. This paper addresses this issue. The problem is analysed in the context of $Q$-learning and policy gradient techniques. We show that the projection approach is generally disruptive in the context of $Q$-learning though a simple alternative solves the issue, while simple corrections can be used in the context of policy gradient methods in order to ensure that the policy gradients are unbiased. The proposed results extend to safe projections based on robust MPC techniques.

eess.SY↗

arXiv2019

A Parallel Decomposition Scheme for Solving Long-Horizon Optimal Control Problems

We present a temporal decomposition scheme for solving long-horizon optimal control problems. In the proposed scheme, the time domain is decomposed into a set of subdomains with partially overlapping regions. Subproblems associated with the subdomains are solved in parallel to obtain local primal-dual trajectories that are assembled to obtain the global trajectories. We provide a sufficient condition that guarantees convergence of the proposed scheme. This condition states that the effect of perturbations on the boundary conditions (i.e., initial state and terminal dual/adjoint variable) should decay asymptotically as one moves away from the boundaries. This condition also reveals that the scheme converges if the size of the overlap is sufficiently large and that the convergence rate improves with the size of the overlap. We prove that linear quadratic problems satisfy the asymptotic decay condition, and we discuss numerical strategies to determine if the condition holds in more general cases. We draw upon a non-convex optimal control problem to illustrate the performance of the proposed scheme.

math.OC↗

arXiv2019

Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case

In this paper, we present a methodology to deploy the deterministic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, where safety is enforced via hard constraints. For continuous input space, imposing safety restrictions on the exploration needed to deploying the deterministic policy gradient method poses some technical difficulties, which we address here. We will investigate in particular policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail the construction of the safe scheme in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the stochastic policy gradient case.

eess.SY↗

arXiv2019

Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part I - Stochastic case

We present a methodology to deploy the stochastic policy gradient method, using actor-critic techniques, when the optimal policy is approximated using a parametric optimization problem, allowing one to enforce safety via hard constraints. For continuous input spaces, imposing safety restrictions on the stochastic policy can make the sampling and evaluation of its density difficult. This paper proposes a computationally effective approach to solve that issue. We will focus on policy approximations based on robust Nonlinear Model Predictive Control (NMPC), where safety can be treated explicitly. For the sake of brevity, we will detail safe policies in the robust linear MPC context only. The extension to the nonlinear case is possible but more complex. We will additionally present a technique to maintain the system safety throughout the learning process in the context of robust linear MPC. This paper has a companion paper treating the deterministic policy gradient case.

eess.SY↗

arXiv2019

Practical Reinforcement Learning of Stabilizing Economic MPC

Reinforcement Learning (RL) has demonstrated a huge potential in learning optimal policies without any prior knowledge of the process to be controlled. Model Predictive Control (MPC) is a popular control technique which is able to deal with nonlinear dynamics and state and input constraints. The main drawback of MPC is the need of identifying an accurate model, which in many cases cannot be easily obtained. Because of model inaccuracy, MPC can fail at delivering satisfactory closed-loop performance. Using RL to tune the MPC formulation or, conversely, using MPC as a function approximator in RL allows one to combine the advantages of the two techniques. This approach has important advantages, but it requires an adaptation of the existing algorithms. We therefore propose an improved RL algorithm for MPC and test it in simulations on a rather challenging example.

eess.SY↗

arXiv2019

Data-driven Economic NMPC using Reinforcement Learning

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature.

eess.SY↗

arXiv2019

Real-Time Constrained Trajectory Planning and Vehicle Control for Proactive Autonomous Driving With Road Users

For motion planning and control of autonomous vehicles to be proactive and safe, pedestrians' and other road users' motions must be considered. In this paper, we present a vehicle motion planning and control framework, based on Model Predictive Control, accounting for moving obstacles. Measured pedestrian states are fed into a prediction layer which translates each pedestrians' predicted motion into constraints for the MPC problem. Simulations and experimental validation were performed with simulated crossing pedestrians to show the performance of the framework. Experimental results show that the controller is stable even under significant input delays, while still maintaining very low computational times. In addition, real pedestrian data was used to further validate the developed framework in simulations.

eess.SY↗

arXiv2018

A Computationally Efficient Model for Pedestrian Motion Prediction

We present a mathematical model to predict pedestrian motion over a finite horizon, intended for use in collision avoidance algorithms for autonomous driving. The model is based on a road map structure, and assumes a rational pedestrian behavior. We compare our model with the state-of-the art and discuss its accuracy, and limitations, both in simulations and in comparison to real data.

eess.SY↗

arXiv2017

A Feasibility-Enforcing Primal-Decomposition SQP Algorithm for Optimal Vehicle Coordination

In this paper we consider the problem of coordinating autonomous vehicles approaching an intersection. We cast the problem in the distributed optimisation framework and propose an algorithm to solve it in real time. We extend previous work on the topic by testing two alternative algorithmic solutions in simulations. Moreover, we test our algorithm in experiments using real cars on a test track. The experimental results demonstrate the applicability and real-time feasibility of the algorithm.

math.OC↗

arXiv2017

Optimal Scheduling of Downlink Communication for a Multi-Agent System with a Central Observation Post

In this paper, we consider a set of agents, which may receive an observation of their state by a central observa- tion post via a shared wireless network. The aim of this work is to design a scheduling mechanism for the central observation post to decide how to allocate the available communication resources. The problem is tackled in two phases: (i) first, the local controllers are designed so as to stabilise the subsystems for the case of perfect communication; (ii) second, the com- munication schedule is decided with the aim of maximising the stability of the subsystems. To this end, we formulate an optimisation problem which explicitly minimises the Lyapunov function increase due to communication limitations. We show how the proposed optimisation can be expressed in terms of Value of Information (VoI), we prove Lyapunov stability in probability and we test our approach in simulations.

math.OC↗