arXiv subjects

Sébastien Gros

Publications and source records attributed to Sébastien Gros.

16 recordsLinked to original sources

arXiv2026

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.

cs.RO↗

arXiv2026

Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.

cs.RO↗

arXiv2025

Direct transfer of optimized controllers to similar systems using dimensionless MPC

Scaled model experiments are commonly used in various engineering fields to reduce experimentation costs and overcome constraints associated with full-scale systems. The relevance of such experiments relies on dimensional analysis and the principle of dynamic similarity. However, transferring controllers to full-scale systems often requires additional tuning. In this paper, we propose a method to enable a direct controller transfer using dimensionless model predictive control, tuned automatically for closed-loop performance. With this reformulation, the closed-loop behavior of an optimized controller transfers directly to a new, dynamically similar system. Additionally, the dimensionless formulation allows for the use of data from systems of different scales during parameter optimization. We demonstrate the method on a cartpole swing-up and a car racing problem, applying either reinforcement learning or Bayesian optimization for tuning the controller parameters. Software used to obtain the results in this paper is publicly available at https://github.com/josipkh/dimensionless-mpcrl.

eess.SY↗

arXiv2024

Robustifying Model Predictive Control of Uncertain Linear Systems with Chance Constraints

This paper proposes a model predictive controller for discrete-time linear systems with additive, possibly unbounded, stochastic disturbances and subject to chance constraints. By computing a polytopic probabilistic positively invariant set for constraint tightening with the help of the computation of the minimal robust positively invariant set, the chance constraints are guaranteed, assuming only the mean and covariance of the disturbance distribution are given. The resulting online optimization problem is a standard strictly quadratic programming, just like in conventional model predictive control with recursive feasibility and stability guarantees and is simple to implement. A numerical example is provided to illustrate the proposed method.

math.OC↗

arXiv2023

Integrated Charging Scheduling and Operational Control for an Electric Bus Network

The last few years have seen the massive deployment of electric buses in many existing transit networks. However, the planning and operation of an electric bus system differ from that of a bus system with conventional vehicles, and some key problems have not yet been studied in the literature. In this work, we address the integrated operational control and charging scheduling problem for a network of electric buses with a limited opportunity charging capacity. We propose a hierarchical control framework to solve this problem, where the charging and operational decisions are taken jointly by solving a mixed-integer linear program in the high-level control layer. Since this optimization problem might become very large as more bus lines are considered, we propose to apply Lagrangian relaxation in such a way as to exploit the structure of the problem and enable a decomposition into independent subproblems. A local search heuristic is then deployed in order to generate good feasible solutions to the original problem. This entire Lagrangian heuristic procedure is shown to scale much better on transit networks with an increasing number of bus lines than trying to solve the original problem with an off-the-shelf solver. The proposed procedure is then tested in the high-fidelity microscopic traffic environment Vissim on a bus network constructed from an openly available dataset of the city of Chicago. The results show the benefits of combining the charging scheduling decisions together with the real-time operational control of the vehicles as the proposed control framework manages to achieve both a better level of service and lower charging costs over control baselines with predetermined charging schedules.

eess.SY↗

arXiv2022

Conflict-free Charging and Real-time Control for an Electric Bus Network

The rapid adoption of electric buses by transit agencies around the world is leading to new challenges in the planning and operation of bus networks. In particular, the limited driving range of electric vehicles imposes operational constraints such as the need to charge buses during service. Research on this topic has mostly focused on the strategic and tactical planning aspects until now, and very little work has been done on the real-time operational aspect. To remedy this, we propose integrating the charging scheduling problem with a real-time speed control strategy in this paper. The control problem is formulated as a mixed-integer linear program and solved to optimality with the branch-and-bound method. Simulations are carried out by repeatedly solving the control problem in a receding horizon fashion over a full day of operation. The results show that the proposed controller manages to anticipate and avoid potential conflicts where the charging demand exceeds the charger capacity. This feature makes the controller achieve lower operational costs, both in terms of service regularity and energy consumption, compared to a standard first-come, first-served charging scheme.

eess.SY↗

arXiv2022

Learning for MPC with Stability & Safety Guarantees

The combination of learning methods with Model Predictive Control (MPC) has attracted a significant amount of attention in the recent literature. The hope of this combination is to reduce the reliance of MPC schemes on accurate models, and to tap into the fast developing machine learning and reinforcement learning tools to exploit the growing amount of data available for many systems. In particular, the combination of reinforcement learning and MPC has been proposed as a viable and theoretically justified approach to introduce explainable, safe and stable policies in reinforcement learning. However, a formal theory detailing how the safety and stability of an MPC-based policy can be maintained through the parameter updates delivered by the learning tools is still lacking. This paper addresses this gap. The theory is developed for the generic Robust MPC case, and applied in simulation in the robust tube-based linear MPC case, where the theory is fairly easy to deploy in practice. The paper focuses on Reinforcement Learning as a learning tool, but it applies to any learning method that updates the MPC parameters online.

cs.LG↗

arXiv2022

Economic MPC of Markov Decision Processes: Dissipativity in Undiscounted Infinite-Horizon Optimal Control

Economic Model Predictive Control (MPC) dissipativity theory is central to discussing the stability of policies resulting from minimizing economic stage costs. In its current form, the dissipativity theory for economic MPC applies to problems based on deterministic dynamics or to very specific classes of stochastic problems, and does not readily extend to generic Markov Decision Processes. In this paper, we clarify the core reason for this difficulty, and propose a generalization of the economic MPC dissipativity theory that circumvents it. This generalization focuses on undiscounted infinite-horizon problems and is based on nonlinear stage cost functionals, allowing one to discuss the Lyapunov asymptotic stability of policies for Markov Decision Processes in terms of the probability measures underlying their stochastic dynamics. This theory is illustrated for the stochastic Linear Quadratic Regulator with Gaussian process noise, for which a storage functional can be provided explicitly. For the sake of brevity, we limit our discussion to undiscounted Markov Decision Processes.

eess.SY↗

arXiv2022

A New Dissipativity Condition for Asymptotic Stability of Discounted Economic MPC

Economic Model Predictive Control has recently gained popularity due to its ability to directly optimize a given performance criterion, while enforcing constraint satisfaction for nonlinear systems. Recent research has developed both numerical algorithms and stability analysis for the undiscounted case. The introduction of a discount factor in the cost, however, can be desirable in some cases of interest, e.g., economics, stochastically terminating processes, Markov decision processes, etc. Unfortunately, the stability theory in this case is still not fully developed. In this paper we propose a new dissipativity condition to prove asymptotic stability in the infinite horizon case and we connect our results with existing ones in the literature on discounted economic optimal control. Numerical examples are provided to illustrate the theoretical results.

eess.SY↗

arXiv2021

Reinforcement Learning based on Scenario-tree MPC for ASVs

In this paper, we present the use of Reinforcement Learning (RL) based on Robust Model Predictive Control (RMPC) for the control of an Autonomous Surface Vehicle (ASV). The RL-MPC strategy is utilized for obstacle avoidance and target (set-point) tracking. A scenario-tree robust MPC is used to handle potential failures of the ship thrusters. Besides, the wind and ocean current are considered as unknown stochastic disturbances in the real system, which are handled via constraints tightening. The tightening and other cost parameters are adjusted by RL, using a Q-learning technique. An economic cost is considered, minimizing the time and energy required to achieve the ship missions. The method is illustrated in simulation on a nonlinear 3-DOF model of a scaled version of the Cybership 2.

eess.SY↗

arXiv2021

Optimal Management of the Peak Power Penalty for Smart Grids Using MPC-based Reinforcement Learning

The cost of the power distribution infrastructures is driven by the peak power encountered in the system. Therefore, the distribution network operators consider billing consumers behind a common transformer in the function of their peak demand and leave it to the consumers to manage their collective costs. This management problem is, however, not trivial. In this paper, we consider a multi-agent residential smart grid system, where each agent has local renewable energy production and energy storage, and all agents are connected to a local transformer. The objective is to develop an optimal policy that minimizes the economic cost consisting of both the spot-market cost for each consumer and their collective peak-power cost. We propose to use a parametric Model Predictive Control (MPC)-scheme to approximate the optimal policy. The optimality of this policy is limited by its finite horizon and inaccurate forecasts of the local power production-consumption. A Deterministic Policy Gradient (DPG) method is deployed to adjust the MPC parameters and improve the policy. Our simulations show that the proposed MPC-based Reinforcement Learning (RL) method can effectively decrease the long-term economic cost for this smart grid problem.

eess.SY↗

arXiv2021

Stability-Constrained Markov Decision Processes Using MPC

In this paper, we consider solving discounted Markov Decision Processes (MDPs) under the constraint that the resulting policy is stabilizing. In practice MDPs are solved based on some form of policy approximation. We will leverage recent results proposing to use Model Predictive Control (MPC) as a structured policy in the context of Reinforcement Learning to make it possible to introduce stability requirements directly inside the MPC-based policy. This will restrict the solution of the MDP to stabilizing policies by construction. The stability theory for MPC is most mature for the undiscounted MPC case. Hence, we will first show in this paper that stable discounted MDPs can be reformulated as undiscounted ones. This observation will entail that the MPC-based policy with stability requirements will produce the optimal policy for the discounted MDP if it is stable, and the best stabilizing policy otherwise.

cs.LG↗

arXiv2020

Safe Reinforcement Learning Using Robust MPC

Reinforcement Learning (RL) has recently impressed the world with stunning results in various applications. While the potential of RL is now well-established, many critical aspects still need to be tackled, including safety and stability issues. These issues, while partially neglected by the RL community, are central to the control community which has been widely investigating them. Model Predictive Control (MPC) is one of the most successful control techniques because, among others, of its ability to provide such guarantees even for uncertain constrained systems. Since MPC is an optimization-based technique, optimality has also often been claimed. Unfortunately, the performance of MPC is highly dependent on the accuracy of the model used for predictions. In this paper, we propose to combine RL and MPC in order to exploit the advantages of both and, therefore, obtain a controller which is optimal and safe. We illustrate the results with a numerical example in simulations.

eess.SY↗

arXiv2020

Reinforcement Learning Based on Real-Time Iteration NMPC

Reinforcement Learning (RL) has proven a stunning ability to learn optimal policies from data without any prior knowledge on the process. The main drawback of RL is that it is typically very difficult to guarantee stability and safety. On the other hand, Nonlinear Model Predictive Control (NMPC) is an advanced model-based control technique which does guarantee safety and stability, but only yields optimality for the nominal model. Therefore, it has been recently proposed to use NMPC as a function approximator within RL. While the ability of this approach to yield good performance has been demonstrated, the main drawback hindering its applicability is related to the computational burden of NMPC, which has to be solved to full convergence. In practice, however, computationally efficient algorithms such as the Real-Time Iteration (RTI) scheme are deployed in order to return an approximate NMPC solution in very short time. In this paper we bridge this gap by extending the existing theoretical framework to also cover RL based on RTI NMPC. We demonstrate the effectiveness of this new RL approach with a nontrivial example modeling a challenging nonlinear system subject to stochastic perturbations with the objective of optimizing an economic cost.

eess.SY↗

arXiv2019

Practical Reinforcement Learning of Stabilizing Economic MPC

Reinforcement Learning (RL) has demonstrated a huge potential in learning optimal policies without any prior knowledge of the process to be controlled. Model Predictive Control (MPC) is a popular control technique which is able to deal with nonlinear dynamics and state and input constraints. The main drawback of MPC is the need of identifying an accurate model, which in many cases cannot be easily obtained. Because of model inaccuracy, MPC can fail at delivering satisfactory closed-loop performance. Using RL to tune the MPC formulation or, conversely, using MPC as a function approximator in RL allows one to combine the advantages of the two techniques. This approach has important advantages, but it requires an adaptation of the existing algorithms. We therefore propose an improved RL algorithm for MPC and test it in simulations on a rather challenging example.

eess.SY↗

arXiv2019

Data-driven Economic NMPC using Reinforcement Learning

Reinforcement Learning (RL) is a powerful tool to perform data-driven optimal control without relying on a model of the system. However, RL struggles to provide hard guarantees on the behavior of the resulting control scheme. In contrast, Nonlinear Model Predictive Control (NMPC) and Economic NMPC (ENMPC) are standard tools for the closed-loop optimal control of complex systems with constraints and limitations, and benefit from a rich theory to assess their closed-loop behavior. Unfortunately, the performance of (E)NMPC hinges on the quality of the model underlying the control scheme. In this paper, we show that an (E)NMPC scheme can be tuned to deliver the optimal policy of the real system even when using a wrong model. This result also holds for real systems having stochastic dynamics. This entails that ENMPC can be used as a new type of function approximator within RL. Furthermore, we investigate our results in the context of ENMPC and formally connect them to the concept of dissipativity, which is central for the ENMPC stability. Finally, we detail how these results can be used to deploy classic RL tools for tuning (E)NMPC schemes. We apply these tools on both a classical linear MPC setting and a standard nonlinear example from the ENMPC literature.

eess.SY↗