arXiv subjects

D. J. Antunes

Publications and source records attributed to D. J. Antunes.

5 recordsLinked to original sources

arXiv2026

Data-Efficient Quadratic Q-Learning Using LMIs

Reinforcement learning (RL) has seen significant research and application results but often requires large amounts of training data. This paper proposes two data-efficient off-policy RL methods that use parametrized Q-learning. In these methods, the Q-function is chosen to be linear in the parameters and quadratic in selected basis functions in the state and control deviations from a base policy. A cost penalizing the $\ell_1$-norm of Bellman errors is minimized. We propose two methods: Linear Matrix Inequality Q-Learning (LMI-QL) and its iterative variant (LMI-QLi), which solve the resulting episodic optimization problem through convex optimization. LMI-QL relies on a convex relaxation that yields a semidefinite programming (SDP) problem with linear matrix inequalities (LMIs). LMI-QLi entails solving sequential iterations of an SDP problem. Both methods combine convex optimization with direct Q-function learning, significantly improving learning speed. A numerical case study demonstrates their advantages over existing parametrized Q-learning methods.

eess.SY↗

arXiv2026

Smart Exploration in Reinforcement Learning using Bounded Uncertainty Models

Reinforcement learning (RL) is a powerful framework for decision-making in uncertain environments, but it often requires large amounts of data to learn an optimal policy. We address this challenge by incorporating prior model knowledge to guide exploration and accelerate the learning process. Specifically, we assume access to a model set that contains the true transition kernel and reward function. We optimize over this model set to obtain upper and lower bounds on the Q-function, which are then used to guide the exploration of the agent. We provide theoretical guarantees on the convergence of the Q-function to the optimal Q-function under the proposed class of exploring policies. Furthermore, we also introduce a data-driven regularized version of the model set optimization problem that ensures the convergence of the class of exploring policies to the optimal policy. Lastly, we show that when the model set has a specific structure, namely the bounded-parameter MDP (BMDP) framework, the regularized model set optimization problem becomes convex and simple to implement. In this setting, we also prove finite-time convergence to the optimal policy under mild assumptions. We demonstrate the effectiveness of the proposed exploration strategy, which we call BUMEX (Bounded Uncertainty Model-based Exploration), in a simulation study. The results indicate that the proposed method can significantly accelerate learning in benchmark examples. A toolbox is available at https://github.com/JvHulst/BUMEX.

cs.LG↗

arXiv2026

Precision Specimen Positioning in Electron Microscopy through Hysteresis Compensation, Iterative Learning, and Vision-Based Sensing

Electron microscopy requires nanometer-scale specimen positioning over a long stroke. Piezo-stepper actuators are well suited for this task, but their accuracy is limited by hysteresis, mechanical misalignments, and non-collocated sensing. Prior work has addressed these limitations on simplified lab setups. However, extending to a full electron microscope stage introduces coupled nonlinear kinematics and, importantly, the absence of a dedicated point-of-interest (POI) sensor. This paper presents an integrated feedforward framework for precision positioning on such a stage inside an operational electron microscope. Per-element hysteresis compensation first linearizes the actuator response. In the absence of a dedicated POI sensor, a POI measurement is constructed from EM images through cross-correlation-based image tracking. From this measurement, we construct an encoder-based proxy for the POI position. Commutation-angle-domain iterative learning control then uses this proxy as its error signal to cancel the repeatable disturbances of stepping. Because the learned corrections are parameterized in the commutation angle, they transfer across the quasi-static range of drive frequencies. The framework reduces the POI tracking error by over 13x on the lab setup and by 7 to 12x on an operational transmission electron microscope.

eess.SY↗

arXiv2026

Estimating Evolving Functions with Dynamic Gaussian Processes

This paper develops the Dynamic Gaussian Process (DGP), a framework for estimating functions governed by integro-difference equations (IDEs). IDEs model continuous functions that evolve with discrete-time dynamics and arise naturally from time-discretization of linear partial differential equations (PDEs). The DGP extends Gaussian process regression to time-varying functions and extends Kalman filtering to infinite-dimensional states. The DGP posterior remains a Gaussian process with closed-form mean and covariance updates, and separable kernel structure reduces the problem to a finite-dimensional Kalman filter on basis function coefficients. This paper extends the DGP to vector-valued states, enabling the treatment of higher-order PDEs, and provides a stability and approximation error analysis for the basis function approximation. The functional L2 estimation error decomposes exactly into in-subspace and out-of-subspace contributions, and all approximation errors vanish as the number of basis functions grows. The framework is demonstrated on the heat equation and on the wave equation, the latter with a vector-valued state. Code is available at https://github.com/JvHulst/Dynamic_Gaussian_Processes.

eess.SY↗

arXiv2020

Decentralized LQ-Consistent Event-triggered Control over a Shared Contention-based Network

Consider a network of multiple independent stochastic linear systems where, for each system, a scheduler collocated with the sensors arbitrates data transmissions to a corresponding remote controller through a shared contention-based communication network. While the systems are physically independent, their optimal controller design problems may, in general, become coupled, due to network contention, if the schedulers trigger transmissions based on state-dependent events. In this article we propose a class of probabilistic admissible schedulers for which the optimal controllers, with respect to local standard LQG costs, have the certainty equivalence property and can still be determined decentrally. Then, two scheduling policies within this class are introduced; a non-event-based and an event-based, both with an easily adjustable triggering probability at every time step. We then prove that, for each closed-loop system, the event-based scheduler and its optimal controller outperforms the closed-loop system with the non-event-based scheduler and its associated optimal controller. Moreover, we show that, for each closed-loop system, the optimal state estimators for both scheduling policies follows a linear iteration. Finally, we provide a method to regulate the triggering probabilities of the schedulers by maximizing a network utility function.

eess.SY↗