Mahbod Gholamali Sinaki

arXiv2025

Optimal Regulation of Nonlinear Input-Affine Systems via an Integral Reinforcement Learning-Based State-Dependent Riccati Equation Approach

The State-Dependent Riccati Equation (SDRE) technique generalizes the classical algebraic Riccati formulation to nonlinear systems by designing an input to the system that optimally(suboptimally) regulates system states toward the origin while simultaneously optimizing a quadratic performance index. In the SDRE technique, we solve the State-Dependent Riccati Equation to determine the control for regulating a nonlinear input-affine system. Since an analytic solution to SDRE is not straightforward, one method is to linearize the system at every state, solve the corresponding Algebraic Riccati Equation (ARE), and apply optimal control until the next state of the system. Completing this task with high frequency gives a result like the original SDRE technique. Both approaches require a complete model; therefore, here we propose a method that solves ARE in every state of the system using a partially model-free approach that learns optimal control in every state of the system, without explicit knowledge of the drift dynamics, based on Integral Reinforcement Learning (IRL). To show the effectiveness of our proposed approach, we apply it to the second-order nonlinear system in simulation and compare its performance with the classical SDRE method, which relies on the system's model and solves the ARE at each state. Our simulation results demonstrate that, with sufficient iterations, the IRL-based approach achieves approximately the same performance as the conventional SDRE method, demonstrating its capability as a reliable alternative for nonlinear system control that does not require an explicit environmental model. Index Terms-Algebraic Riccati Equation (ARE), Integral Reinforcement Learning (IRL), Nonlinear Input-Affine Systems, Optimal Regulation, State-Dependent Riccati Equation (SDRE)

eess.SY↗

arXiv2025

LSTM-Based Modeling and Reinforcement Learning Control of a Magnetically Actuated Catheter

Autonomous magnetic catheter systems are emerging as a promising approach for the future of minimally invasive interventions. This study presents a novel approach that begins by modeling the nonlinear and hysteretic dynamics of a magnetically actuated catheter system, consists of a magnetic catheter manipulated by servo-controlled magnetic fields generated by two external permanent magnets, and its complex behavior is captured using a Long Short-Term Memory (LSTM) neural network. This model validated against experimental setup's data with a root mean square error (RMSE) of 0.42 mm and 99.8% coverage within 3 mm, establishing it as a reliable surrogate model. This LSTM enables the training of Reinforcement Learning (RL) agents for controlling the system and avoiding damage to the real setup, with the potential for subsequent fine-tuning on the physical system. We implemented Deep Q-Network (DQN) and actor-critic RL controllers, comparing these two agents first for regulation and subsequently for path following along linear and half-sinusoidal paths for the catheter tip. The actor-critic outperforms DQN, offering greater accuracy and faster performance with less error, along with smoother trajectories at a 10 Hz sampling rate, in both regulation and path following compared to the DQN controller. This performance, due to the continuous action space, suits dynamic navigation tasks like navigating curved vascular structures for practical applications.

eess.SY↗