arXiv subjects

Arunava Naha

Publications and source records attributed to Arunava Naha.

7 recordsLinked to original sources

arXiv2026

Model-free LQG Control with Chance Constraints

This paper studies model-free optimal control design and its convergence properties for linear time-invariant systems subject to probabilistic risk or chance constraints. In particular, we study a natural policy gradient (NPG)-based actor-critic (AC) algorithm with two timescales, using a Lagrangian primal-dual framework to enforce the constraint. Furthermore, the risk is defined as the probability that a function of the one-step-ahead state exceeds a user-specified threshold. To our knowledge, this is the first work to study the analytical convergence properties for NPG-based AC in a chance-constrained linear-quadratic Gaussian (LQG) regulator setting without model knowledge. We establish the coercivity and gradient dominance properties of the Lagrangian function, which ensure linear convergence and closed-loop stability during training for the actor. On the other hand, we analyse the convergence properties of the temporal difference (TD(0)) learning for the critic, applying stochastic approximation theory. Also, we demonstrate no duality gap in the constrained optimisation problem. Additionally, we have performed numerical analysis of the convergence properties and accuracy of the proposed method, comparing it with model-based chance-constrained LQR and scenario-based MPC. Results show that our approach effectively limits risk while maintaining near-optimal performance, without requiring full model knowledge or real-time optimisation.

eess.SY↗

arXiv2025

Convergence of Flow-Policy Gradient Learning for Linear Quadratic Regulator Problems

Flow $Q$-learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure. Central to this innovation is the ``the one-step policy'' network, which is optimized through a $Q$-function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings. Our theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss. Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy. We verify our theoretical finding with simulation results on a linearized inverted pendulum.

eess.SY↗

arXiv2024

Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint

In this paper, we investigate a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data. In particular, we consider linear time-invariant systems and design an optimal controller within the class of linear state feedback control. Three different policy gradient (PG) based algorithms, natural policy gradient (NPG), Gauss-Newton policy gradient (GNPG), and deep deterministic policy gradient (DDPG), are developed, and compared with the optimal risk-neutral linear-quadratic regulator (LQR) and a scenario-based model predictive control (MPC) technique via numerical simulations. The convergence properties and the accuracy of all the algorithms are compared numerically. We also establish analytical convergence properties of the NPG and GNPG algorithms under the known model scenario, while the proof of convergence for the unknown model scenario is part of our ongoing work.

eess.SY↗

arXiv2023

Deterministic policy gradient based optimal control with probabilistic constraints

This paper studies a deep deterministic policy gradient (DDPG) based actor critic (AC) reinforcement learning (RL) technique to control a linear discrete-time system with a quadratic control cost while ensuring a constraint on the probability of potentially risky or undesirable events. The proposed methodology can be applied to both known and unknown system models with minor adjustments to the reward structure (negative cost). The problem is formulated by considering the average expected quadratic cost of the states and inputs over an infinite time horizon. Risky or undesirable events are represented as functions of the states at the next time step exceeding a user-defined limit. Two strategies are employed to manage the probabilistic constraint in scenarios of known and unknown system models. In the case of a known system model, the probabilistic constraint is replaced with an upper bound, such as the Chernoff bound. For unknown system models, the expected value of the indicator function of the occurrence of the risky or undesirable event is used. We have adopted a deterministic policy gradient (DPG) based AC method to derive a parameterised optimal policy. Extensive numerical simulations are performed using a second- and a fourth-order system, and the proposed method is compared with the standard risk-neutral linear quadratic regulator (LQR) and a chance-constrained model predictive control (MPC) method. The results demonstrate the effectiveness of the proposed approach in both known and unknown system model scenarios.

eess.SY↗

arXiv2023

Quickest Detection of Deception Attacks on Cyber-Physical Systems with a Parsimonious Watermarking Policy

The addition of a physical watermarking signal to the control input increases the detection probability of data deception attacks at the expense of increased control cost. In this paper, we propose a parsimonious policy to reduce the average number of watermarking events when the attack is not present, which in turn reduces the control cost. We model the system as a stochastic optimal control problem and apply the dynamic programming to minimize the average detection delay (ADD) for fixed upper bounds on false alarm rate (FAR) and increased control cost. The optimal solution results in a two threshold policy on the posterior probability of attack, which is derived from the Shiryaev statistics for sequential change detection assuming the change point is a random variable with a geometric distribution. We derive approximate expressions of ADD and FAR applying the non-linear renewal theory. The relationship between the average number of watermarking added before the attack and the increase in control cost is also derived. We design the optimal watermarking that maximizes the Kullback-Leibler divergence for a fixed increase in the control cost. Simulation studies are performed to illustrate and validate the theoretical results.

math.OC↗

arXiv2021

Quickest Detection of Deception Attacks in Networked Control Systems with Physical Watermarking

In this paper, we propose and analyze an attack detection scheme for securing the physical layer of a networked control system against attacks where the adversary replaces the true observations with stationary false data. An independent and identically distributed watermarking signal is added to the optimal linear quadratic Gaussian (LQG) control inputs, and a cumulative sum (CUSUM) test is carried out using the joint distribution of the innovation signal and the watermarking signal for quickest attack detection. We derive the expressions of the supremum of the average detection delay (SADD) for a multi-input and multi-output (MIMO) system under the optimal and sub-optimal CUSUM tests. The SADD is asymptotically inversely proportional to the expected Kullback-Leibler divergence (KLD) under certain conditions. The expressions for the MIMO case are simplified for multi-input and single-output systems and explored further to distil design insights. We provide insights into the design of an optimal watermarking signal to maximize KLD for a given fixed increase in LQG control cost when there is no attack. Furthermore, we investigate how the attacker and the control system designer can accomplish their respective objectives by changing the relative power of the attack signal and the watermarking signal. Simulations and numerical studies are carried out to validate the theoretical results.

math.OC↗

arXiv2020

Sequential detection of Replay attacks

One of the most studied forms of attacks on the cyber-physical systems is the replay attack. The statistical similarities of the replay signal and the true observations make the replay attack difficult to detect. In this paper, we have addressed the problem of replay attack detection by adding watermarking to the control inputs and then performed resilient detection using cumulative sum (CUSUM) test on the joint statistics of the innovation signal and the watermarking signal. We derive the expression of the Kullback-Liebler divergence (KLD) between the two joint distributions before and after the replay attack, which is asymptotically inversely proportional to the detection delay. We perform structural analysis of the derived KLD expression and suggest a technique to improve the KLD for the systems with relative degree greater than one. A scheme to find the optimal watermarking signal variance for a fixed increase in the control cost to maximize the KLD under the CUSUM test is presented. We provide various numerical simulation results to support our theory. The proposed method is also compared with a state-of-the-art method.

math.OC↗