arXiv subjects

Leilei Cui

Publications and source records attributed to Leilei Cui.

12 recordsLinked to original sources

arXiv2026

LQR for Systems with Probabilistic Parametric Uncertainties: A Gradient Method

A gradient-based method is proposed for solving the linear quadratic regulator (LQR) problem for linear systems with nonlinear dependence on time-invariant probabilistic parametric uncertainties. The approach explicitly accounts for model uncertainty and ensures robust performance. By leveraging polynomial chaos theory (PCT) in conjunction with policy optimization techniques, the original stochastic system is lifted into a high-dimensional linear time-invariant (LTI) system with structured state-feedback control. A first-order gradient descent algorithm is then developed to directly optimize the structured feedback gain and iteratively minimize the LQR cost. We rigorously establish linear convergence of the gradient descent algorithm and show that the PCT-based approximation error decays algebraically at a rate $O(N^{-p})$ for any positive integer $p$, where $N$ denotes the order of the polynomials. Numerical examples demonstrate that the proposed method achieves significantly higher computational efficiency than conventional bilinear matrix inequality (BMI)-based approaches.

eess.SY↗

arXiv2025

Small-Covariance Noise-to-State Stability of Stochastic Systems and Its Applications to Stochastic Gradient Dynamics

This paper studies gradient dynamics subject to additive random noise, which may arise from sources such as stochastic gradient estimation, measurement noise, or stochastic sampling errors. To analyze the robustness of such stochastic gradient systems, the concept of small-covariance noise-to-state stability (NSS) is introduced, along with a Lyapunov-based characterization. Furthermore, the classical Polyak-Lojasiewicz (PL) condition on the objective function is generalized to the $\mathcal{K}$-PL condition via comparison functions, thereby extending its applicability to a broader class of optimization problems. It is shown that the stochastic gradient dynamics exhibit small-covariance NSS if the objective function satisfies the $\mathcal{K}$-PL condition and possesses a globally Lipschitz continuous gradient. This result implies that the trajectories of stochastic gradient dynamics converge to a neighborhood of the optimum with high probability, with the size of the neighborhood determined by the noise covariance. Moreover, if the $\mathcal{K}$-PL condition is strengthened to a $\mathcal{K}_\infty$-PL condition, the dynamics are NSS; whereas if it is weakened to a general positive-definite-PL condition, the dynamics exhibit integral NSS. The results further extend to objectives without globally Lipschitz gradients through appropriate step-size tuning. The proposed framework is further applied to the robustness analysis of policy optimization for the linear quadratic regulator (LQR) and logistic regression.

eess.SY↗

arXiv2025

Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stable

This article investigates the robustness of gradient descent algorithms under perturbations. The concept of small-disturbance input-to-state stability (ISS) for discrete-time nonlinear dynamical systems is introduced, along with its Lyapunov characterization. The conventional linear Polyak-Lojasiewicz (PL) condition is then extended to a nonlinear version, and it is shown that the gradient descent algorithm is small-disturbance ISS provided the objective function satisfies the generalized nonlinear PL condition. This small-disturbance ISS property guarantees that the gradient descent algorithm converges to a small neighborhood of the optimum under sufficiently small perturbations. As a direct application of the developed framework, we demonstrate that the LQR cost satisfies the generalized nonlinear PL condition, thereby establishing that the policy gradient algorithm for LQR is small-disturbance ISS. Additionally, other popular policy gradient algorithms, including natural policy gradient and Gauss-Newton method, are also proven to be small-disturbance ISS.

math.OC↗

arXiv2025

A Fully Data-Driven Value Iteration for Stochastic LQR: Convergence, Robustness and Stability

Unlike traditional model-based reinforcement learning approaches that estimate system parameters from data, non-model-based data-driven control learns the optimal policy directly from input-state data without any intermediate model identification. Although this direct reinforcement learning approach offers increased adaptability and resilience to model misspecification, its reliance on raw data leaves it vulnerable to system noise and disturbances that may undermine convergence, robustness, and stability. In this article, we establish the convergence, robustness, and stability of value iteration (VI) for data-driven control of stochastic linear quadratic (LQ) systems in discrete-time with entirely unknown dynamics and cost. Our contributions are three-fold. First, we prove that VI is globally exponentially stable for any positive semidefinite initial value matrix in noise-free settings, thereby significantly relaxing restrictive assumptions on initial value functions in existing literature. Second, we extend our analysis to settings with external disturbances, proving that VI maintains small-disturbance input-to-state stability (ISS) and converges within a small neighborhood of the optimal solution when disturbances are sufficiently small. Third, we propose a new non-model-based robust adaptive dynamic programming (ADP) algorithm for adaptive optimal controller design, which, unlike existing procedures, requires no prior knowledge of an initial admissible control policy. Numerical experiments on a ``data center cooling'' problem demonstrate the convergence and stability of the algorithm compared to established methods, highlighting its robustness and adaptability for data-driven control in noisy environments. Finally, we apply the method to dynamic portfolio allocation, demonstrating its practical relevance outside traditional control tasks.

math.OC↗

arXiv2025

Remarks on the Polyak-Lojasiewicz inequality and the convergence of gradient systems

This work explores generalizations of the Polyak-Lojasiewicz inequality (PLI) and their implications for the convergence behavior of gradient flows in optimization problems. Motivated by the continuous-time linear quadratic regulator (CT-LQR) policy optimization problem -- where only a weaker version of the PLI is characterized in the literature -- this work shows that while weaker conditions are sufficient for global convergence to, and optimality of the set of critical points of the cost function, the "profile" of the gradient flow solution can change significantly depending on which "flavor" of inequality the cost satisfies. After a general theoretical analysis, we focus on fitting the CT-LQR policy optimization problem to the proposed framework, showing that, in fact, it can never satisfy a PLI in its strongest form. We follow up our analysis with a brief discussion on the difference between continuous- and discrete-time LQR policy optimization, and end the paper with some intuition on the extension of this framework to optimization problems with L1 regularization and solved through proximal gradient flows.

math.OC↗

arXiv2023

Small-Disturbance Input-to-State Stability of Perturbed Gradient Flows: Applications to LQR Problem

This paper studies the effect of perturbations on the gradient flow of a general nonlinear programming problem, where the perturbation may arise from inaccurate gradient estimation in the setting of data-driven optimization. Under suitable conditions on the objective function, the perturbed gradient flow is shown to be small-disturbance input-to-state stable (ISS), which implies that, in the presence of a small-enough perturbation, the trajectories of the perturbed gradient flow must eventually enter a small neighborhood of the optimum. This work was motivated by the question of robustness of direct methods for the linear quadratic regulator problem, and specifically the analysis of the effect of perturbations caused by gradient estimation or round-off errors in policy optimization. We show small-disturbance ISS for three of the most common optimization algorithms: standard gradient flow, natural gradient flow, and Newton gradient flow.

math.OC↗

arXiv2023

Affine Toda system of $\mathbf{A}$ and $\mathbf{C}^t$ type: compactness and affine Weyl group

The local mass is a fundamental quantized information that characterizes the blow-up solution to the Toda system and has a profound relationship with its underlying algebraic structure. In \cite{Lin-Yang-Zhong-2020}, it was observed that the associated Weyl group can be employed to represent this information for the $\mathbf{A}_n$, $\mathbf{B}_n$, $\mathbf{C}_n$ and $\mathbf{G}_2$ type Toda systems. The relationship between the local mass of blow-up solution and the corresponding affine Weyl group is further explored for some affine $\mathbf{B}$ type Toda systems in \cite{Cui-Wei-Yang-Zhang-2022}, where the possible local masses are explicitly expressed in terms of $8$ types. The current work presents a comprehensive study of the general affine $\mathbf{A}$ and $\mathbf{C}^t$ type Toda systems with arbitrary rank. At each stage of the blow-up process (via scaling), we can employ certain elements (known as "set chains") in the corresponding affine Weyl group to measure the variation of local mass. Consequently, we obtain the a priori estimate of the affine $\mathbf{A}$ and $\mathbf{C}^t$ type Toda systems with arbitrary number of singularities.

math.AP↗

arXiv2023

Normalized solutions for a Kirchhoff type equations with potential in $\mathbb{R}^3$

In the present paper, we study the existence of normalized solutions to the following Kirchhoff type equations \begin{equation*} -\left(a+b\int_{\R^3}|\nabla u|^2\right)\Delta u+V(x)u+\lambda u=g(u)~\hbox{in}~\R^3 \end{equation*} satisfying the normalized constraint $\displaystyle\int_{\R^3}u^2=c$, where $a,b,c>0$ are prescribed constants, and the nonlinearities $g(u)$ are very general and of mass super-critical. Under some suitable assumptions on $V(x)$ and $g(u)$, we can prove the existence of ground state normalized solutions $(u_c, \lambda_c)\in H^1(\R^3)\times\mathbb{R}$, for any given $c>0$. Due to the presence of the nonlocal term, the weak limit $u$ of any $(PS)_C$ sequence $\{w_n\}$ may not belong to the corresponding Pohozaev manifold, which is different from the local problem. So we have to overcome some new difficulties to gain the compactness of a $(PS)_C$ sequence.

math.AP↗

arXiv2022

Robust Reinforcement Learning for Risk-Sensitive Linear Quadratic Gaussian Control

This paper proposes a novel robust reinforcement learning framework for discrete-time linear systems with model mismatch that may arise from the sim-to-real gap. A key strategy is to invoke advanced techniques from control theory. Using the formulation of the classical risk-sensitive linear quadratic Gaussian control, a dual-loop policy optimization algorithm is proposed to generate a robust optimal controller. The dual-loop policy optimization algorithm is shown to be globally and uniformly convergent, and robust against disturbances during the learning process. This robustness property is called small-disturbance input-to-state stability and guarantees that the proposed policy optimization algorithm converges to a small neighborhood of the optimal controller as long as the disturbance at each learning step is relatively small. In addition, when the system dynamics is unknown, a novel model-free off-policy policy optimization algorithm is proposed. Finally, numerical examples are provided to illustrate the proposed algorithm.

eess.SY↗

arXiv2022

Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning and adaptive dynamic programming and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving.

eess.SY↗

arXiv2022

Robust Policy Optimization in Continuous-time Mixed $\mathcal{H}_2/\mathcal{H}_\infty$ Stochastic Control

Following the recent resurgence in establishing linear control theoretic benchmarks for reinforcement leaning (RL)-based policy optimization (PO) for complex dynamical systems with continuous state and action spaces, an optimal control problem for a continuous-time infinite-dimensional linear stochastic system possessing additive Brownian motion is optimized on a cost that is an exponent of the quadratic form of the state, input, and disturbance terms. We lay out a model-based and model-free algorithm for RL-based stochastic PO. For the model-based algorithm, we establish rigorous convergence guarantees. For the sampling-based algorithm, over trajectory arcs that emanate from the phase space, we find that the Hamilton-Jacobi Bellman equation parameterizes trajectory costs -- resulting in a discrete-time (input and state-based) sampling scheme accompanied by unknown nonlinear dynamics with continuous-time policy iterates. The need for known dynamics operators is circumvented and we arrive at a reinforced PO algorithm (via policy iteration) where an upper bound on the $\mathcal{H}_2$ norm is minimized (to guarantee stability) and a robustness metric is enforced by maximizing the cost with respect to a controller that includes the level of noise attenuation specified by the system's $H_\infty$ norm. Rigorous robustness analyses is prescribed in an input-to-state stability formalism. Our analyses and contributions are distinguished by many natural systems characterized by additive Wiener process, amenable to \^Ito's stochastic differential calculus in dynamic game settings.

eess.SY↗

arXiv2022

The Blow-up Analysis on $\mathbf{B}_2^{(1)}$ Affine Toda system: Local mass and Affine Weyl group

It has been established that the local mass of blow-up solutions to Toda systems associated with the simple Lie algebras $\mathbf{A}_n,~\mathbf{B}_n,~\mathbf{C}_n$ and $\mathbf{G}_2$ can be represented by a finite Weyl group. In particular, at each blow-up point, after a sequence of bubbling steps (via scaling) is performed, the transformation of the local mass at each step corresponds to the action of an element in the Weyl group. In this article, we present the results in the same spirit for the affine $\mathbf{B}_2^{(1)}$ Toda system with singularities. Compared with the Toda system with simple Lie algebras, the computation of local masses is more challenging due to the infinite number of elements of the {affine Weyl group of type $\mathbf{B}_{2}^{(1)}$}. In order to give an explicit expression for the local mass formula we introduce two free integers and write down all the possibilities into 8 types. This shows a striking difference to previous results on Toda systems with simple Lie algebras. The main result of this article seems to provide the first major advance in understanding the relation between the blow-up analysis of affine Toda system and the {affine Weyl group} of the associated Lie algebras.

math.AP↗