arXiv subjects

Caio Kalil Lauand

Publications and source records attributed to Caio Kalil Lauand.

8 recordsLinked to original sources

arXiv2026

Stochastic Online Feedback Optimization for Networks of Non-Compliant Agents

In several applications of online optimization to networked systems such as power grids and robotic networks, information about the system model and its disturbances is not generally available. Within the optimization community, increasing interest has been devoted to the framework of online feedback optimization (OFO), which aims to address these challenges by leveraging real-time input-output measurements to empower online optimization. We extend the OFO framework to a stochastic setting, allowing the subsystems comprising the network (the $\textit{agents}$) to be $\textit{non-compliant}$. This means that the actual control input implemented by the agents is a random variable depending upon the control setpoint generated by the OFO algorithm. Mean-square error bounds are obtained for the general algorithm and the theory is illustrated in application to power systems.

math.OC↗

arXiv2026

Online Feedback Optimization for Constrained Stochastic Problems with Decision-Dependent Distributions: Extended Version

Online feedback optimization (OFO) leverages real-time output measurements to optimize the operation of networked systems without requiring full knowledge of system dynamics or disturbances. We develop an OFO approach for constrained stochastic optimization problems in which the distribution of the system's random parameters shifts in response to the control actions. We propose a projected primal-dual algorithm where the true dual constraint sets are replaced by surrogate sets. Our main result is an upper bound on the mean-square tracking error, which decomposes into four interpretable terms reflecting: (i) the stochasticity of the problem, (ii) output measurement errors, (iii) time-variability of the problem, and (iv) the mismatch between surrogate and true dual constraint sets. The theory is illustrated in a numerical experiment for power grids with price-responsive assets.

math.OC↗

arXiv2025

The case for and against fixed step-size: Stochastic approximation algorithms in optimization and machine learning

Theory and application of stochastic approximation (SA) have become increasingly relevant due in part to applications in optimization and reinforcement learning. This paper takes a new look at SA with constant step-size $α>0$, defined by the recursion, $$θ_{n+1} = θ_{n}+ αf(θ_n,Φ_{n+1})$$ in which $θ_n\in\mathbb{R}^d$ and $\{Φ_{n}\}$ is a Markov chain. The goal is to approximately solve root finding problem $\bar{f}(θ^*)=0$, where $\bar{f}(θ)=\mathbb{E}[f(θ,Φ)]$ and $Φ$ has the steady-state distribution of $\{Φ_{n}\}$. The following conclusions are obtained under an ergodicity assumption on the Markov chain, compatible assumptions on $f$, and for $α>0$ sufficiently small: $\textbf{1.}$ The pair process $\{(θ_n,Φ_n)\}$ is geometrically ergodic in a topological sense. $\textbf{2.}$ For every $1\le p\le 4$, there is a constant $b_p$ such that $\limsup_{n\to\infty}\mathbb{E}[\|θ_n-θ^*\|^p]\le b_p α^{p/2}$ for each initial condition. $\textbf{3.}$ The Polyak-Ruppert-style averaged estimates $θ^{\text{PR}}_n=n^{-1}\sum_{k=1}^{n}θ_k$ converge to a limit $θ^{\text{PR}}_\infty$ almost surely and in mean square, which satisfies $θ^{\text{PR}}_\infty=θ^*+α\barΥ^*+O(α^2)$ for an identified non-random $\barΥ^*\in\mathbb{R}^d$. Moreover, the covariance is approximately optimal: The limiting covariance matrix of $θ^{\text {PR}}_n$ is approximately minimal in a matricial sense. The two main take-aways for practitioners are application-dependent. It is argued that, in applications to optimization, constant gain algorithms may be preferable even when the objective has multiple local minima; while a vanishing gain algorithm is preferable in applications to reinforcement learning due to the presence of bias.

math.ST↗

arXiv2025

Global Convergence and Acceleration for Single Observation Gradient Free Optimization

Simultaneous perturbation stochastic approximation (SPSA) is an approach to gradient-free optimization introduced by Spall as a simplification of the approach of Kiefer and Wolfowitz. In many cases the most attractive option is the single-sample version known as 1SPSA, which is the focus of the present paper, containing two major contributions: a modification of the algorithm designed to ensure convergence from arbitrary initial condition, and a new approach to exploration to dramatically accelerate the rate of convergence. Examples are provided to illustrate the theory, and to demonstrate that estimates from unmodified 1SPSA may diverge even for a quadratic objective function.

math.OC↗

arXiv2025

Revisiting Step-Size Assumptions in Stochastic Approximation

Many machine learning and optimization algorithms are built upon the framework of stochastic approximation (SA), for which the selection of step-size (or learning rate) $\{α_n\}$ is crucial for success. An essential condition for convergence is the assumption that $\sum_n α_n = \infty$. Moreover, in all theory to date it is assumed that $\sum_n α_n^2 < \infty$ (the sequence is square summable). In this paper it is shown for the first time that this assumption is not required for convergence and finer results. The main results are restricted to the special case $α_n = α_0 n^{-ρ}$ with $ρ\in (0,1)$. The theory allows for parameter dependent Markovian noise as found in many applications of interest to the machine learning and optimization research communities. Rates of convergence are obtained for the standard algorithm, and for estimates obtained via the averaging technique of Polyak and Ruppert. $\bullet$ Parameter estimates converge with probability one, and in $L_p$ for any $p\ge 1$. Moreover, the rate of convergence of the the mean-squared error (MSE) is $O(α_n)$, which is improved to $O(\max\{ α_n^2,1/n \})$ with averaging. Finer results are obtained for linear SA: $\bullet$ The covariance of the estimates is optimal in the sense of prior work of Polyak and Ruppert. $\bullet$ Conditions are identified under which the bias decays faster than $O(1/n)$. When these conditions are violated, the bias at iteration $n$ is approximately $β_θα_n$ for a vector $β_θ$ identified in the paper. Results from numerical experiments illustrate that $β_θ$ may be large due to a combination of multiplicative noise and Markovian memory.

math.ST↗

arXiv2024

Markovian Foundations for Quasi-Stochastic Approximation in Two Timescales: Extended Version

Many machine learning and optimization algorithms can be cast as instances of stochastic approximation (SA). The convergence rate of these algorithms is known to be slow, with the optimal mean squared error (MSE) of order $O(n^{-1})$. In prior work it was shown that MSE bounds approaching $O(n^{-4})$ can be achieved through the framework of quasi-stochastic approximation (QSA); essentially SA with careful choice of deterministic exploration. These results are extended to two time-scale algorithms, as found in policy gradient methods of reinforcement learning and extremum seeking control. The extensions are made possible in part by a new approach to analysis, allowing for the interpretation of two timescale algorithms as instances of single timescale QSA, made possible by the theory of negative Lyapunov exponents for QSA. The general theory is illustrated with applications to extremum seeking control (ESC).

math.OC↗

arXiv2024

Markovian Foundations for Quasi-Stochastic Approximation with Applications to Extremum Seeking Control

This paper concerns quasi-stochastic approximation (QSA) to solve root finding problems commonly found in applications to optimization and reinforcement learning. The general constant gain algorithm may be expressed as the time-inhomogeneous ODE $ \frac{d}{dt}Θ_t=αf_t (Θ_t)$, with state process $Θ$ evolving on $\mathbb{R}^d$. Theory is based on an almost periodic vector field, so that in particular the time average of $f_t(θ)$ defines the time-homogeneous mean vector field $\bar{f} \colon \mathbb{R}^d \to \mathbb{R}^d$ with $\bar{f}(θ^*)=0$. Under smoothness assumptions on the functions involved, the following exact representation is obtained: \[\frac{d}{dt}Θ_t=α[\bar{f}(Θ_t)-α\barΥ_t+α^2\mathcal{W}_t^0+α\frac{d}{dt}\mathcal{W}_t^1+\frac{d^2}{dt^2}\mathcal{W}_t^2]\] along with formulae for the smooth signals $\{\bar Υ_t , \mathcal{W}_t^i : i=0, 1, 2\}$. This new representation, combined with new conditions for ultimate boundedness, has many applications for furthering the theory of QSA and its applications, including the following implications that are developed in this paper: (i) A proof that the estimation error $\|Θ_t-θ^*\|$ is of order $O(α)$, but can be reduced to $O(α^2)$ using a second order linear filter. (ii) In application to extremum seeking control, it is found that the results do not apply because the standard algorithms are not Lipschitz continuous. A new approach is presented to ensure that the required Lipschitz bounds hold, and from this we obtain stability, transient bounds, and asymptotic bias of order $O(α^2)$, and asymptotic variance of order $O(α^4)$. (iii) It is in general possible to obtain better than $O(α)$ bounds on error in traditional stochastic approximation when there is Markovian noise.

math.OC↗

arXiv2024

Extremely Fast Convergence Rates for Extremum Seeking Control with Polyak-Ruppert Averaging

Stochastic approximation is a foundation for many algorithms found in machine learning and optimization. It is in general slow to converge: the mean square error vanishes as $O(n^{-1})$. A deterministic counterpart known as quasi-stochastic approximation is a viable alternative in many applications, including gradient-free optimization and reinforcement learning. It was assumed in prior research that the optimal achievable convergence rate is $O(n^{-2})$. It is shown in this paper that through design it is possible to obtain far faster convergence, of order $O(n^{-4+δ})$, with $δ>0$ arbitrary. Two techniques are introduced for the first time to achieve this rate of convergence. The theory is also specialized within the context of gradient-free optimization, and tested on standard benchmarks. The main results are based on a combination of novel application of results from number theory and techniques adapted from stochastic approximation theory.

math.OC↗