arXiv subjects

Omar Saadi

Publications and source records attributed to Omar Saadi.

8 recordsLinked to original sources

arXiv2026

Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise

We study stochastic zeroth-order (ZO) optimization of smooth nonconvex objectives under heavy-tailed sample-gradient noise. This regime is motivated by empirical evidence that gradient noise in modern machine learning can violate the bounded-variance assumptions used in classical ZO theory. While first-order methods have optimal rates under bounded $p$-th moment noise for $p\in(1,2]$, analogous high-probability guarantees for nonconvex ZO methods are much less understood. The ZO setting is not a direct corollary of first-order theory. First-order methods observe stochastic gradients, whereas derivative-free methods only query noisy function values and build finite-difference estimates. Thus, weak-$L_p$ control of $\nabla F(x;\xi)-\nabla f(x)$ must first be transferred to scalar directional estimates. We propose the Robust Scalar-Clipped Zeroth-Order method (RSC-ZO), a two-point method that clips each scalar directional derivative before aggregation. Under sample-wise smoothness and a weak-$L_p$ tail condition on the sample-gradient noise, RSC-ZO finds an $\varepsilon$-stationary point with high probability using $$ \widetilde{O}\!\left( d^{\frac{p}{2(p-1)}}\varepsilon^{-\frac{3p-2}{p-1}} \right) $$ noisy function evaluations. This matches the optimal first-order $\varepsilon$-dependence. At $p=2$, the bound becomes $\widetilde{O}(d\varepsilon^{-4})$, matching the classical stochastic ZO dimension--accuracy dependence, but with a high-probability guarantee and under a weaker weak-$L_2$ condition that can allow infinite variance. We also analyze a momentum variant and quantify its batch-size/stepsize tradeoff.

math.OC↗

arXiv2026

Nonsmooth Optimization with Zeroth Order Comparison Feedback

We study unconstrained optimization problems of nonsmooth, nonconvex Lipschitz functions, using only noisy pairwise comparisons governed by a known link function. Our goal is to compute a $(\delta,\varepsilon)$-Goldstein stationary point. We combine randomized smoothing with a novel unbiased reduction from comparisons to local value differences. By leveraging a Russian-roulette truncation on the Bernoulli-product expansion of the inverse link, we construct an exactly unbiased estimator for directional differences. This estimator has finite expected cost and variance scaling quadratically with the function gap, $\mathcal{O}(B^2)$, under mild conditions. Plugging this into the smoothed gradient identity enables a standard nonconvex SGD analysis, yielding explicit comparison-complexity bounds for common symmetric links such as logistic, probit, and cauchit.

math.OC↗

arXiv2026

Noisy Pairwise-Comparison Random Search for Smooth Nonconvex Optimization

We study smooth nonconvex optimization using only noisy pairwise comparisons, without access to gradients or function values. We propose Noisy-Comparison Random Search (NCRS), a simple direct-search method that samples random directions and performs accept/reject updates from comparison feedback. Under a low-dimensional active-subspace structure, NCRS adapts to the intrinsic dimension $k\le d$ rather than the ambient dimension $d$. For a uniform-margin comparison oracle with advantage $p$, NCRS achieves $\epsilon$-first-order stationarity with comparison complexity $\mathcal{O}(k/(p^2\epsilon^2))$. We also introduce a gap-dependent confidence model, where comparison reliability decreases as the objective-value gap between the two candidates becomes small, and analyze a confidence-weighted voting variant of NCRS. For this oracle, the method achieves $\epsilon$-first-order stationarity with total comparison complexity $\mathcal{O}(k^2/\epsilon^4)$. These results provide intrinsic-dimension convergence guarantees for noisy comparison-based random search in smooth nonconvex optimization.

math.OC↗

arXiv2025

Stochastic Optimization with Random Search

We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than previously considered, and that stronger assumptions enable improved guarantees. In the finite-sum setting, we design a variance-reduced variant that leverages multiple samples to accelerate convergence. Our analysis relies on a simple translation invariance property, which provides a principled way to balance noise and reduce variance.

math.OC↗

arXiv2025

On the Almost Sure Convergence of the Stochastic Three Points Algorithm

The stochastic three points (STP) algorithm is a derivative-free optimization technique designed for unconstrained optimization problems in $\mathbb{R}^d$. In this paper, we analyze this algorithm for three classes of functions: smooth functions that may lack convexity, smooth convex functions, and smooth functions that are strongly convex. Our work provides the first almost sure convergence results of the STP algorithm, alongside some convergence results in expectation. For the class of smooth functions, we establish that the best gradient iterate of the STP algorithm converges almost surely to zero at a rate of $o(1/{T^{\frac{1}{2}-\epsilon}})$ for any $\epsilon\in (0,\frac{1}{2})$, where $T$ is the number of iterations. Furthermore, within the same class of functions, we establish both almost sure convergence and convergence in expectation of the final gradient iterate towards zero. For the class of smooth convex functions, we establish that $f(\theta^T)$ converges to $\inf_{\theta \in \mathbb{R}^d} f(\theta)$ almost surely at a rate of $o(1/{T^{1-\epsilon}})$ for any $\epsilon\in (0,1)$, and in expectation at a rate of $O(\frac{d}{T})$ where $d$ is the dimension of the space. Finally, for the class of smooth functions that are strongly convex, we establish that when step sizes are obtained by approximating the directional derivatives of the function, $f(\theta^T)$ converges to $\inf_{\theta \in \mathbb{R}^d} f(\theta)$ in expectation at a rate of $O((1-\frac{\mu}{2\pi dL})^T)$, and almost surely at a rate of $o((1-s\frac{\mu}{2\pi dL})^T)$ for any $s\in (0,1)$, where $\mu$ and $L$ are the strong convexity and smoothness parameters of the function.

math.OC↗

arXiv2021

Multiply Accelerated Value Iteration for Non-Symmetric Affine Fixed Point Problems and application to Markov Decision Processes

We analyze a modified version of Nesterov accelerated gradient algorithm, which applies to affine fixed point problems with non self-adjoint matrices, such as the ones appearing in the theory of Markov decision processes with discounted or mean payoff criteria. We characterize the spectra of matrices for which this algorithm does converge with an accelerated asymptotic rate. We also introduce a $d$th-order algorithm, and show that it yields a multiply accelerated rate under more demanding conditions on the spectrum. We subsequently apply these methods to develop accelerated schemes for non-linear fixed point problems arising from Markov decision processes. This is illustrated by numerical experiments.

math.OC↗

arXiv2021

Tropical linear regression and mean payoff games: or, how to measure the distance to equilibria

We study a tropical linear regression problem consisting in finding the best approximation of a set of points by a tropical hyperplane. We establish a strong duality theorem, showing that the value of this problem coincides with the maximal radius of a Hilbert's ball included in a tropical polyhedron. We also show that this regression problem is polynomial-time equivalent to mean payoff games. We illustrate our results by solving an inverse problem from auction theory. In this setting, a tropical hyperplane represents the set of equilibrium prices. Tropical linear regression allows us to quantify the distance of a market to the set of equilibria, and infer secret preferences of a decision maker.

math.CO↗

arXiv2019

Solving Ergodic Markov Decision Processes and Perfect Information Zero-sum Stochastic Games by Variance Reduced Deflated Value Iteration

Recently, Sidford, Wang, Wu and Ye (2018) developed an algorithm combining variance reduction techniques with value iteration to solve discounted Markov decision processes. This algorithm has a sublinear complexity when the discount factor is fixed. Here, we extend this approach to mean-payoff problems, including both Markov decision processes and perfect information zero-sum stochastic games. We obtain sublinear complexity bounds, assuming there is a distinguished state which is accessible from all initial states and for all policies. Our method is based on a reduction from the mean payoff problem to the discounted problem by a Doob h-transform, combined with a deflation technique. The complexity analysis of this algorithm uses at the same time the techniques developed by Sidford et al. in the discounted case and non-linear spectral theory techniques (Collatz-Wielandt characterization of the eigenvalue).

math.OC↗