arXiv subjects

Victor Priser

Publications and source records attributed to Victor Priser.

5 recordsLinked to original sources

arXiv2026

Importance Sampling Optimization with Laplace Principle

Grid search and random search are widely used techniques for hyperparameter tuning in machine learning, especially when gradient information is unavailable. In these methods, a finite set of candidate configurations is evaluated, and the best-performing one is selected. We propose a simple and computationally inexpensive refinement of this paradigm: instead of selecting a single best point, we form a weighted average of the evaluated configurations, where the weights are chosen using an importance sampling scheme inspired by the Laplace principle. This scheme can be implemented as a post-processing step on top of a random search, with no additional function evaluations. We also propose an iterative variant, where the sampling distributions are chosen adaptively to generate new candidate points around the previous estimate, in the spirit of Evolution Strategy (ES) methods. In a general non-convex setting, we show that, after n evaluations, the error of the proposed methods is of smaller order than n -2/(d+2) . This compares favorably to random search or grid search rates of n -1/d as soon as d > 2. We illustrate the practical benefits of this averaging strategy on several examples.

math.OC↗

arXiv2025

Consensus-Based Optimization Beyond Finite-Time Analysis

We analyze a zeroth-order particle algorithm for the global optimization of a non-convex function, focusing on a variant of Consensus-Based Optimization (CBO) with small but fixed noise intensity. Unlike most previous studies restricted to finite horizons, we investigate its long-time behavior with fixed parameters. In the mean-field limit, a quantitative Laplace principle shows exponential convergence to a neighborhood of the minimizer x * . For finitely many particles, a block-wise analysis yields explicit error bounds: individual particles achieve long-time consistency near x * , and the global best particle converge to x * . The proof technique combines a quantitative Laplace principle with block-wise control of Wasserstein distances, avoiding the exponential blow-up typical of Gr{\"o}nwall-based estimates.

math.OC↗

arXiv2024

Stochastic mirror descent for nonparametric adaptive importance sampling

This paper addresses the problem of approximating an unknown probability distribution with density $f$ -- which can only be evaluated up to an unknown scaling factor -- with the help of a sequential algorithm that produces at each iteration $n\geq 1$ an estimated density $q_n$.The proposed method optimizes the Kullback-Leibler divergence using a mirror descent (MD) algorithm directly on the space of density functions, while a stochastic approximation technique helps to manage between algorithm complexity and variability. One of the key innovations of this work is the theoretical guarantee that is provided for an algorithm with a fixed MD learning rate $\eta \in (0,1 )$. The main result is that the sequence $q_n$ converges almost surely to the target density $f$ uniformly on compact sets. Through numerical experiments, we show that fixing the learning rate $\eta \in (0,1 )$ significantly improves the algorithm's performance, particularly in the context of multi-modal target distributions where a small value of $\eta$ allows to increase the chance of finding all modes. Additionally, we propose a particle subsampling method to enhance computational efficiency and compare our method against other approaches through numerical experiments.

math.ST↗

arXiv2024

Long-time asymptotics of noisy SVGD outside the population limit

Stein Variational Gradient Descent (SVGD) is a widely used sampling algorithm that has been successfully applied in several areas of Machine Learning. SVGD operates by iteratively moving a set of interacting particles (which represent the samples) to approximate the target distribution. Despite recent studies on the complexity of SVGD and its variants, their long-time asymptotic behavior (i.e., after numerous iterations ) is still not understood in the finite number of particles regime. We study the long-time asymptotic behavior of a noisy variant of SVGD. First, we establish that the limit set of noisy SVGD for large is well-defined. We then characterize this limit set, showing that it approaches the target distribution as increases. In particular, noisy SVGD provably avoids the variance collapse observed for SVGD. Our approach involves demonstrating that the trajectories of noisy SVGD closely resemble those described by a McKean-Vlasov process.

cs.LG↗

arXiv2024

Long run convergence of discrete-time interacting particle systems of the McKean-Vlasov type

We consider a discrete-time system of n coupled random vectors, a.k.a. interacting particles. The dynamics involve a vanishing step size, some random centered perturbations, and a mean vector field which induces the coupling between the particles. We study the doubly asymptotic regime where both the number of iterations and the number n of particles tend to infinity, without any constraint on the relative rates of convergence of these two parameters. We establish that the empirical measure of the interpolated trajectories of the particles converges in probability, in an ergodic sense, to the set of recurrent Mc-Kean-Vlasov distributions. A first application example is the granular media equation, where the particles are shown to converge to a critical point of the Helmholtz energy. A second example is the convergence of stochastic gradient descent to the global minimizer of the risk, in a wide two-layer neural networks using random features.

math.PR↗