arXiv subjects

Gabriel Clara

Publications and source records attributed to Gabriel Clara.

4 recordsLinked to original sources

arXiv2025

The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights

We analyze gradient descent with randomly weighted data points in a linear regression model, under a generic weighting distribution. This includes various forms of stochastic gradient descent, importance sampling, but also extends to weighting distributions with arbitrary continuous values, thereby providing a unified framework to analyze the impact of various kinds of noise on the training trajectory. We characterize the implicit regularization induced through the random weighting, connect it with weighted linear regression, and derive non-asymptotic bounds for convergence in first and second moments. Leveraging geometric moment contraction, we also investigate the stationary distribution induced by the added noise. Based on these results, we discuss how specific choices of weighting distribution influence both the underlying optimization problem and statistical properties of the resulting estimator, as well as some examples for which weightings that lead to fast convergence cause bad statistical performance.

stat.ML↗

arXiv2025

Training Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization

We analyze the landscape and training dynamics of diagonal linear networks in a linear regression task, with the network parameters being perturbed by isotropic normal noise during training. The addition of such noise may be interpreted as a stochastic form of sharpness-aware minimization (SAM) and we prove several results that relate its action on the underlying landscape and training dynamics to the sharpness of the loss. In particular, the noise induces a weighted mixture of fractional norm penalties on the network parameters, which forces the individual layers to balance at a fast rate and changes the underlying landscape to favor solutions that result from a shrinkage-thresholding operator applied to the true parameter. We show that balancing the layers equates to minimizing the average sharpness, as well as the trace of the Hessian matrix, among all possible factorizations of the same linear predictor. Further, we characterize how the noise level of the normal perturbations acts as a regularization parameter, with exact descriptions of its effect on shrinkage, thresholding, and balancing speed.

cs.LG↗

arXiv2023

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model

We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator

math.ST↗

arXiv2021

Spike and slab variational Bayes for high dimensional logistic regression

Variational Bayes (VB) is a popular scalable alternative to Markov chain Monte Carlo for Bayesian inference. We study a mean-field spike and slab VB approximation of widely used Bayesian model selection priors in sparse high-dimensional logistic regression. We provide non-asymptotic theoretical guarantees for the VB posterior in both $\ell_2$ and prediction loss for a sparse truth, giving optimal (minimax) convergence rates. Since the VB algorithm does not depend on the unknown truth to achieve optimality, our results shed light on effective prior choices. We confirm the improved performance of our VB algorithm over common sparse VB approaches in a numerical study.

stat.ML↗