arXiv subjects

Mahsa Yousefi

Publications and source records attributed to Mahsa Yousefi.

7 recordsLinked to original sources

arXiv2026

A Direction Adaptation Evaluation Strategy for Noisy Derivative-Free Optimization

In this paper, we develop a direction adaptation evolution strategy (DAES) -- a new MAES-type method -- for noisy derivative-free optimization, designed to reconcile the population-based search mechanisms of evolution strategies with rigorous complexity analysis. Unlike standard MAES schemes, DAES fixes the adaptation matrix to the identity and replaces matrix adaptation with a structured direction-generation mechanism based on symmetric sampling, joint sorting-selection of noisy function values, three-group recombination, and a new triangular search direction. Specifically, candidates are sampled along paired positive and negative directions, their inexact function values are jointly ranked, and the reordered directions are partitioned into three groups to construct three recombination points whose geometry defines the triangular direction. A signed sufficient-decrease search and extrapolation mechanism is then applied along this direction. This structure yields a population-based MAES-type algorithm that retains competitive practical behavior while being amenable to nonasymptotic analysis under noisy evaluations. We establish high-probability complexity bounds for nonconvex, convex, and strongly convex objective functions and derive corresponding guarantees at the noise-limited accuracy level. To the best of our knowledge, these results provide the first high-probability complexity guarantees for a noisy MAES-type derivative-free method with this direction-adaptation structure. Finally, numerical experiments on the 655 prince test problems from the BARON collection compare DAES with the advanced MAES-type solver MADFO and show a favorable trade-off between evaluation efficiency and ultimate robustness.

math.OC↗

arXiv2026

A Projected Stochastic Gradient Method for Finite-Sum Problems with Linear Equality Constraints

A stochastic gradient method for finite-sum minimization subject to deterministic linear constraints is proposed and analyzed. The procedure presented adapts the projected gradient method on convex set to the use of both a stochastic gradient and a possibly inexact projection map. Under standard assumptions in the field of stochastic gradient methods, we provide theoretical results in agreement with the theory for unconstrained problems. Numerical results are presented to show the practical behavior of the procedure.

math.OC↗

arXiv2024

Fully stochastic trust-region methods with Barzilai-Borwein steplengths

We investigate stochastic gradient methods and stochastic counterparts of the Barzilai-Borwein steplengths and their application to finite-sum minimization problems. Our proposal is based on the Trust-Region-ish (TRish) framework introduced in [F. E. Curtis, K. Scheinberg, R. Shi, {\it A stochastic trust region algorithm based on careful step normalization}, Informs Journal on Optimization, 1, 2019]. The new framework, named TRishBB, aims to enhance the performance of TRish and at reducing the computational cost of the second-order TRish variant. We propose three different methods belonging to the TRishBB framework and present the convergence analysis for possibly nonconvex objective functions, considering biased and unbiased gradient approximations. Our analysis requires neither diminishing step-sizes nor full gradient evaluation. The numerical experiments in machine learning applications demonstrate the effectiveness of applying the Barzilai-Borwein steplength with stochastic gradients and show improved testing accuracy compared to the TRish method.

math.OC↗

arXiv2024

Soil analysis with machine-learning-based processing of stepped-frequency GPR field measurements: Preliminary study

Ground Penetrating Radar (GPR) has been widely studied as a tool for extracting soil parameters relevant to agriculture and horticulture. When combined with Machine Learning (ML) methods, air-coupled Stepped Frequency Continuous Wave Ground Penetrating Radar (SFCW GPR) measurements could offer a cost-effective way to obtain depth-resolved soil data. As a first step of our study in this direction, we conducted an extensive field survey using a tractor-mounted air-coupled SFCW GPR instrument. Leveraging ML-based data processing, we evaluate the GPR instrument's ability by predicting the apparent electrical conductivity (ECaR) measured by a co-recorded Electromagnetic Induction (EMI) instrument. The large-scale field measurement campaign with 3472 co-registered and geo-located GPR and EMI data samples distributed over approximately 6600 square meters was performed on a golf course. This terrain offers high surface homogeneity but also presents the challenge of subtle soil parameter variations. Based on the results, we discuss challenges in this multi-sensor regression setting and propose the use of the nugget-to-sill ratio as a performance metric for evaluating ML models in agricultural field survey applications.

eess.SP↗

arXiv2023

On the efficiency of Stochastic Quasi-Newton Methods for Deep Learning

While first-order methods are popular for solving optimization problems that arise in large-scale deep learning problems, they come with some acute deficiencies. To diminish such shortcomings, there has been recent interest in applying second-order methods such as quasi-Newton based methods which construct Hessians approximations using only gradient information. The main focus of our work is to study the behaviour of stochastic quasi-Newton algorithms for training deep neural networks. We have analyzed the performance of two well-known quasi-Newton updates, the limited memory Broyden-Fletcher-Goldfarb-Shanno (BFGS) and the Symmetric Rank One (SR1). This study fills a gap concerning the real performance of both updates and analyzes whether more efficient training is obtained when using the more robust BFGS update or the cheaper SR1 formula which allows for indefinite Hessian approximations and thus can potentially help to better navigate the pathological saddle points present in the non-convex loss functions found in deep learning. We present and discuss the results of an extensive experimental study which includes the effect of batch normalization and network's architecture, the limited memory parameter, the batch size and the type of sampling strategy. we show that stochastic quasi-Newton optimizers are efficient and able to outperform in some instances the well-known first-order Adam optimizer run with the optimal combination of its numerous hyperparameters.

cs.LG↗

arXiv2023

Majorization-Minimization for sparse SVMs

Several decades ago, Support Vector Machines (SVMs) were introduced for performing binary classification tasks, under a supervised framework. Nowadays, they often outperform other supervised methods and remain one of the most popular approaches in the machine learning arena. In this work, we investigate the training of SVMs through a smooth sparse-promoting-regularized squared hinge loss minimization. This choice paves the way to the application of quick training methods built on majorization-minimization approaches, benefiting from the Lipschitz differentiabililty of the loss function. Moreover, the proposed approach allows us to handle sparsity-preserving regularizers promoting the selection of the most significant features, so enhancing the performance. Numerical tests and comparisons conducted on three different datasets demonstrate the good performance of the proposed methodology in terms of qualitative metrics (accuracy, precision, recall, and F 1 score) as well as computational cost.

cs.LG↗

arXiv2023

A non-monotone trust-region method with noisy oracles and additional sampling

In this work, we introduce a novel stochastic second-order method, within the framework of a non-monotone trust-region approach, for solving the unconstrained, nonlinear, and non-convex optimization problems arising in the training of deep neural networks. The proposed algorithm makes use of subsampling strategies which yield noisy approximations of the finite sum objective function and its gradient. To effectively control the resulting approximation error, we introduce an adaptive sample size strategy based on inexpensive additional sampling. Depending on the estimated progress of the algorithm, this can yield sample size scenarios ranging from mini-batch to full sample functions. We provide convergence analysis for all possible scenarios and show that the proposed method achieves almost sure convergence under standard assumptions for the trust-region framework. We report numerical experiments showing that the proposed algorithm outperforms its state-of-the-art counterpart in deep neural network training for image classification and regression tasks while requiring a significantly smaller number of gradient evaluations.

math.OC↗