arXiv subjects

Christophe Tribes

Publications and source records attributed to Christophe Tribes.

13 recordsLinked to original sources

arXiv2026

Parallel versions of the mesh adaptive direct search algorithm

This work surveys the different parallel variants of the mesh adaptive direct search (MADS) algorithm for constrained blackbox optimization. These problems can inherently imply high computational costs due to the possible large number of variables and multi-modality of the search space. In addition, the potential time-intensive nature and time heterogeneity of the blackboxes defining the problem prompts the need for efficient implementations. Parallelism emerges as an actionable solution to mitigate computation time, as modern computer systems rely on multi-core architecture. The reviewed methods employ diverse levels of parallelism and distinct parallel strategies to effectively tackle each aspect outlined above. The manuscript details the practical implementations, provides computational results, and offers insights into the advantages and limitations of each MADS parallel method.

math.OC↗

arXiv2026

Adaptive direct search algorithms with relaxable and quantifiable constraints

This work introduces ADS-PB, an extension of the Adaptive Direct Search (ADS) framework for solving constrained blackbox optimization problems. With ADS, iterates progress without relying on mesh structures or sufficient decrease conditions on the objective function value. Unlike the extreme barrier approach used in ADS, where only unrelaxable constraints are considered, the proposed method also handles quantifiable and relaxable constraints using a Progressive Barrier (PB) mechanism that exploits both constraint and objective function values. A convergence analysis of the proposed framework under mild assumptions is presented. The performance of the proposed method is assessed using sets of analytical and simulation-based constrained test problems and is compared with state-of-the-art blackbox optimization solvers, including the PB approach within the Mesh Adaptive Direct Search (MADS) framework.

math.OC↗

arXiv2026

Surrogate-based categorical neighborhoods for mixed-variable blackbox optimization

In simulation-based engineering, design choices are often obtained following the optimization of complex blackbox models. These models frequently involve mixed-variable domains with quantitative and categorical variables. Unlike quantitative variables, categorical variables lack an inherent structure, which makes them difficult to handle, especially in the presence of constraints. This work proposes a systematic approach to structure and model categorical variables in constrained mixed-variable blackbox optimization. Surrogate models of the objective and constraint functions are used to induce problem-specific categorical distances. From these distances, surrogate-based neighborhoods are constructed using notions of dominance from bi-objective optimization, jointly accounting for information from both the objective and the constraint functions. This study addresses the lack of automatic and constraint-aware categorical neighborhood construction in mixed-variable blackbox optimization. As a proof of concept, these neighborhoods are employed within CatMADS, an extension of the MADS algorithm for categorical variables. The surrogate models are Gaussian processes, and the resulting method is called CatMADS-GP. The method is benchmarked on the Cat-Suite collection of 60 mixed-variable optimization problems and compared against state-of-the-art solvers. Data profiles indicate that CatMADS-GP achieves superior performance for both unconstrained and constrained problems.

math.OC↗

arXiv2026

A penalty-interior point method combined with MADS for equality and inequality constrained optimization

This work introduces MADS-PIP, an efficient framework that integrates a penalty-interior point strategy into the mesh adaptive direct search (MADS) algorithm for solving nonsmooth blackbox optimization problems with general inequality and equality constraints. Inequality constraints are partitioned into two subsets: one treated via a logarithmic barrier applied to an aggregated interior constraint violation, and the other handled through an exterior quadratic penalty. All equality constraints are treated by the exterior penalty. A merit function defines a sequence of unconstrained subproblems, which are solved approximately using MADS, while a carefully designed update rule drives the penalty-barrier parameter to zero. In the nonsmooth setting, we establish convergence results ensuring feasibility for general constraints as well as Clarke stationarity for inequality-constrained problems. Computational experiments on both analytical test sets and challenging blackbox problems demonstrate that the proposed MADS-PIP algorithm is competitive with, and often outperforms, MADS with the progressive barrier strategy, particularly in the presence of equality constraints.

math.OC↗

arXiv2025

Adaptive direct search algorithms for constrained optimization

Two families of directional direct search methods have emerged in derivative-free and blackbox optimization (DFO and BBO), each based on distinct principles: Mesh Adaptive Direct Search (MADS) and Sufficient Decrease Direct Search (SDDS). MADS restricts trial points to a mesh and accepts any improvement, ensuring none are missed, but at the cost of restraining the placement of trial points. SDDS allows greater freedom by evaluating points anywhere in the space, but accepts only those yielding a sufficient decrease in the objective function value, which may lead to discarding improving points. This work introduces a new class of methods, Adaptive Direct Search (ADS), which uses a novel acceptance rule based on the so-called punctured space, avoiding both meshes and sufficient decrease conditions. ADS enables flexible search while addressing the limitations of MADS and SDDS, and retains the theoretical foundations of directional direct search. Computational results in constrained and unconstrained settings highlight its performance compared to both MADS and SDDS.

math.OC↗

arXiv2025

CatMADS: Mesh Adaptive Direct Search for constrained blackbox optimization with categorical variables

Solving optimization problems in which functions are blackboxes and variables involve different types poses significant theoretical and algorithmic challenges. Nevertheless, such settings frequently occur in simulation-based engineering design and machine learning. This paper extends the Mesh Adaptive Direct Search (MADS) algorithm to address mixed-variable problems with categorical, integer and continuous variables. MADS is a robust derivative-free optimization framework with a well-established convergence analysis for constrained quantitative problems. CatMADS generalizes MADS by incorporating categorical variables through distance-induced neighborhoods. A detailed convergence analysis of CatMADS is provided, with flexible choices balancing computational cost and local optimality strength. Four types of mixed-variable local minima are introduced, corresponding to progressively stronger notions of local optimality. CatMADS integrates the progressive barrier strategy for handling constraints, and ensures Clarke stationarity. An instance of \catmads employs cross-validation to construct problem-specific categorical distances. This instance is compared to state-of-the-art solvers on 32 mixed-variable problems, half of which are constrained. Data profiles show that CatMADS achieves the best results, demonstrating that the framework is empirically efficient in addition to having strong theoretical foundations.

math.OC↗

arXiv2025

Efficient search strategies for constrained multiobjective blackbox optimization

Multiobjective blackbox optimization deals with problems where the objective and constraint functions are the outputs of a numerical simulation. In this context, no derivatives are available, nor can they be approximated by finite differences, which precludes the use of classical gradient-based techniques. The DMulti-MADS algorithm implements a state-of-the-art direct search procedure for multiobjective blackbox optimization based on the mesh adaptive direct search (MADS) algorithm. Since its conception, many search strategies have been proposed to improve the practical efficiency of the single-objective MADS algorithm. Inspired by this previous research, this work proposes the integration of two search heuristics into the DMulti-MADS algorithm. The first uses quadratic models, built from previously evaluated points, which act as surrogates for the true objectives and constraints, to suggest new promising candidates. The second exploits the sampling strategy of the Nelder-Mead algorithm to explore the decision space for new non-dominated points. Computational experiments on analytical problems and three engineering applications show that the use of such search steps considerably improves the performance of the DMulti-MADS algorithm.

math.OC↗

arXiv2024

solar: A solar thermal power plant simulator for blackbox optimization benchmarking

This work introduces solar, a collection of ten optimization problem instances for benchmarking blackbox optimization solvers. The instances present different design aspects of a concentrated solar power plant simulated by blackbox numerical models. The type of variables (discrete or continuous), dimensionality, and number and types of constraints (including hidden constraints) differ across instances. Some are deterministic, others are stochastic with possibilities to execute several replications to control stochasticity. Most instances offer variable fidelity surrogates, two are biobjective and one is unconstrained. The solar plant model takes into account various subsystems: a heliostats field, a central cavity receiver (the receiver), a molten salt thermal energy storage, a steam generator and an idealized power block. Several numerical methods are implemented throughout the solar code and most of the executions are time-consuming. Great care was applied to guarantee reproducibility across platforms. The solar tool encompasses most of the characteristics that can be found in industrial and real-life blackbox optimization problems, all in an open-source and stand-alone code.

math.OC↗

arXiv2023

Hyperparameter Optimization for Large Language Model Instruction-Tuning

The fine-tuning of Large Language Models (LLMs) has enabled them to recently achieve milestones in natural language processing applications. The emergence of ever larger LLMs has paved the way for more efficient fine-tuning methods. Among these, the Low-Rank Adaptation (LoRA) method keeps most of the weights of the pre-trained LLM frozen while introducing a low-rank decomposition of the weight matrix, enabling the tuning of only a very small proportion of the network. The performance on downstream tasks of models fine-tuned with LoRA heavily relies on a set of hyperparameters including the rank of the decomposition. In this work, we investigate the choice of these hyperparameters through two main blackbox optimization (BBO) techniques. We examine the whole pipeline of performing fine-tuning and validation on a pre-trained LLM as a blackbox and efficiently explore the space of hyperparameters with the \nomad algorithm, achieving a boost in performance and human alignment of the tuned model.

cs.CL↗

arXiv2023

Efficient Training Under Limited Resources

Training time budget and size of the dataset are among the factors affecting the performance of a Deep Neural Network (DNN). This paper shows that Neural Architecture Search (NAS), Hyper Parameters Optimization (HPO), and Data Augmentation help DNNs perform much better while these two factors are limited. However, searching for an optimal architecture and the best hyperparameter values besides a good combination of data augmentation techniques under low resources requires many experiments. We present our approach to achieving such a goal in three steps: reducing training epoch time by compressing the model while maintaining the performance compared to the original model, preventing model overfitting when the dataset is small, and performing the hyperparameter tuning. We used NOMAD, which is a blackbox optimization software based on a derivative-free algorithm to do NAS and HPO. Our work achieved an accuracy of 86.0 % on a tiny subset of Mini-ImageNet at the ICLR 2021 Hardware Aware Efficient Training (HAET) Challenge and won second place in the competition. The competition results can be found at haet2021.github.io/challenge and our source code can be found at github.com/DouniaLakhmiri/ICLR\_HAET2021.

cs.LG↗

arXiv2023

Scaling Deep Networks with the Mesh Adaptive Direct Search algorithm

Deep neural networks are getting larger. Their implementation on edge and IoT devices becomes more challenging and moved the community to design lighter versions with similar performance. Standard automatic design tools such as \emph{reinforcement learning} and \emph{evolutionary computing} fundamentally rely on cheap evaluations of an objective function. In the neural network design context, this objective is the accuracy after training, which is expensive and time-consuming to evaluate. We automate the design of a light deep neural network for image classification using the \emph{Mesh Adaptive Direct Search}(MADS) algorithm, a mature derivative-free optimization method that effectively accounts for the expensive blackbox nature of the objective function to explore the design space, even in the presence of constraints.Our tests show competitive compression rates with reduced numbers of trials.

stat.ML↗

arXiv2021

NOMAD version 4: Nonlinear optimization with the MADS algorithm

NOMAD is software for optimizing blackbox problems. In continuous development since 2001, it constantly evolved with the integration of new algorithmic features published in scientific publications. These features are motivated by real applications encountered by industrial partners. The latest major release of NOMAD, version 3, dates from 2008. Minor releases are produced as new features are incorporated. The present work describes NOMAD 4, a complete redesign of the previous version, with a new architecture providing more flexible code, added functionalities and reusable code. We introduce algorithmic components, which are building blocks for more complex algorithms, and can initiate other components, launch nested algorithms, or perform specialized tasks. They facilitate the implementation of new ideas, including the MegaSearchPoll component, warm and hot restarts, and a revised version of the PSD-MADS algorithm. Another main improvement of NOMAD 4 is the usage of parallelism, to simultaneously compute multiple blackbox evaluations, and to maximize usage of available cores. Running different algorithms, tuning their parameters, and comparing their performance for optimization is simpler than before, while overall optimization performance is maintained between versions 3 and 4. NOMAD is freely available at www.gerad.ca/nomad and the whole project is visible at github.com/bbopt/nomad.

math.OC↗

arXiv2019

HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search

The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time consuming process that is often described as a "dark art". This explains the necessity of automating the calibration of these hyperparameters. Derivative-free optimization is a field that develops methods designed to optimize time consuming functions without relying on derivatives. This work introduces the HyperNOMAD package, an extension of the NOMAD software that applies the MADS algorithm [7] to simultaneously tune the hyperparameters responsible for both the architecture and the learning process of a deep neural network (DNN), and that allows for an important flexibility in the exploration of the search space by taking advantage of categorical variables. This new approach is tested on the MNIST and CIFAR-10 data sets and achieves results comparable to the current state of the art.

cs.LG↗