arXiv subjects

Samuel Muller

Publications and source records attributed to Samuel Muller.

At least 19 recordsLinked to original sources

arXiv2026

Response-guided knockoffs for directional FDR control in linear models

We consider the problem of feature selection in linear models with finite-sample control of the false discovery rate (FDR). While existing knockoff-based methods control the directional FDR, which penalises incorrect sign estimates, they do not target discoveries in a pre-specified direction, and their knockoff constructions are entirely response-agnostic. We introduce the response-guided knockoff filter, which leverages a noise-perturbed version of the response to guide knockoff construction toward features likely to have the target sign, while provably controlling the directional FDR. The method operates under a weaker sample-size requirement $n > p + 2$, compared to $n \geq 2p$ required by existing fixed-X generators. Simulations and HIV drug resistance experiments demonstrate power gains over existing methods.

stat.ME↗

arXiv2026

Robust Best Subset Selection via Fast Approximate MM-Estimation

Best subset selection procedures typically rely on a squared error loss, where a small number of outlying observations may distort the entire solution path. Replacing this loss with a robust alternative, such as the MM-estimator robust model selection criterion, is computationally prohibitive, as it requires an iterative fit across near-exhaustive candidate model spaces. To preserve the robust properties of the MM-estimator while avoiding its repeated iterative computation, we introduce FAMM, a Fast Approximate MM-Estimator based on a weighted least squares fit with weights obtained from a full data MM-estimator. Although the resulting estimator is no longer an MM-estimator, we prove that model selection consistency is retained when aligning the weights of the selection criterion with those of the estimator. We embed FAMM within COMBSS, whose continuous relaxation of the discrete best subset search enables scalable best subset selection for large numbers of predictor variables. Consequently, robust model selection and best subset selection with large and contaminated datasets are made feasible.

stat.ME↗

arXiv2026

2D Stability Selection: Design Jittering for Doubly Stable Feature Selection

We study feature selection in high-dimensional regression under two distinct sources of instability: sampling variability and measurement error in the design matrix. Stability Selection addresses the former through sub-sampling and aggregation, but does not explicitly stress-test robustness to noisy predictors. We introduce doubly stable feature selection, a perturb-and-aggregate framework that targets features whose inclusion is stable both across randomization and across increasing levels of design noise. The method injects controlled additive noise into the design matrix, fits a fixed base selector such as the Lasso on the perturbed data, and aggregates selection frequencies. Sweeping over a grid of noise levels yields a stability path that summarizes robustness to measurement error while using the full sample size and isolating the effect of design perturbations. On the theory side, we show that classical model-selection conditions are preserved under sufficiently small perturbations, with a high-probability extension for Gaussian noise. Empirically, experiments on synthetic and real datasets show improved robustness compared with Stability Selection and standard base selectors.

stat.ME↗

arXiv2026

Parsimonious Subset Selection for Generalized Linear Models with Biomedical Applications

High-dimensional biomedical studies require models that are simultaneously accurate, sparse, and interpretable, yet exact best subset selection for generalized linear models is computationally intractable. We develop a scalable method that combines a continuous Boolean relaxation of the subset problem with a Frank--Wolfe algorithm driven by envelope gradients. The resulting method, which we refer to as COMBSS-GLM, is simple to implement, requires one penalized generalized linear model fit per iteration, and produces sparse models along a model-size path. Theoretically, we identify a curvature-based parameter regime in which the relaxed objective is concave in the selection weights, implying that global minimizers occur at binary corners. Empirically, in logistic and multinomial simulations across low- and high-dimensional correlated settings, the proposed method consistently improves variable-selection quality relative to established penalised likelihood competitors while maintaining strong predictive performance. In biomedical applications, it recovers established loci in a binary-outcome rice genome-wide association study and achieves perfect multiclass test accuracy on the Khan SRBCT cancer dataset using a small subset of genes. Open-source implementations are available in R at https://github.com/benoit-liquet/COMBSS-GLM-R and in Python at https://github.com/saratmoka/COMBSS-GLM-Python.

stat.ME↗

arXiv2025

A Stable Lasso

The Lasso has been widely used as a method for variable selection, valued for its simplicity and empirical performance. However, Lasso's selection stability deteriorates in the presence of correlated predictors. Several approaches have been developed to mitigate this limitation. In this paper, we provide a brief review of existing approaches, highlighting their limitations. We then propose a simple technique to improve the selection stability of Lasso by integrating a weighting scheme into the Lasso penalty function, where the weights are defined as an increasing function of a correlation-adjusted ranking that reflects the predictive power of predictors. Empirical evaluations on both simulated and real-world datasets demonstrate the efficacy of the proposed method. Additional numerical results demonstrate the effectiveness of the proposed approach in stabilizing other regularization-based selection methods, indicating its potential as a general-purpose solution.

stat.ME↗

arXiv2025

A Single Index Approach to Integrated Species Distribution Modeling for Fisheries Abundance Data

In fisheries ecology, species abundance data are often collected by multiple surveys, each with unique characteristics. This article is motivated by a dataset of Atlantic sea scallop abundance records along the northeast coast of the United States, collected from two bottom trawl surveys which cover a larger spatial domain but have low catch efficiency, and a dredge survey which is more efficient but more bounded in domain. Over the past decade, integrated species distribution models (ISDMs) that include common environmental effects along with correlated survey-specific spatial fields have been used to incorporate information from multiple surveys. While flexible, ISDMs can be susceptible to overfitting, which can complicate interpretability of the shared environmental effects, and potentially lead to poor predictive performance. To overcome these drawbacks, we introduce a novel single index ISDM, built from a single index (with spatial random effects) that represents a latent measure of the true species distribution, and survey-specific catch efficiency functions which map the single index to the survey-specific expected catch. In this article, these functions are constructed via logistic functions or semiparametric spline-based functions. Simulations and application to the motivating sea scallop abundance data demonstrate that the proposed single index ISDM offers more meaningful interpretations of the environmental effects and survey catch efficiency differences, while achieving similar to or better predictive performance than existing ISDMs.

stat.AP↗

arXiv2025

The Lasso Distribution: Properties, Sampling Methods, and Applications in Bayesian Lasso Regression

In this paper, we introduce a new probability distribution, the Lasso distribution. We derive several fundamental properties of the distribution, including closed-form expressions for its moments and moment-generating function. Additionally, we present an efficient and numerically stable algorithm for generating random samples from the distribution, facilitating its use in both theoretical and applied settings. We establish that the Lasso distribution belongs to the exponential family. A direct application of the Lasso distribution arises in the context of an existing Gibbs sampler, where the full conditional distribution of each regression coefficient follows this distribution. This leads to a more computationally efficient and theoretically grounded sampling scheme. To facilitate the adoption of our methodology, we provide an R package, BayesianLasso, available on CRAN, implementing the proposed methods. Our findings offer new insights into the probabilistic structure underlying the Lasso penalty and provide practical improvements in Bayesian inference for high-dimensional regression problems.

stat.CO↗

arXiv2025

Data-Adaptive Automatic Threshold Calibration for Stability Selection

Stability selection has gained popularity as a method for enhancing the performance of variable selection algorithms while controlling false discovery rates. However, achieving these desirable properties depends on correctly specifying the stable threshold parameter, which can be challenging. An arbitrary choice of this parameter can substantially alter the set of selected variables, as the variables' selection probabilities are inherently data-dependent. To address this issue, we propose Exclusion Automatic Threshold Selection (EATS), a data-adaptive algorithm that streamlines stability selection by automating the threshold specification process. EATS initially filters out potential noise variables using an exclusion probability threshold, derived from applying stability selection to a randomly shuffled version of the dataset. Following this, EATS selects the stable threshold parameter using the elbow method, balancing the marginal utility of including additional variables against the risk of selecting superfluous variables. We evaluate our approach through an extensive simulation study, benchmarking across commonly used variable selection algorithms and static stable threshold values.

stat.ME↗

arXiv2025

Stability Selection via Variable Decorrelation

The Lasso is a prominent algorithm for variable selection. However, its instability in the presence of correlated variables in the high-dimensional setting is well-documented. Although previous research has attempted to address this issue by modifying the Lasso loss function, this paper introduces an approach that simplifies the data processed by Lasso. We propose that decorrelating variables before applying the Lasso improves the stability of variable selection regardless of the direction of correlation among predictors. Furthermore, we highlight that the irrepresentable condition, which ensures consistency for the Lasso, is satisfied after variable decorrelation under two assumptions. In addition, by noting that the instability of the Lasso is not limited to high-dimensional settings, we demonstrate the effectiveness of the proposed approach for low-dimensional data. Finally, we present empirical results that indicate the efficacy of the proposed method across different variable selection techniques, highlighting its potential for broader application. The DVS R package is developed to facilitate the implementation of the methodology proposed in this paper.

stat.ME↗

arXiv2025

A Scalable Gradient-Based Optimization Framework for Sparse Minimum-Variance Portfolio Selection

Portfolio optimization involves selecting asset weights to minimize a risk-reward objective, such as the portfolio variance in the classical minimum-variance framework. Sparse portfolio selection extends this by imposing a cardinality constraint: only $k$ assets from a universe of $p$ may be included. The standard approach models this problem as a mixed-integer quadratic program and relies on commercial solvers to find the optimal solution. However, the computational costs of such methods increase exponentially with $k$ and $p$, making them too slow for problems of even moderate size. We propose a fast and scalable gradient-based approach that transforms the combinatorial sparse selection problem into a constrained continuous optimization task via Boolean relaxation, while preserving equivalence with the original problem on the set of binary points. Our algorithm employs a tunable parameter that transmutes the auxiliary objective from a convex to a concave function. This allows a stable convex starting point, followed by a controlled path toward a sparse binary solution as the tuning parameter increases and the objective moves toward concavity. In practice, our method matches commercial solvers in asset selection for most instances and, in rare instances, the solution differs by a few assets whilst showing a negligible error in portfolio variance.

stat.ML↗

arXiv2024

High-dimensional sliced inverse regression with endogeneity

Sliced inverse regression (SIR) is a popular sufficient dimension reduction method that identifies a few linear transformations of the covariates without losing regression information with the response. In high-dimensional settings, SIR can be combined with sparsity penalties to achieve sufficient dimension reduction and variable selection simultaneously. Nevertheless, both classical and sparse estimators assume the covariates are exogenous. However, endogeneity can arise in a variety of situations, such as when variables are omitted or are measured with error. In this article, we show such endogeneity invalidates SIR estimators, leading to inconsistent estimation of the true central subspace. To address this challenge, we propose a two-stage Lasso SIR estimator, which first constructs a sparse high-dimensional instrumental variables model to obtain fitted values of the covariates spanned by the instruments, and then applies SIR augmented with a Lasso penalty on these fitted values. We establish theoretical bounds for the estimation and selection consistency of the true central subspace for the proposed estimators, allowing the number of covariates and instruments to grow exponentially with the sample size. Simulation studies and applications to two real-world datasets in nutrition and genetics illustrate the superior empirical performance of the two-stage Lasso SIR estimator compared with existing methods that disregard endogeneity and/or nonlinearity in the outcome model.

stat.ME↗

arXiv2024

Random Effects Misspecification and its Consequences for Prediction in Generalized Linear Mixed Models

When fitting generalized linear mixed models (GLMMs), one important decision to make relates to the choice of the random effects distribution. As the random effects are unobserved, misspecification of this distribution is a real possibility. In this article, we investigate the consequences of random effects misspecification for point prediction and prediction inference in GLMMs, a topic on which there is considerably less research compared to consequences for parameter estimation and inference. We use theory, simulation, and a real application to explore the effect of using the common normality assumption for the random effects distribution when the correct specification is a mixture of normal distributions, focusing on the impacts on point prediction, mean squared prediction errors (MSEPs), and prediction intervals. We found that the optimal shrinkage is different under the two random effect distributions, so is impacted by misspecification. The unconditional MSEPs for the random effects are almost always larger under the misspecified normal random effects distribution, especially when cluster sizes are small. Results for the MSEPs conditional on the random effects are more complicated, but they remain generally larger under the misspecified distribution when the true random effect is close to the mean of one of the component distributions in the true mixture distribution. Results for prediction intervals indicate that overall coverage probability is not greatly impacted by misspecification.

stat.ME↗

arXiv2024

On the Selection Stability of Stability Selection and Its Applications

Stability selection is a widely adopted resampling-based framework for high-dimensional variable selection. This paper seeks to broaden the use of an established stability estimator to evaluate the overall stability of the stability selection results, moving beyond single-variable analysis. We suggest that the stability estimator offers two advantages: it can serve as a reference to reflect the robustness of the results obtained, and it can help identify a Pareto optimal regularization value to improve stability. By determining the regularization value, we calibrate key stability selection parameters, namely, the decision-making threshold and the expected number of falsely selected variables, within established theoretical bounds. In addition, the convergence of stability values over successive sub-samples sheds light on the required number of sub-samples addressing a notable gap in prior studies. The \texttt{stabplot} R package is developed to facilitate the use of the methodology featured in this paper.

stat.ME↗

arXiv2024

Bayesian Stability Selection and Inference on Selection Probabilities

Stability selection is a versatile framework for structure estimation and variable selection in high-dimensional setting, primarily grounded in frequentist principles. In this paper, we propose an enhanced methodology that integrates Bayesian analysis to refine the inference of selection probabilities within the stability selection framework. Traditional approaches rely on selection frequencies for decision-making, often disregarding domain-specific knowledge. Our methodology uses prior information to derive posterior distributions of selection probabilities, thereby improving both inference and decision-making. We present a two-step process for engaging with domain experts, enabling statisticians to construct prior distributions informed by expert knowledge while allowing experts to control the weight of their input on the final results. Using posterior distributions, we offer Bayesian credible intervals to quantify uncertainty in the variable selection process. Furthermore, we demonstrate how the integration of prior knowledge reduces the variance of selection probabilities, thereby improving the stability of decision-making. Our approach preserves the versatility of stability selection and is suitable for a broad range of structure estimation challenges.

stat.ME↗

arXiv2024

Air-HOLP: Adaptive Regularized Feature Screening for High Dimensional Correlated Data

Handling high-dimensional datasets presents substantial computational challenges, particularly when the number of features far exceeds the number of observations and when features are highly correlated. A modern approach to mitigate these issues is feature screening. In this work, the High-dimensional Ordinary Least-squares Projection (HOLP) feature screening method is advanced by employing adaptive ridge regularization. The impact of the ridge tuning parameter on the Ridge-HOLP method is examined and Adaptive iterative ridge-HOLP (Air-HOLP) is proposed, a data-adaptive advance to Ridge-HOLP where the ridge-regularization tuning parameter is selected iteratively and optimally for better feature screening performance. The proposed method addresses the challenges of tuning parameter selection in high dimensions by offering a computationally efficient and stable alternative to traditional methods like bootstrapping and cross-validation. Air-HOLP is evaluated using simulated data and a prostate cancer genetic dataset. The empirical results demonstrate that Air-HOLP has improved performance over a large range of simulation settings. We provide R codes implementing the Air-HOLP feature screening method and integrating it into existing feature screening methods that utilize the HOLP formula.

stat.ME↗

arXiv2024

Best Subset Solution Path for Linear Dimension Reduction Models using Continuous Optimization

The selection of best variables is a challenging problem in supervised and unsupervised learning, especially in high dimensional contexts where the number of variables is usually much larger than the number of observations. In this paper, we focus on two multivariate statistical methods: principal components analysis and partial least squares. Both approaches are popular linear dimension-reduction methods with numerous applications in several fields including in genomics, biology, environmental science, and engineering. In particular, these approaches build principal components, new variables that are combinations of all the original variables. A main drawback of principal components is the difficulty to interpret them when the number of variables is large. To define principal components from the most relevant variables, we propose to cast the best subset solution path method into principal component analysis and partial least square frameworks. We offer a new alternative by exploiting a continuous optimization algorithm for best subset solution path. Empirical studies show the efficacy of our approach for providing the best subset solution path. The usage of our algorithm is further exposed through the analysis of two real datasets. The first dataset is analyzed using the principle component analysis while the analysis of the second dataset is based on partial least square framework.

stat.ME↗

arXiv2023

CR-Lasso: Robust cellwise regularized sparse regression

Cellwise contamination remains a challenging problem for data scientists, particularly in research fields that require the selection of sparse features. Traditional robust methods may not be feasible nor efficient in dealing with such contaminated datasets. We propose CR-Lasso, a robust Lasso-type cellwise regularization procedure that performs feature selection in the presence of cellwise outliers by minimising a regression loss and cell deviation measure simultaneously. To evaluate the approach, we conduct empirical studies comparing its selection and prediction performance with several sparse regression methods. We show that CR-Lasso is competitive under the settings considered. We illustrate the effectiveness of the proposed method on real data through an analysis of a bone mineral density dataset.

stat.ME↗

arXiv2022

COMBSS: Best Subset Selection via Continuous Optimization

The problem of best subset selection in linear regression is considered with the aim to find a fixed size subset of features that best fits the response. This is particularly challenging when the total available number of features is very large compared to the number of data samples. Existing optimal methods for solving this problem tend to be slow while fast methods tend to have low accuracy. Ideally, new methods perform best subset selection faster than existing optimal methods but with comparable accuracy, or, being more accurate than methods of comparable computational speed. Here, we propose a novel continuous optimization method that identifies a subset solution path, a small set of models of varying size, that consists of candidates for the single best subset of features, that is optimal in a specific sense in linear regression. Our method turns out to be fast, making the best subset selection possible when the number of features is well in excess of thousands. Because of the outstanding overall performance, framing the best subset selection challenge as a continuous optimization problem opens new research directions for feature extraction for a large variety of regression models.

stat.ME↗