arXiv subjects

Trevor Hastie

Publications and source records attributed to Trevor Hastie.

At least 19 recordsLinked to original sources

arXiv2026

Non-smooth optimization meets automated material model discovery

Automated material model discovery disrupts the tedious and time-consuming cycle of iteratively calibrating and modifying manually designed models. Non-smooth L1-norm regularization is the backbone of automated model discovery; however, the current literature on automated material model discovery offers limited insights into the robust and efficient minimization of non-smooth objective functions. In this work, we examine the minimization of functions of the form f(w) + a ||w||_1, where w are the material model parameters, f is a metric that quantifies the mismatch between the material model and the observed data, and a is a regularization parameter that determines the sparsity of the solution. We investigate both the straightforward case where f is quadratic and the more complex scenario where it is non-quadratic or even non-convex. Importantly, we do not only focus on methods that solve the sparse regression problem for a given value of the regularization parameter a, but propose methods to efficiently compute the entire regularization path, facilitating the selection of a suitable a. Specifically, we present four algorithms and discuss their roles for automated material model discovery in mechanics: First, we recapitulate a well-known coordinate descent algorithm that solves the minimization problem assuming that f is quadratic for a given value of a, also known as the LASSO. Second, we discuss the algorithm LARS, which automatically determines the critical values of a, at which material parameters in w are set to zero. Third, we propose to use the proximal gradient method ISTA for automated material model discovery if f is not quadratic, and fourth, we suggest a pathwise extension of ISTA for computing the regularization path. We demonstrate the applicability of all algorithms for the discovery of hyperelastic material models from uniaxial tension and simple shear data.

cs.CE↗

arXiv2026

Weighted Low-Rank Matrix Approximation: Acceleration and Applications

Weighted low-rank matrix approximation (WLRMA) generalizes classical low-rank approximation and matrix completion by allowing arbitrary elementwise weights. Such formulations arise naturally in a broad class of statistical models, including generalized linear low-rank models, where WLRMA serves as the computational primitive for parameter estimation. Despite its broad applicability, efficient optimization methods for general WLRMA remain relatively underdeveloped. In this paper, we formulate both the rank-constrained and nuclear-norm WLRMA problems within a unified first-order optimization framework by showing that the corresponding iterative algorithms are projected and proximal gradient methods. Building on this perspective, we develop accelerated algorithms based on Nesterov momentum and Anderson acceleration, together with a regularized Anderson scheme that improves numerical stability for non-convex problems. We further propose scalable implementations for large sparse data matrices and introduce a practical effective-rank criterion that provides a meaningful correspondence between rank-constrained and nuclear-norm solutions. We further show that fitting generalized linear low-rank models can be reduced to a sequence of weighted low-rank matrix approximation problems, allowing the proposed algorithms to be used as computational building blocks for their estimation. Simulation studies demonstrate substantial computational gains achieved by the proposed accelerated algorithms. Applications to the MovieLens dataset further illustrate the proposed framework for matrix completion, heteroscedastic Gaussian low-rank modeling, and logistic low-rank modeling.

stat.ML↗

arXiv2026

Enhancing a Risk Model by Adding Transient Statistical Factors

Estimating the covariance of asset returns, i.e., the risk model, is a key component of financial portfolio construction and evaluation. Most risk modeling approaches produce a factor model that decomposes the asset variability into two components: the first attributed to a small number of factors that are common among the assets and the second attributed to the idiosyncratic behavior of each asset. Third-party providers typically provide risk models to investors, and while these models are typically of high quality, they may fail to capture important information, e.g., changing market regimes and transient factors. To overcome these limitations, we propose a systematic method based on maximum likelihood estimation to enhance an existing factor model by both refining the given model and adding new statistical factors. Our approach relies only on the observed sequence of realized returns and on the choice of two hyperparameters: the number of additional factors and the half-life parameter that determines the weights assigned to returns in the log-likelihood objective. Importantly, our methodology applies to the situation where asset returns may be missing, making it suitable for typical equity datasets. We demonstrate our approach on the Barra short-term US risk model, a high-quality risk model used in practice, for a universe of US high-capitalization equities. We show that the proposed extension captures structure in the returns that is missed by the original model.

stat.AP↗

arXiv2026

balnet: Pathwise Estimation of Covariate Balancing Propensity Scores

We present balnet, an R package for scalable pathwise estimation of covariate balancing propensity scores via logistic covariate balancing loss functions. Regularization paths are computed with Yang and Hastie (2024)'s generic elastic net solver, supporting convex losses with non-smooth penalties, as well as group penalties and feature-specific penalty factors. For lasso penalization, balnet computes a regularization path of balancing weights from the largest observed covariate imbalance to a user-specified fraction of this maximum. We illustrate the method with an application to spatial pixel-level balancing for constructing synthetic control weights for the average treatment effect on the treated, using satellite data on wildfires.

stat.ME↗

arXiv2026

Single-Asset Adaptive Leveraged Volatility Control

This paper introduces a methodology for constructing a market index composed of a liquid risky asset and a liquid risk-free asset that achieves a fixed target volatility. Existing volatility-targeting strategies typically scale portfolio exposure inversely with a variance forecast, but such open-loop approaches suffer from high turnover, leverage spikes, and sensitivity to estimation error -- issues that limit practical adoption in index construction. We propose a proportional-control approach for setting the index weights that explicitly corrects tracking error through feedback. The method requires only a few interpretable parameters, making it transparent and practical for index construction. We demonstrate in simulation that this approach is more effective at consistently achieving the target volatility than the open-loop alternative.

q-fin.PM↗

arXiv2026

Adaptive Strategies for Pension Fund Management

This paper proposes a simulation-based framework for assessing and improving the performance of a pension fund management scheme. This framework is modular and allows the definition of customized performance metrics that are used to assess and iteratively improve asset and liability management policies. We illustrate our framework with a simple implementation that showcases the power of including adaptable features. We show that it is possible to dissipate longevity and volatility risks by permitting adaptability in asset allocation and payout levels. The numerical results show that by including a small amount of flexibility, there can be a substantial reduction in the cost to run the pension plan as well as a substantial decrease in the probability of defaulting.

math.OC↗

arXiv2026

Univariate-Guided Sparse Regression for Biobank-Scale High-Dimensional Omics Data

We present a scalable framework for computing polygenic risk scores (PRS) in high-dimensional genomic settings using the recently introduced Univariate-Guided Sparse Regression (uniLasso). UniLasso is a two-stage penalized regression procedure that leverages univariate coefficients and magnitudes to stabilize feature selection and enhance interpretability. Building on its theoretical and empirical advantages, we adapt uniLasso for application to the UK Biobank, a population-based repository comprising over one million genetic variants measured on hundreds of thousands of individuals from the United Kingdom. We further extend the framework to incorporate external summary statistics to increase predictive accuracy. Our results demonstrate that uniLasso attains predictive performance comparable to standard Lasso while selecting substantially fewer variants, yielding sparser and more interpretable models. Moreover, it exhibits superior performance in estimating PRS relative to its competitors, such as PRS-CS. Integrating external scores further improves prediction while maintaining sparsity.

stat.ME↗

arXiv2025

Factor Fitting, Rank Allocation, and Partitioning in Multilevel Low Rank Matrices

We consider multilevel low rank (MLR) matrices, defined as a row and column permutation of a sum of matrices, each one a block diagonal refinement of the previous one, with all blocks low rank given in factored form. MLR matrices extend low rank matrices but share many of their properties, such as the total storage required and complexity of matrix-vector multiplication. We address three problems that arise in fitting a given matrix by an MLR matrix in the Frobenius norm. The first problem is factor fitting, where we adjust the factors of the MLR matrix. The second is rank allocation, where we choose the ranks of the blocks in each level, subject to the total rank having a given value, which preserves the total storage needed for the MLR matrix. The final problem is to choose the hierarchical partition of rows and columns, along with the ranks and factors. This paper is accompanied by an open source package that implements the proposed methods.

stat.ML↗

arXiv2025

Scalable solution to crossed random effects model with random slopes

The crossed random effects model is widely used, finding applications in various fields such as longitudinal studies, e-commerce, and recommender systems, among others. However, these models encounter scalability challenges, as the computational time for standard algorithms grows superlinearly with the number N of observations in the data set, commonly $Ω(N^{3/2})$ or worse. Recent published works present scalable methods for crossed random effects in linear models and some generalized linear models, but those methods only allow for random intercepts. In this paper, we devise scalable algorithms for models that include random slopes. This addition brings substantial difficulty in estimating the random-effect covariance matrices in a scalable way. We address this issue by using a variational EM algorithm. Our proposed approach accommodates both diagonal covariance matrices and cases where no structure is assumed-a scenario common in fields such as psychology and neuroscience. In simulations, the proposed method is substantially faster than standard methods for large $N$. It is also more efficient than ordinary least squares which has a problem of greatly underestimating the sampling uncertainty in parameter estimates. We illustrate the new method on a MovieLens dataset, as well as a large data set (five million observations) from the online retailer Stitch Fix.

stat.ME↗

arXiv2025

Fitting Multilevel Factor Models

We examine a special case of the multilevel factor model, with covariance given by multilevel low rank (MLR) matrix~\cite{parshakova2023factor}. We develop a novel, fast implementation of the expectation-maximization algorithm, tailored for multilevel factor models, to maximize the likelihood of the observed data. This method accommodates any hierarchical structure and maintains linear time and storage complexities per iteration. This is achieved through a new efficient technique for computing the inverse of the positive definite MLR matrix. We show that the inverse of positive definite MLR matrix is also an MLR matrix with the same sparsity in factors, and we use the recursive Sherman-Morrison-Woodbury matrix identity to obtain the factors of the inverse. Additionally, we present an algorithm that computes the Cholesky factorization of an expanded matrix with linear time and space complexities, yielding the covariance matrix as its Schur complement. This paper is accompanied by an open-source package that implements the proposed methods.

stat.ML↗

arXiv2025

Univariate-Guided Sparse Regression

In this paper, we introduce ``UniLasso'' -- a novel statistical method for sparse regression. This two-stage approach preserves the signs of the univariate coefficients and leverages their magnitude. Both of these properties are attractive for stability and interpretation of the model. Through comprehensive simulations and applications to real-world datasets, we demonstrate that UniLasso outperforms Lasso in various settings, particularly in terms of sparsity and model interpretability. We prove asymptotic support recovery and mean-squared error consistency under a set of conditions different from the well-known irrepresentability conditions for the Lasso. Extensions to generalized linear models (GLMs) and Cox regression are also discussed.

stat.ME↗

arXiv2025

Pre-validation Revisited

Pre-validation is a way to build prediction model with two datasets of significantly different feature dimensions. Previous work showed that the asymptotic distribution of the resulting test statistic for the pre-validated predictor deviates from a standard Normal, hence leads to issues in hypothesis testing. In this paper, we revisit the pre-validation procedure and extend the problem formulation without any independence assumption on the two feature sets. We propose not only an analytical distribution of the test statistic for the pre-validated predictor under certain models, but also a generic bootstrap procedure to conduct inference. We show properties and benefits of pre-validation in prediction, inference and error estimation by simulations and applications, including analysis of a breast cancer study and a synthetic GWAS example.

stat.ME↗

arXiv2024

Ridge Regularization: an Essential Concept in Data Science

Ridge or more formally $\ell_2$ regularization shows up in many areas of statistics and machine learning. It is one of those essential devices that any good data scientist needs to master for their craft. In this brief ridge fest I have collected together some of the magic and beauty of ridge that my colleagues and I have encountered over the past 40 years in applied statistics.

stat.ME↗

arXiv2024

A Fast Coordinate Descent Method for High-Dimensional Non-Negative Least Squares using a Unified Sparse Regression Framework

We develop theoretical results that establish a connection across various regression methods such as the non-negative least squares, bounded variable least squares, simplex constrained least squares, and lasso. In particular, we show in general that a polyhedron constrained least squares problem admits a locally unique sparse solution in high dimensions. We demonstrate the power of our result by concretely quantifying the sparsity level for the aforementioned methods. Furthermore, we propose a novel coordinate descent based solver for NNLS in high dimensions using our theoretical result as motivation. We show through simulated data and a real data example that our solver achieves at least a 5x speed-up from the state-of-the-art solvers.

stat.CO↗

arXiv2024

A Statistical View of Column Subset Selection

We consider the problem of selecting a small subset of representative variables from a large dataset. In the computer science literature, this dimensionality reduction problem is typically formalized as Column Subset Selection (CSS). Meanwhile, the typical statistical formalization is to find an information-maximizing set of Principal Variables. This paper shows that these two approaches are equivalent, and moreover, both can be viewed as maximum likelihood estimation within a certain semi-parametric model. Within this model, we establish suitable conditions under which the CSS estimate is consistent in high dimensions, specifically in the proportional asymptotic regime where the number of variables over the sample size converges to a constant. Using these connections, we show how to efficiently (1) perform CSS using only summary statistics from the original dataset; (2) perform CSS in the presence of missing and/or censored data; and (3) select the subset size for CSS in a hypothesis testing framework.

stat.ME↗

arXiv2024

The mosaic permutation test: an exact and nonparametric goodness-of-fit test for factor models

Financial firms often rely on fundamental factor models to explain correlations among asset returns and manage risk. Yet after major events, e.g., COVID-19, analysts may reassess whether existing risk models continue to fit well: specifically, after accounting for a set of known factor exposures, are the residuals of the asset returns independent? With this motivation, we introduce the mosaic permutation test, a nonparametric goodness-of-fit test for preexisting factor models. Our method can leverage modern machine learning techniques to detect model violations while provably controlling the false positive rate, i.e., the probability of rejecting a well-fitting model, without making asymptotic approximations or parametric assumptions. This property helps prevent analysts from unnecessarily rebuilding accurate models, which can waste resources and increase risk. To illustrate our methodology, we apply the mosaic permutation test to the BlackRock Fundamental Equity Risk (BFRE) model. Although the BFRE model generally explains the most significant correlations among assets, we find evidence of unexplained correlations among certain real estate stocks, and we show that adding new factors improves model fit. We implement our methods in the python package mosaicperm.

stat.ME↗

arXiv2024

Scalable recommender system based on factor analysis

Recommender systems have become crucial in the modern digital landscape, where personalized content, products, and services are essential for enhancing user experience. This paper explores statistical models for recommender systems, focusing on crossed random effects models and factor analysis. We extend the crossed random effects model to include random slopes, enabling the capture of varying covariate effects among users and items. Additionally, we investigate the use of factor analysis in recommender systems, particularly for settings with incomplete data. The paper also discusses scalable solutions using the Expectation Maximization (EM) and variational EM algorithms for parameter estimation, highlighting the application of these models to predict user-item interactions effectively.

stat.ME↗

arXiv2024

MMIL: A novel algorithm for disease associated cell type discovery

Single-cell datasets often lack individual cell labels, making it challenging to identify cells associated with disease. To address this, we introduce Mixture Modeling for Multiple Instance Learning (MMIL), an expectation maximization method that enables the training and calibration of cell-level classifiers using patient-level labels. Our approach can be used to train e.g. lasso logistic regression models, gradient boosted trees, and neural networks. When applied to clinically-annotated, primary patient samples in Acute Myeloid Leukemia (AML) and Acute Lymphoblastic Leukemia (ALL), our method accurately identifies cancer cells, generalizes across tissues and treatment timepoints, and selects biologically relevant features. In addition, MMIL is capable of incorporating cell labels into model training when they are known, providing a powerful framework for leveraging both labeled and unlabeled data simultaneously. Mixture Modeling for MIL offers a novel approach for cell classification, with significant potential to advance disease understanding and management, especially in scenarios with unknown gold-standard labels and high dimensionality.

q-bio.QM↗