Nonparametric shrinkage estimation of high-dimensional variance component matrices

Ran Xie; Iain M. Johnstone

Home/Archive/arXiv

arXiv · 2610.33431

Nonparametric shrinkage estimation of high-dimensional variance component matrices

Ran Xie·Iain M. Johnstone

View original source ↗Read source document ↗Export metadata ↓

Abstract

We develop a nonparametric spectral estimation method for variance component covariance matrices in high-dimensional multivariate mixed effects models. Although classical MANOVA estimators are unbiased entrywise, their eigenvalues can be severely dispersed and biased when the number of traits is comparable to the number of realizations of the random effects. Building on a system of self-consistent equations that characterizes the limiting spectral distribution of a MANOVA estimator, we construct a kernel-based nonlinear transformation of the sample eigenvalues that targets the optimal rotation-invariant estimator under Frobenius loss. The resulting procedure corrects for high-dimensional eigenvalue spreading without imposing a parametric model on the unknown covariance spectrum. Our work is motivated by applications in quantitative genetics, where estimating genetic covariance components across many phenotypic traits is of central interest. We implement the method in a new Python package mlmm-shrinkage and compare the proposed estimator with classical estimators across several common experimental designs in this domain through simulations and a real-data application.

Explore related subjects

stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Ran Xie, Iain M. Johnstone. 2026-10-24. Nonparametric shrinkage estimation of high-dimensional variance component matrices. https://arxiv.org/abs/2610.33431

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Principal Decomposition with Nested Submanifolds

Over the past decades, the increasing dimensionality of data has increased the need for effective data decomposition methods. Existing approaches, however, often rely on linear models or lack sufficient interpretability or flexibility. To address this issue, we introduce a novel nonlinear decomposition technique called the principal nested submanifolds, which builds on the foundational concepts of principal component analysis. This method exploits the local geometric information of data sets by projecting samples onto a series of nested principal submanifolds with progressively decreasing dimensions. It effectively isolates complex information within the data in a backward stepwise manner by targeting variations associated with smaller eigenvalues in local covariance matrices. The fitted structures are smooth nested manifolds rather than linear or special-shape subspaces. Simulation studies and an analysis of single-cell RNA-sequencing data illustrate how the method summarizes locally varying nonlinear structure while retaining a compatible hierarchy across dimensions. It provides flexible subspace constraints that facilitate the extraction of data components and noise reduction. This approach advances the non-Euclidean statistical analysis of data with low-dimensional intrinsic structure within Euclidean spaces and offers a complementary perspective for high-dimensional noisy data sets in fields such as bioinformatics and machine learning.

stat.ME↗

arXiv2026

Two-sample comparison through additive tree models for density ratios

The ratio of two densities provides a direct characterization of their differences. We consider the two-sample comparison problem by estimating this ratio given i.i.d. observations from two distributions. To this end, we propose additive tree models for density ratio estimation along with efficient algorithms using a new loss function, the balancing loss. The loss allows tree-based models to be trained using several algorithms originally designed for supervised learning, such as forward-stagewise optimization and gradient boosting. Moreover, the balancing loss resembles an exponential family kernel, and it can serve as a pseudo-likelihood with conjugate priors. This property enables generalized Bayesian inference on the density ratio using backfitting samplers designed for Bayesian additive regression trees (BART). Our Bayesian strategy provides uncertainty quantification for the inferred density ratio, which is critical for applications involving high-dimensional and data-limited distributions with potentially substantial uncertainty. We further show connections of the balancing loss to the exponential loss in binary classification and to the variational form of f-divergence, particularly the squared Hellinger distance. Numerical experiments demonstrate that the proposed boosting algorithms achieve high accuracy at low computational cost, and that the Bayesian additive tree is competitive in accuracy while additionally providing uncertainty quantification, which the compared estimators do not offer. Finally, we demonstrate its application to assessing the quality of generative models for microbiome compositional data.

stat.ME↗

arXiv2026

Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes

Learning a common policy from multiple source environments requires accounting for cross-source heterogeneity, and the offline setting introduces an additional challenge of limited data coverage. We formulate this problem as a distributionally robust Markov decision process under a shared linear MDP representation with source-specific transition and reward components. The uncertainty set is constructed through feature-wise mixtures of these components, yielding a $d$-rectangular structure that captures cross-source heterogeneity and admits a robust Bellman recursion. Building on this formulation, we develop Group-Robust Pessimistic Value Iteration (GR-PEVI), an offline policy-learning algorithm that combines feature-wise worst-case aggregation with a data-dependent pessimism penalty. Under a robust partial coverage condition, we establish finite-sample suboptimality guarantees for GR-PEVI and extend the analysis to clustered sources, characterizing the trade-off between the statistical gain from pooling similar sources and the approximation error induced by within-cluster heterogeneity. Simulation studies demonstrate the effectiveness of GR-PEVI, with empirical behavior consistent with the theoretical analysis. We apply GR-PEVI to a multi-source electronic health record study of multiple sclerosis treatment decisions to illustrate its use in a clinical setting.

stat.ME↗