arXiv subjects

Sanvesh Srivastava

Publications and source records attributed to Sanvesh Srivastava.

At least 19 recordsLinked to original sources

arXiv2026

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

cs.LG↗

arXiv2026

Sparse Separable Factor Analysis in the Complex Domain with an Application to Local Field Potential Data

Complex-valued arrays arise in signal processing, where scientific interpretation depends on retaining amplitude and phase information. Existing covariance estimation methods either ignore the multiway organization of such data or rely on real-domain embeddings that do not directly exploit their complex structure. We develop sparse separable factor analysis (SSFA), a latent factor model for complex-valued arrays with a separable covariance structure across modes. Each mode-specific covariance matrix is modeled through a low-rank Hermitian factor structure and a diagonal residual covariance matrix. To obtain interpretable estimates, we impose elementwise lasso penalties on the complex loading matrices and estimate the SSFA parameters using a mode-wise parameter-expanded expectation-maximization procedure. The resulting loading updates admit closed-form complex soft-thresholding solutions, which shrink the modulus of each loading while preserving its phase. A separate balancing step resolves the scale nonidentifiability of the separable covariance structure. Simulation studies show that SSFA improves covariance estimation relative to vectorization-based methods, including complex principal component analysis. We apply SSFA to local field potential recordings from mice, where we compare separability structures induced by different groupings of brain region, frequency, and time and perform model-based imputation of recordings missing because of electrode misplacement.

stat.ML↗

arXiv2026

Moment bounds for condition numbers and singular values of high-dimensional Gaussian random matrices: Applications and limitations

Spectral properties of Gram matrices are central to high dimensional asymptotic analyses of statistical estimators in regression and covariance estimation. These properties, in turn, depend critically on the extreme singular values and condition numbers of Gaussian random matrices. For many applications, sharp positive and negative moment bounds for these quantities are required to control expected prediction risk and related performance metrics. Although extensive work provides concentration and tail bounds for extreme singular values of Gaussian random matrices, these results do not readily yield the moment bounds needed in such analyses. Motivated by this gap, we establish non asymptotic moment bounds for arbitrary positive moments of the largest singular value and arbitrary negative moments of the smallest singular value, and uniform bounds for arbitrary positive moments of the condition number of high dimensional Gaussian random matrices. We demonstrate the utility of these bounds by applying them to derive explicit risk guarantees in high dimensional regression and covariance estimation, as well as to obtain bounds on the mean iteration complexity of gradient descent for solving Gram linear systems. Finally, we present counterexamples demonstrating that the positive condition number moment bounds and negative smallest singular value moment bounds cannot, in general, be extended to the broader class of sub Gaussian random matrices.

math.ST↗

arXiv2026

CoMET: A Compressed Bayesian Mixed-Effects Model for High-Dimensional Tensors

Mixed-effects models are fundamental tools for analyzing clustered and repeated-measures data, but existing high-dimensional methods largely focus on penalized estimation with vector-valued covariates. Bayesian alternatives in this regime are limited, with no sampling-based mixed-effects framework that supports tensor-valued fixed- and random-effects covariates while remaining computationally tractable. We propose the Compressed Mixed-Effects Tensor (CoMET) model for high-dimensional repeated-measures data with scalar responses and tensor-valued covariates. CoMET performs structured, mode-wise random projection of the random-effects covariance, yielding a low-dimensional covariance parameter that admits simple Gaussian prior specification and enables efficient imputation of compressed random-effects. For the mean structure, CoMET leverages a low-rank tensor decomposition and margin-structured Horseshoe priors to enable fixed-effects selection. These design choices lead to an efficient collapsed Gibbs sampler whose computational complexity grows approximately linearly with the tensor covariate dimensions. We establish high-dimensional theoretical guarantees by identifying regularity conditions under which CoMET's posterior predictive risk decays to zero. Empirically, CoMET outperforms penalized competitors across a range of simulation studies and two benchmark applications involving facial-expression prediction and music emotion modeling.

stat.ME↗

arXiv2025

Asynchronous Distributed ECME Algorithm for Matrix Variate Non-Gaussian Responses

We propose a regression model with matrix-variate skew-t response (REGMVST) for analyzing irregular longitudinal data with skewness, symmetry, or heavy tails. REGMVST models matrix-variate responses and predictors, with rows indexing longitudinal measurements per subject. It uses the matrix-variate skew-t (MVST) distribution to handle skewness and heavy tails, a damped exponential correlation (DEC) structure for row-wise dependencies across irregular time profiles, and leaves the column covariance unstructured. For estimation, we initially develop an ECME algorithm for parameter estimation and further mitigate its computational bottleneck via an asynchronous and distributed ECME (ADECME) extension. ADECME accelerates the E-step through parallelization, and retains the simplicity of the conditional M-step, enabling scalable inference. Simulations using synthetic data and a case study exploring matrix-variate periodontal disease endpoints derived from electronic health records demonstrate ADECME's superiority in efficiency and convergence, over the alternatives. We also provide theoretical support for our empirical observations and identify regularity assumptions for ADECME's optimal performance. An accompanying R package is available at https://github.com/rh8liuqy/STMATREG.

stat.ME↗

arXiv2025

SLIM-LLMs: Modeling of Style-Sensory Language RelationshipsThrough Low-Dimensional Representations

Sensorial language -- the language connected to our senses including vision, sound, touch, taste, smell, and interoception, plays a fundamental role in how we communicate experiences and perceptions. We explore the relationship between sensorial language and traditional stylistic features, like those measured by LIWC, using a novel Reduced-Rank Ridge Regression (R4) approach. We demonstrate that low-dimensional latent representations of LIWC features r = 24 effectively capture stylistic information for sensorial language prediction compared to the full feature set (r = 74). We introduce Stylometrically Lean Interpretable Models (SLIM-LLMs), which model non-linear relationships between these style dimensions. Evaluated across five genres, SLIM-LLMs with low-rank LIWC features match the performance of full-scale language models while reducing parameters by up to 80%.

cs.CL↗

arXiv2025

Bayesian Compressed Mixed-Effects Models

Penalized likelihood and quasi-likelihood methods dominate inference in high-dimensional linear mixed-effects models. Sampling-based Bayesian inference is less explored due to the computational bottlenecks introduced by the random effects covariance matrix. To address this gap, we propose the compressed mixed-effects (CME) model, which defines a quasi-likelihood using low-dimensional covariance parameters obtained via random projections of the random effects covariance. This dimension reduction, combined with a global-local shrinkage prior on the fixed effects, yields an efficient collapsed Gibbs sampler for prediction and fixed effects selection. Theoretically, when the compression dimension grows slowly relative to the number of fixed effects and observations, the Bayes risk for prediction is asymptotically negligible, ensuring accurate prediction using the CME model. Empirically, the CME model outperforms existing approaches in terms of predictive accuracy, interval coverage, and fixed-effects selection across varied simulation settings and a real-world dataset.

stat.ME↗

arXiv2025

Regularized Parameter Estimation in Mixed Model Trace Regression

We introduce mixed model trace regression (MMTR), a mixed model linear regression extension for scalar responses and high-dimensional matrix-valued covariates. MMTR's fixed effects component is equivalent to trace regression, with an element-wise lasso penalty imposed on the regression coefficients matrix to facilitate the estimation of a sparse mean parameter. MMTR's key innovation lies in modeling the covariance structure of matrix-variate random effects as a Kronecker product of low-rank row and column covariance matrices, enabling sparse estimation of the covariance parameter through low-rank constraints. We establish identifiability conditions for the estimation of row and column covariance matrices and use them for rank selection by applying group lasso regularization on the columns of their respective Cholesky factors. We develop an Expectation-Maximization (EM) algorithm extension for numerically stable parameter estimation in high-dimensional applications. MMTR achieves estimation accuracy comparable to leading regularized quasi-likelihood competitors across diverse simulation studies and attains the lowest mean square prediction error compared to its competitors on a publicly available image dataset.

stat.ME↗

arXiv2023

Pseudo Bayesian Estimation of One-way ANOVA Model in Complex Surveys

We devise survey-weighted pseudo posterior distribution estimators under two-stage informative sampling of both primary clusters and secondary nested units for a one-way analysis of variance (ANOVA) population generating model as a simple canonical case where population model random effects are defined to be coincident with the primary clusters, for example student performance based on a survey of schools and students such as the 2000 OECD Programme for International Student Assessment (PISA). We consider estimation on an observed informative sample under both an augmented pseudo likelihood that co-samples the random effects, as well as an integrated likelihood that marginalizes out the random effects from the survey-weighted augmented pseudo likelihood. This paper includes a theoretical exposition that enumerates easily verified conditions for which estimation under the augmented pseudo posterior is guaranteed to be consistent at the true generating parameters. We reveal in simulation that both approaches produce asymptotically unbiased estimation of the generating hyperparameters for the random effects when a key condition on the sum of within cluster weighted residuals is met. We present a comparison with two frequentist alternatives, an expectation-maximization approach and a composite likelihood method that requires pairwise sampling weights.

stat.ME↗

arXiv2023

Machine Learning and the Future of Bayesian Computation

Bayesian models are a powerful tool for studying complex data, allowing the analyst to encode rich hierarchical dependencies and leverage prior information. Most importantly, they facilitate a complete characterization of uncertainty through the posterior distribution. Practical posterior computation is commonly performed via MCMC, which can be computationally infeasible for high dimensional models with many observations. In this article we discuss the potential to improve posterior computation using ideas from machine learning. Concrete future directions are explored in vignettes on normalizing flows, Bayesian coresets, distributed Bayesian inference, and variational inference.

stat.ML↗

arXiv2022

Distributed Bayesian Varying Coefficient Modeling Using a Gaussian Process Prior

Varying coefficient models (VCMs) are widely used for estimating nonlinear regression functions for functional data. Their Bayesian variants using Gaussian process priors on the functional coefficients, however, have received limited attention in massive data applications, mainly due to the prohibitively slow posterior computations using Markov chain Monte Carlo (MCMC) algorithms. We address this problem using a divide-and-conquer Bayesian approach. We first create a large number of data subsamples with much smaller sizes. Then, we formulate the VCM as a linear mixed-effects model and develop a data augmentation algorithm for obtaining MCMC draws on all the subsets in parallel. Finally, we aggregate the MCMC-based estimates of subset posteriors into a single Aggregated Monte Carlo (AMC) posterior, which is used as a computationally efficient alternative to the true posterior distribution. Theoretically, we derive minimax optimal posterior convergence rates for the AMC posteriors of both the varying coefficients and the mean regression function. We provide quantification on the orders of subset sample sizes and the number of subsets. The empirical results show that the combination schemes that satisfy our theoretical assumptions, including the AMC posterior, have better estimation performance than their main competitors across diverse simulations and in a real data analysis.

stat.ME↗

arXiv2021

Asynchronous and Distributed Data Augmentation for Massive Data Settings

Data augmentation (DA) algorithms are widely used for Bayesian inference due to their simplicity. In massive data settings, however, DA algorithms are prohibitively slow because they pass through the full data in any iteration, imposing serious restrictions on their usage despite the advantages. Addressing this problem, we develop a framework for extending any DA that exploits asynchronous and distributed computing. The extended DA algorithm is indexed by a parameter $r \in (0, 1)$ and is called Asynchronous and Distributed (AD) DA with the original DA as its parent. Any ADDA starts by dividing the full data into $k$ smaller disjoint subsets and storing them on $k$ processes, which could be machines or processors. Every iteration of ADDA augments only an $r$-fraction of the $k$ data subsets with some positive probability and leaves the remaining $(1-r)$-fraction of the augmented data unchanged. The parameter draws are obtained using the $r$-fraction of new and $(1-r)$-fraction of old augmented data. For many choices of $k$ and $r$, the fractional updates of ADDA lead to a significant speed-up over the parent DA in massive data settings, and it reduces to the distributed version of its parent DA when $r=1$. We show that the ADDA Markov chain is Harris ergodic with the desired stationary distribution under mild conditions on the parent DA algorithm. We demonstrate the numerical advantages of the ADDA in three representative examples corresponding to different kinds of massive data settings encountered in applications. In all these examples, our DA generalization is significantly faster than its parent DA algorithm for all the choices of $k$ and $r$. We also establish geometric ergodicity of the ADDA Markov chain for all three examples, which in turn yields asymptotically valid standard errors for estimates of desired posterior quantities.

stat.CO↗

arXiv2021

Gaussian Process Regression and Classification using International Classification of Disease Codes as Covariates

International Classification of Disease (ICD) codes are widely used for encoding diagnoses in electronic health records (EHR). Automated methods have been developed over the years for predicting biomedical responses using EHR that borrow information among diagnostically similar patients. Relatively less attention has been paid to developing patient similarity measures that model the structure of ICD codes and the presence of multiple chronic conditions, where a chronic condition is defined as a set of ICD codes. Motivated by this problem, we first develop a type of string kernel function for measuring similarity between a pair of subsets of ICD codes, which uses the definition of chronic conditions. Second, we extend this similarity measure to define a family of covariance functions on subsets of ICD codes. Using this family, we develop Gaussian process (GP) priors for Bayesian nonparametric regression and classification using diagnoses and other demographic information as covariates. Markov chain Monte Carlo (MCMC) algorithms are used for posterior inference and predictions. The proposed methods are free of any tuning parameters and are well-suited for automated prediction of continuous and categorical biomedical responses that depend on chronic conditions. We evaluate the practical performance of our method on EHR data collected from 1660 patients at the University of Iowa Hospitals and Clinics (UIHC) with six different primary cancer sites. Our method has better sensitivity and specificity than its competitors in classifying different primary cancer sites and estimates the marginal associations between chronic conditions and primary cancer sites.

stat.ME↗

arXiv2021

Asymptotic Normality of the Posterior Distributions in a Class of Hidden Markov Models

We show that the posterior distribution of parameters in a hidden Markov model with parametric emission distributions and discrete and known state space is asymptotically normal. The main novelty of our proof is that it is based on a testing condition and the sequence of test functions is obtained using an optimal transportation inequality.

math.ST↗

arXiv2021

Divide-and-Conquer Bayesian Inference in Hidden Markov Models

Divide-and-conquer Bayesian methods consist of three steps: dividing the data into smaller computationally manageable subsets, running a sampling algorithm in parallel on all the subsets, and combining parameter draws from all the subsets. The combined parameter draws are used for efficient posterior inference in massive data settings. A major restriction of existing divide-and-conquer methods is that their first two steps assume that the observations are independent. We address this problem by developing a divide-and-conquer method for Bayesian inference in parametric hidden Markov models, where the state space is known and finite. Our main contributions are two-fold. First, after partitioning the data into smaller blocks of consecutive observations, we modify the likelihood for performing posterior computations on the subsets such that the posterior variances of the subset and true posterior distributions have the same asymptotic order. Second, if the number of subsets is chosen appropriately depending on the mixing properties of the hidden Markov chain, then we show that the subset posterior distributions defined using the modified likelihood are asymptotically normal as the subset sample size tends to infinity. The latter result also implies that we can use any existing combination algorithm in the third step. We show that the combined posterior distribution obtained using one such algorithm is close to the true posterior distribution in 1-Wasserstein distance under widely used regularity assumptions. Our numerical results show that the proposed method provides an accurate approximation of the true posterior distribution than its competitors in diverse simulation studies and a real data analysis.

stat.ME↗

arXiv2020

An Algorithm for Distributed Bayesian Inference in Generalized Linear Models

Monte Carlo algorithms, such as Markov chain Monte Carlo (MCMC) and Hamiltonian Monte Carlo (HMC), are routinely used for Bayesian inference in generalized linear models; however, these algorithms are prohibitively slow in massive data settings because they require multiple passes through the full data in every iteration. Addressing this problem, we develop a scalable extension of these algorithms using the divide-and-conquer (D&C) technique that divides the data into a sufficiently large number of subsets, draws parameters in parallel on the subsets using a \textit{powered} likelihood, and produces Monte Carlo draws of the parameter by combining parameter draws obtained from each subset. These combined parameter draws play the role of draws from the original sampling algorithm. Our main contributions are two-fold. First, we demonstrate through diverse simulated and real data analyses that our distributed algorithm is comparable to the current state-of-the-art D&C algorithm in terms of statistical accuracy and computational efficiency. Second, providing theoretical support for our empirical observations, we identify regularity assumptions under which the proposed algorithm leads to asymptotically optimal inference. We illustrate our methodology through normal linear and logistic regressions, where parts of our D&C algorithm are analytically tractable.

stat.CO↗

arXiv2019

A Divide-and-Conquer Bayesian Approach to Large-Scale Kriging

We propose a three-step divide-and-conquer strategy within the Bayesian paradigm that delivers massive scalability for any spatial process model. We partition the data into a large number of subsets, apply a readily available Bayesian spatial process model on every subset, in parallel, and optimally combine the posterior distributions estimated across all the subsets into a pseudo-posterior distribution that conditions on the entire data. The combined pseudo posterior distribution replaces the full data posterior distribution for predicting the responses at arbitrary locations and for inference on the model parameters and spatial surface. Based on distributed Bayesian inference, our approach is called "Distributed Kriging" (DISK) and offers significant advantages in massive data applications where the full data are stored across multiple machines. We show theoretically that the Bayes $L_2$-risk of the DISK posterior distribution achieves the near optimal convergence rate in estimating the true spatial surface with various types of covariance functions, and provide upper bounds for the number of subsets as a function of the full sample size. The model-free feature of DISK is demonstrated by scaling posterior computations in spatial process models with a stationary full-rank and a nonstationary low-rank Gaussian process (GP) prior. A variety of simulations and a geostatistical analysis of the Pacific Ocean sea surface temperature data validate our theoretical results.

stat.ME↗

arXiv2018

Scalable Bayes via Barycenter in Wasserstein Space

Divide-and-conquer based methods for Bayesian inference provide a general approach for tractable posterior inference when the sample size is large. These methods divide the data into smaller subsets, sample from the posterior distribution of parameters in parallel on all the subsets, and combine posterior samples from all the subsets to approximate the full data posterior distribution. The smaller size of any subset compared to the full data implies that posterior sampling on any subset is computationally more efficient than sampling from the true posterior distribution. Since the combination step takes negligible time relative to sampling, posterior computations can be scaled to massive data by dividing the full data into a sufficiently large number of data subsets. One such approach relies on the geometry of posterior distributions estimated across different subsets and combines them through their barycenter in a Wasserstein space of probability measures. We provide theoretical guarantees on the accuracy of approximation that are valid in many applications. We show that the geometric method approximates the full data posterior distribution better than its competitors across diverse simulations and reproduces known results when applied to a movie ratings database.

stat.ME↗