arXiv subjects

Natesh Pillai

Publications and source records attributed to Natesh Pillai.

At least 19 recordsLinked to original sources

arXiv2026

Mixing on $k$ Columns of the Transvection Walk

In Diaconis and Saloff-Coste (1996), the authors introduced the simple ``transvection" walk on $\mathrm{GL}_n(\mathbb F_2)$: at each step, choose two distinct rows and add one to the other. In Ben-Hamou (2025), the author recently proved that this walk has mixing time $O(n^2\log n)$. Inspired by applications in cryptography (see Sotiraki (2016)), Ben-Hamou and Peres (2018) conjectured that the first $k$ columns of this walk mixed in $O(nk \log(n))$ steps. Our main result is a proof of this conjecture uniformly in $n$ and $k.$ Our proof is based on a local-to-global entropy estimate, in the spirit of block factorization results such as Caputo et al (2015), Caputo et al (2021). In our setting, the kernels that correspond roughly to the block kernels of Caputo et al (2021) do not have uniformly large log-Sobolev constants, and so naively applying these techniques does not improve over Ben-Hamou (2025). We avoid these bad blocks by combining our entropy estimates with a burn-in argument similar to classical drift-and-minorization arguments of Rosenthal (1995). This method may be of broader interest, and so we illustrate it by proving an analogous result for a family of product-replacement algorithms on the Heisenberg group.

math.PR↗

arXiv2025

Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems

Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendation systems to generative tasks. Although scaling laws indicate that larger models generally yield better generalization and performance, their substantial computational requirements often render them impractical for many real-world scenarios at scale. In this paper, we present a comprehensive set of insights for training and deploying small language models (SLMs) that deliver high performance for a variety of industry use cases. We focus on two key techniques: (1) knowledge distillation and (2) model compression via structured pruning and quantization. These approaches enable SLMs to retain much of the quality of their larger counterparts while significantly reducing training/serving costs and latency. We detail the impact of these techniques on a variety of use cases in a large professional social network platform and share deployment lessons, including hardware optimization strategies that improve speed and throughput for both predictive and reasoning-based applications in Recommendation Systems.

cs.IR↗

arXiv2025

BP-Seg: A graphical model approach to unsupervised and non-contiguous text segmentation using belief propagation

Text segmentation based on the semantic meaning of sentences is a fundamental task with broad utility in many downstream applications. In this paper, we propose a graphical model-based unsupervised learning approach, named BP-Seg for efficient text segmentation. Our method not only considers local coherence, capturing the intuition that adjacent sentences are often more related, but also effectively groups sentences that are distant in the text yet semantically similar. This is achieved through belief propagation on the carefully constructed graphical models. Experimental results on both an illustrative example and a dataset with long-form documents demonstrate that our method performs favorably compared to competing approaches.

cs.CL↗

arXiv2025

Adaptive Divide and Conquer with Two Rounds of Communication

We introduce a two-round adaptive communication strategy that enables rate-optimal estimation in the white noise model without requiring prior knowledge of the underlying smoothness. In the first round, local machines send summary statistics using $(\log_2(n))^2$ bits to enable the central machine to select the tuning parameters of the procedure. In the second round, another set of statistics are transmitted using optimal number of bits, enabling the central machine to aggregate and produce a final estimator that adapts to the true smoothness level. This approach achieves optimal convergence rates across a wider range of regularities, offering a potential improvement in the adaptability and efficiency of distributed estimation compared to existing one-round methods.

math.ST↗

arXiv2025

A note on simulation methods for the Dirichlet-Laplace prior

Bhattacharya et al. (2015, Journal of the American Statistical Association 110(512): 1479-1490) introduce a novel prior, the Dirichlet-Laplace (DL) prior, and propose a Markov chain Monte Carlo (MCMC) method to simulate posterior draws under this prior in a conditionally Gaussian setting. The original algorithm samples from conditional distributions in the wrong order, i.e., it does not correctly sample from the joint posterior distribution of all latent variables. This note details the issue and provides two simple solutions: A correction to the original algorithm and a new algorithm based on an alternative, yet equivalent, formulation of the prior. This corrigendum does not affect the theoretical results in Bhattacharya et al. (2015).

stat.CO↗

arXiv2025

AlphaPO: Reward Shape Matters for LLM Alignment

Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular examples of DAAs include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce \textbf{AlphaPO}, a new DAA method that leverages an $α$-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7\% to 10\% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15\% to 50\% relative improvement over DPO on the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynamics, as well as improve alignment performance.

cs.CL↗

arXiv2023

A Bayesian Gaussian Process for Estimating a Causal Exposure Response Curve in Environmental Epidemiology

Motivated by environmental policy questions, we address the challenges of estimation, change point detection, and uncertainty quantification of a causal exposure-response function (CERF). Under a potential outcome framework, the CERF describes the relationship between a continuously varying exposure (or treatment) and its causal effect on an outcome. We propose a new Bayesian approach that relies on a Gaussian process (GP) model to estimate the CERF nonparametrically. To achieve the desired separation of design and analysis phases, we parametrize the covariance (kernel) function of the GP to mimic matching via a Generalized Propensity Score (GPS). The hyper-parameters as well as the form of the kernel function of the GP are chosen to optimize covariate balance. Our approach achieves automatic uncertainty evaluation of the CERF with high computational efficiency, and enables change point detection through inference on derivatives of the CERF. We provide theoretical results showing the correspondence between our Bayesian GP framework and traditional approaches in causal inference for estimating causal effects of a continuous exposure. We apply the methods to 520,711 ZIP-code-level observations to estimate the causal effect of long-term exposures to PM2.5, ozone, and NO2 on all-cause mortality among Medicare enrollees in the US. A computationally efficient implementation of the proposed GP models is provided in the GPCERF R package, which is available on CRAN.

stat.ME↗

arXiv2022

Rate-optimal refinement strategies for local approximation MCMC

Many Bayesian inference problems involve target distributions whose density functions are computationally expensive to evaluate. Replacing the target density with a local approximation based on a small number of carefully chosen density evaluations can significantly reduce the computational expense of Markov chain Monte Carlo (MCMC) sampling. Moreover, continual refinement of the local approximation can guarantee asymptotically exact sampling. We devise a new strategy for balancing the decay rate of the bias due to the approximation with that of the MCMC variance. We prove that the error of the resulting local approximation MCMC (LA-MCMC) algorithm decays at roughly the expected $1/\sqrt{T}$ rate, and we demonstrate this rate numerically. We also introduce an algorithmic parameter that guarantees convergence given very weak tail bounds, significantly strengthening previous convergence results. Finally, we apply LA-MCMC to a computationally intensive Bayesian inverse problem arising in groundwater hydrology.

stat.CO↗

arXiv2020

Late 19th-Century Navigational Uncertainties and Their Influence on Sea Surface Temperature Estimates

Accurate estimates of historical changes in sea surface temperatures (SSTs) and their uncertainties are important for documenting and understanding historical changes in climate. A source of uncertainty that has not previously been quantified in historical SST estimates stems from position errors. A Bayesian inference framework is proposed for quantifying errors in reported positions and their implications on SST estimates. The analysis framework is applied to data from the International Comprehensive Ocean-Atmosphere Data Set (ICOADS3.0) in 1885, a time when astronomical and chronometer estimation of position was common, but predating the use of radio signals. Focus is upon a subset of 943 ship tracks from ICOADS3.0 that report their position every two hours to a precision of 0.01° longitude and latitude. These data are interpreted as positions determined by dead reckoning that are periodically updated by celestial correction techniques. The posterior medians of uncertainties in celestial correction are 33.1 km (0.30° on the equator) in longitude and 24.4 km (0.22°) in latitude, respectively. The posterior medians of two-hourly dead reckoning uncertainties are 19.2% for ship speed and 13.2° for ship heading, leading to random position uncertainties with median 0.18° (20 km on the equator) in longitude and 0.15° (17 km) in latitude. Reported ship tracks also contain systematic position uncertainties relating to precursor dead-reckoning positions not being updated after obtaining celestial position estimates, indicating that more accurate positions can be provided for SST observations. Finally, we translate position errors into SST uncertainties by sampling an ensemble of SSTs from the Multi-scale Ultra-high resolution Sea Surface Temperature (MURSST) data set.

stat.AP↗

arXiv2019

Weak separation in mixture models and implications for principal stratification

Principal stratification is a widely used framework for addressing post-randomization complications. After using principal stratification to define causal effects of interest, researchers are increasingly turning to finite mixture models to estimate these quantities. Unfortunately, standard estimators of mixture parameters, like the MLE, are known to exhibit pathological behavior. We study this behavior in a simple but fundamental example, a two-component Gaussian mixture model in which only the component means and variances are unknown, and focus on the setting in which the components are weakly separated. In this case, we show that the asymptotic convergence rate of the MLE is quite poor, such as $O(n^{-1/6})$ or even $O(n^{-1/8})$. We then demonstrate via theoretical arguments as well as extensive simulations that, in finite samples, the MLE behaves like a threshold estimator, in the sense that the MLE can give strong evidence that the means are equal when the truth is otherwise. We also explore the behavior of the MLE when the MLE is non-zero, showing that it is difficult to estimate both the sign and magnitude of the means in this case. We provide diagnostics for all of these pathologies and apply these ideas to re-analyzing two randomized evaluations of job training programs, JOBS II and Job Corps. Our results suggest that the corresponding maximum likelihood estimates should be interpreted with caution in these cases.

stat.ME↗

arXiv2018

Unbiased estimation of log normalizing constants with applications to Bayesian cross-validation

Posterior distributions often feature intractable normalizing constants, called marginal likelihoods or evidence, that are useful for model comparison via Bayes factors. This has motivated a number of methods for estimating ratios of normalizing constants in statistics. In computational physics the logarithm of these ratios correspond to free energy differences. Combining unbiased Markov chain Monte Carlo estimators with path sampling, also called thermodynamic integration, we propose new unbiased estimators of the logarithm of ratios of normalizing constants. As a by-product, we propose unbiased estimators of the Bayesian cross-validation criterion. The proposed estimators are consistent, asymptotically Normal and can easily benefit from parallel processing devices. Various examples are considered for illustration.

stat.CO↗

arXiv2018

Bias correction in daily maximum and minimum temperature measurements through Gaussian process modeling

The Global Historical Climatology Network-Daily database contains, among other variables, daily maximum and minimum temperatures from weather stations around the globe. It is long known that climatological summary statistics based on daily temperature minima and maxima will not be accurate, if the bias due to the time at which the observations were collected is not accounted for. Despite some previous work, to our knowledge, there does not exist a satisfactory solution to this important problem. In this paper, we carefully detail the problem and develop a novel approach to address it. Our idea is to impute the hourly temperatures at the location of the measurements by borrowing information from the nearby stations that record hourly temperatures, which then can be used to create accurate summaries of temperature extremes. The key difficulty is that these imputations of the temperature curves must satisfy the constraint of falling between the observed daily minima and maxima, and attaining those values at least once in a twenty-four hour period. We develop a spatiotemporal Gaussian process model for imputing the hourly measurements from the nearby stations, and then develop a novel and easy to implement Markov Chain Monte Carlo technique to sample from the posterior distribution satisfying the above constraints. We validate our imputation model using hourly temperature data from four meteorological stations in Iowa, of which one is hidden and the data replaced with daily minima and maxima, and show that the imputed temperatures recover the hidden temperatures well. We also demonstrate that our model can exploit information contained in the data to infer the time of daily measurements.

stat.AP↗

arXiv2018

Diagnosing missing always at random in multivariate data

Models for analyzing multivariate data sets with missing values require strong, often unassessable, assumptions. The most common of these is that the mechanism that created the missing data is ignorable - a twofold assumption dependent on the mode of inference. The first part, which is the focus here, under the Bayesian and direct-likelihood paradigms, requires that the missing data are missing at random; in contrast, the frequentist-likelihood paradigm demands that the missing data mechanism always produces missing at random data, a condition known as missing always at random. Under certain regularity conditions, assuming missing always at random leads to an assumption that can be tested using the observed data alone namely, the missing data indicators only depend on fully observed variables. Here, we propose three different diagnostic tests that not only indicate when this assumption is incorrect but also suggest which variables are the most likely culprits. Although missing always at random is not a necessary condition to ensure validity under the Bayesian and direct-likelihood paradigms, it is sufficient, and evidence for its violation should encourage the careful statistician to conduct targeted sensitivity analyses.

stat.AP↗

arXiv2017

Parallel local approximation MCMC for expensive models

Performing Bayesian inference via Markov chain Monte Carlo (MCMC) can be exceedingly expensive when posterior evaluations invoke the evaluation of a computationally expensive model, such as a system of partial differential equations. In recent work [Conrad et al. JASA 2016, arXiv:1402.1694], we described a framework for constructing and refining local approximations of such models during an MCMC simulation. These posterior--adapted approximations harness regularity of the model to reduce the computational cost of inference while preserving asymptotic exactness of the Markov chain. Here we describe two extensions of that work. First, we prove that samplers running in parallel can collaboratively construct a shared posterior approximation while ensuring ergodicity of each associated chain, providing a novel opportunity for exploiting parallel computation in MCMC. Second, focusing on the Metropolis--adjusted Langevin algorithm, we describe how a proposal distribution can successfully employ gradients and other relevant information extracted from the approximation. We investigate the practical performance of our strategies using two challenging inference problems, the first in subsurface hydrology and the second in glaciology. Using local approximations constructed via parallel chains, we successfully reduce the run time needed to characterize the posterior distributions in these problems from days to hours and from months to days, respectively, dramatically improving the tractability of Bayesian inference.

stat.CO↗

arXiv2017

MCMC for Imbalanced Categorical Data

Many modern applications collect highly imbalanced categorical data, with some categories relatively rare. Bayesian hierarchical models combat data sparsity by borrowing information, while also quantifying uncertainty. However, posterior computation presents a fundamental barrier to routine use; a single class of algorithms does not work well in all settings and practitioners waste time trying different types of MCMC approaches. This article was motivated by an application to quantitative advertising in which we encountered extremely poor computational performance for common data augmentation MCMC algorithms but obtained excellent performance for adaptive Metropolis. To obtain a deeper understanding of this behavior, we give strong theory results on computational complexity in an infinitely imbalanced asymptotic regime. Our results show computational complexity of Metropolis is logarithmic in sample size, while data augmentation is polynomial in sample size. The root cause of poor performance of data augmentation is a discrepancy between the rates at which the target density and MCMC step sizes concentrate. In general, MCMC algorithms that have a similar discrepancy will fail in large samples - a result with substantial practical impact.

math.ST↗

arXiv2017

Designs for estimating the treatment effect in networks with interference

In this paper we introduce new, easily implementable designs for drawing causal inference from randomized experiments on networks with interference. Inspired by the idea of matching in observational studies, we introduce the notion of considering a treatment assignment as a quasi-coloring" on a graph. Our idea of a perfect quasi-coloring strives to match every treated unit on a given network with a distinct control unit that has identical number of treated and control neighbors. For a wide range of interference functions encountered in applications, we show both by theory and simulations that the classical Neymanian estimator for the direct effect has desirable properties for our designs. This further extends to settings where homophily is present in addition to interference.

math.ST↗

arXiv2016

A Divide and Conquer Strategy for High Dimensional Bayesian Factor Models

We propose a distributed computing framework, based on a divide and conquer strategy and hierarchical modeling, to accelerate posterior inference for high-dimensional Bayesian factor models. Our approach distributes the task of high-dimensional covariance matrix estimation to multiple cores, solves each subproblem separately via a latent factor model, and then combines these estimates to produce a global estimate of the covariance matrix. Existing divide and conquer methods focus exclusively on dividing the total number of observations $n$ into subsamples while keeping the dimension $p$ fixed. Our approach is novel in this regard: it includes all of the $n$ samples in each subproblem and, instead, splits the dimension $p$ into smaller subsets for each subproblem. The subproblems themselves can be challenging to solve when $p$ is large due to the dependencies across dimensions. To circumvent this issue, we specify a novel hierarchical structure on the latent factors that allows for flexible dependencies across dimensions, while still maintaining computational efficiency. Our approach is readily parallelizable and is shown to have computational efficiency of several orders of magnitude in comparison to fitting a full factor model. We report the performance of our method in synthetic examples and a genomics application.

stat.ME↗

arXiv2016

The Use of a Single Pseudo-Sample in Approximate Bayesian Computation

We analyze the computational efficiency of approximate Bayesian computation (ABC), which approximates a likelihood function by drawing pseudo-samples from the associated model. For the rejection sampling version of ABC, it is known that multiple pseudo-samples cannot substantially increase (and can substantially decrease) the efficiency of the algorithm as compared to employing a high-variance estimate based on a single pseudo-sample. We show that this conclusion also holds for a Markov chain Monte Carlo version of ABC, implying that it is unnecessary to tune the number of pseudo-samples used in ABC-MCMC. This conclusion is in contrast to particle MCMC methods, for which increasing the number of particles can provide large gains in computational efficiency.

stat.CO↗