arXiv subjects

Manit Paul

Publications and source records attributed to Manit Paul.

9 recordsLinked to original sources

arXiv2026

On a Probability Inequality for Order Statistics with Applications to Bootstrap, Conformal Prediction, and more

``Behind every limit theorem, there is an inequality'' said Kolmogorov. We say ``for every inequality, there is an approximate inequality under approximate regularity conditions.'' Suppose $X, X'$ are independent and identically distributed random variables. Then $X \le X'$ with a probability of at least $1/2$, irrespective of the underlying (common) distribution. One can ask what happens to the probability if $X, X'$ are independent but not identically distributed. It should be approximately $1/2$ if the distributions are approximately equal. Similarly, what if the random variables are dependent? It should, again, be approximately $1/2$ if the random variables are approximately independent. We explore an extension of this probability inequality involving order statistics and develop approximate versions of such an inequality under violations of independence and identical distribution assumptions. We further show that this inequality can be used as a basis to prove asymptotic validity of bootstrap/subsampling, finite-sample validity of conformal prediction, permutation tests, and asymptotic validity of rank tests without group invariance. Specifically, in the context of resampling inference, our results can be seen as a finite-sample instantiation of some results by Peter Hall and yield an alternative ``cheap bootstrap'' that applies to high-dimensional data.

math.ST↗

arXiv2025

Multiply Robust Conformal Risk Control with Coarsened Data

Conformal Prediction (CP) has recently received a tremendous amount of interest, leading to a wide range of new theoretical and methodological results for predictive inference with formal theoretical guarantees. However, the vast majority of CP methods assume that all units in the training data have fully observed data on both the outcome and covariates of primary interest, an assumption that rarely holds in practice. In reality, training data are often missing the outcome, a subset of covariates, or both on some units. In addition, time-to-event outcomes in the training set may be censored due to dropout or administrative end-of-follow-up. Accurately accounting for such coarsened data in the training sample while fulfilling the primary objective of well-calibrated conformal predictive inference, requires robustness and efficiency considerations. In this paper, we consider the general problem of obtaining distribution-free valid prediction regions for an outcome given coarsened training data. Leveraging modern semiparametric theory, we achieve our goal by deriving the efficient influence function of the quantile of the outcome we aim to predict, under a given semiparametric model for the coarsened data, carefully combined with a novel conformal risk control procedure. Our principled use of semiparametric theory has the key advantage of facilitating flexible machine learning methods such as random forests to learn the underlying nuisance functions of the semiparametric model. A straightforward application of the proposed general framework produces prediction intervals with stronger coverage properties under covariate shift, as well as the construction of multiply robust prediction sets in monotone missingness scenarios. We further illustrate the performance of our methods through various simulation studies.

math.ST↗

arXiv2025

N$^2$: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completion

Nearest neighbor (NN) methods have re-emerged as competitive tools for matrix completion, offering strong empirical performance and recent theoretical guarantees, including entry-wise error bounds, confidence intervals, and minimax optimality. Despite their simplicity, recent work has shown that NN approaches are robust to a range of missingness patterns and effective across diverse applications. This paper introduces N$^2$, a unified Python package and testbed that consolidates a broad class of NN-based methods through a modular, extensible interface. Built for both researchers and practitioners, N$^2$ supports rapid experimentation and benchmarking. Using this framework, we introduce a new NN variant that achieves state-of-the-art results in several settings. We also release a benchmark suite of real-world datasets, from healthcare and recommender systems to causal inference and LLM evaluation, designed to stress-test matrix completion methods beyond synthetic scenarios. Our experiments demonstrate that while classical methods excel on idealized data, NN-based techniques consistently outperform them in real-world settings.

cs.LG↗

arXiv2025

Adaptively-weighted Nearest Neighbors for Matrix Completion

In this technical note, we introduce and analyze AWNN: an adaptively weighted nearest neighbor method for performing matrix completion. Nearest neighbor (NN) methods are widely used in missing data problems across multiple disciplines such as in recommender systems and for performing counterfactual inference in panel data settings. Prior works have shown that in addition to being very intuitive and easy to implement, NN methods enjoy nice theoretical guarantees. However, the performance of majority of the NN methods rely on the appropriate choice of the radii and the weights assigned to each member in the nearest neighbor set and despite several works on nearest neighbor methods in the past two decades, there does not exist a systematic approach of choosing the radii and the weights without relying on methods like cross-validation. AWNN addresses this challenge by judiciously balancing the bias variance trade off inherent in weighted nearest-neighbor regression. We provide theoretical guarantees for the proposed method under minimal assumptions and support the theory via synthetic experiments.

stat.ML↗

arXiv2025

Finite sample valid confidence sets of mode

Estimating the mode of a unimodal distribution is a classical problem in statistics. Although there are several approaches for point-estimation of mode in the literature, very little has been explored about the interval-estimation of mode. Our work proposes a collection of novel methods of obtaining finite sample valid confidence set of the mode of a unimodal distribution. We analyze the behaviour of the width of the proposed confidence sets under some regularity assumptions of the density about the mode and show that the width of these confidence sets shrink to zero near optimally. Simply put, we show that it is possible to build finite sample valid confidence sets for the mode that shrink to a singleton as sample size increases. We support the theoretical results by showing the performance of the proposed methods on some synthetic data-sets. We believe that our confidence sets can be improved both in construction and in terms of rate.

math.ST↗

arXiv2024

Two-Sided Nearest Neighbors: An adaptive and minimax optimal procedure for matrix completion

Nearest neighbor (NN) algorithms have been extensively used for missing data problems in recommender systems and sequential decision-making systems. Prior theoretical analysis has established favorable guarantees for NN when the underlying data is sufficiently smooth and the missingness probabilities are lower bounded. Here we analyze NN with non-smooth non-linear functions with vast amounts of missingness. In particular, we consider matrix completion settings where the entries of the underlying matrix follow a latent non-linear factor model, with the non-linearity belonging to a \Holder function class that is less smooth than Lipschitz. Our results establish following favorable properties for a suitable two-sided NN: (1) The mean squared error (MSE) of NN adapts to the smoothness of the non-linearity, (2) under certain regularity conditions, the NN error rate matches the rate obtained by an oracle equipped with the knowledge of both the row and column latent factors, and finally (3) NN's MSE is non-trivial for a wide range of settings even when several matrix entries might be missing deterministically. We support our theoretical findings via extensive numerical simulations and a case study with data from a mobile health study, HeartSteps.

stat.ML↗

arXiv2024

Inference for Median and a Generalization of HulC

It is well-documented in the literature that sample splitting offers significant methodological and theoretical advantages in statistical inference. This, for example, includes cross-fitting in double machine learning, universal inference for parametric inference, and split conformal prediction. The recently proposed inference method, HulC, also falls into this category. HulC operates by viewing the target functional of interest as an approximate median of an estimator and applying the classical distribution-free confidence intervals for median with minimal sample size. When the estimators are asymptotically normal, HulC intervals are shown to be 50\% wider than the Wald intervals asymptotically, on average, for $95\%$ coverage. Interestingly, this ratio of widths converges to a non-degenerate distribution. In this paper, we propose a generalization of HulC that are only 25\% wider than the Wald intervals for asymptotically normal estimators, irrespective of the nominal coverage. Furthermore, similarly to HulC, these generalized intervals remain valid for a significantly wider class of problems with non-normal limiting distributions. To better understand width properties under non-normal limiting distributions, we analyze distribution-free confidence intervals for the median when the Lebesgue density at the median is either zero or infinite. Surprisingly, we find that properly scaled, the interval width converges to a non-degenerate random variable.

math.ST↗

arXiv2022

Feedback Interacting Urn Models

We introduce and discuss a special type of feedback interacting urn model with deterministic interaction. This is a generalisation of the very well known Eggenberger and Polya (1923) urn model. In our model, balls are added to a particular urn depending on the replacement matrix of that urn and the color of ball chosen from some other urn. This urn model can help in studying how various interacting models might behave in real life in the long run. We have also introduced a special type of interacting urn model with non-deterministic interaction and studied its behaviour. Furthermore, we have provided some nice examples to illustrate the various consequences of these interacting urn models.

math.PR↗

arXiv2022

Effect of influence in voter models and its application in detecting significant interference in political elections

In this article, we study the effect of vector-valued interventions in votes under a binary voter model, where each voter expresses their vote as a $0-1$ valued random variable to choose between two candidates. We assume that the outcome is determined by the majority function, which is true for a democratic system. The term intervention includes cases of counting errors, reporting irregularities, electoral malpractice etc. Our focus is to analyze the effect of the intervention on the final outcome. We construct statistical tests to detect significant irregularities in elections under two scenarios, one where exit poll data is available and more broadly under the assumption of a cost function associated with causing the interventions. Relevant theoretical results on the consistency of the test procedures are also derived. Through a detailed simulation study, we show that the test procedure has good power and is robust across various settings. We also implement our method on three real-life data sets. The applications provide results consistent with existing knowledge and establish that the method can be adopted for crucial problems related to political elections.

stat.AP↗