arXiv subjects

Pratik Patil

Publications and source records attributed to Pratik Patil.

34 records · Page 2Linked to original sources

arXiv2024

Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning

We employ random matrix theory to establish consistency of generalized cross validation (GCV) for estimating prediction risks of sketched ridge regression ensembles, enabling efficient and consistent tuning of regularization and sketching parameters. Our results hold for a broad class of asymptotically free sketches under very mild data assumptions. For squared prediction risk, we provide a decomposition into an unsketched equivalent implicit ridge bias and a sketching-based variance, and prove that the risk can be globally optimized by only tuning sketch size in infinite ensembles. For general subquadratic prediction risk functionals, we extend GCV to construct consistent risk estimators, and thereby obtain distributional convergence of the GCV-corrected predictions in Wasserstein-2 metric. This in particular allows construction of prediction intervals with asymptotically correct coverage conditional on the training data. We also propose an "ensemble trick" whereby the risk for unsketched ridge regression can be efficiently estimated via GCV using small sketched ridge ensembles. We empirically validate our theoretical results using both synthetic and real large-scale datasets with practical sketches including CountSketch and subsampled randomized discrete cosine transforms.

math.ST↗

arXiv2024

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent

We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped gradient descent (GD) in high-dimensional least squares regression. We prove that GCV is generically inconsistent as an estimator of the prediction risk of early-stopped GD, even for a well-specified linear model with isotropic features. In contrast, we show that LOOCV converges uniformly along the GD trajectory to the prediction risk. Our theory requires only mild assumptions on the data distribution and does not require the underlying regression function to be linear. Furthermore, by leveraging the individual LOOCV errors, we construct consistent estimators for the entire prediction error distribution along the GD trajectory and consistent estimators for a wide class of error functionals. This in particular enables the construction of pathwise prediction intervals based on GD iterates that have asymptotically correct nominal coverage conditional on the training data.

math.ST↗

arXiv2023

Extrapolated cross-validation for randomized ensembles

Ensemble methods such as bagging and random forests are ubiquitous in various fields, from finance to genomics. Despite their prevalence, the question of the efficient tuning of ensemble parameters has received relatively little attention. This paper introduces a cross-validation method, ECV (Extrapolated Cross-Validation), for tuning the ensemble and subsample sizes in randomized ensembles. Our method builds on two primary ingredients: initial estimators for small ensemble sizes using out-of-bag errors and a novel risk extrapolation technique that leverages the structure of prediction risk decomposition. By establishing uniform consistency of our risk extrapolation technique over ensemble and subsample sizes, we show that ECV yields $δ$-optimal (with respect to the oracle-tuned risk) ensembles for squared prediction risk. Our theory accommodates general ensemble predictors, only requires mild moment assumptions, and allows for high-dimensional regimes where the feature dimension grows with the sample size. As a practical case study, we employ ECV to predict surface protein abundances from gene expressions in single-cell multiomics using random forests. In comparison to sample-split cross-validation and $K$-fold cross-validation, ECV achieves higher accuracy avoiding sample splitting. At the same time, its computational cost is considerably lower owing to the use of the risk extrapolation technique. Additional numerical results validate the finite-sample accuracy of ECV for several common ensemble predictors under a computational constraint on the maximum ensemble size.

stat.ME↗

arXiv2023

Bagging in overparameterized learning: Risk characterization and risk monotonization

Bagging is a commonly used ensemble technique in statistics and machine learning to improve the performance of prediction procedures. In this paper, we study the prediction risk of variants of bagged predictors under the proportional asymptotics regime, in which the ratio of the number of features to the number of observations converges to a constant. Specifically, we propose a general strategy to analyze the prediction risk under squared error loss of bagged predictors using classical results on simple random sampling. Specializing the strategy, we derive the exact asymptotic risk of the bagged ridge and ridgeless predictors with an arbitrary number of bags under a well-specified linear model with arbitrary feature covariance matrices and signal vectors. Furthermore, we prescribe a generic cross-validation procedure to select the optimal subsample size for bagging and discuss its utility to eliminate the non-monotonic behavior of the limiting risk in the sample size (i.e., double or multiple descents). In demonstrating the proposed procedure for bagged ridge and ridgeless predictors, we thoroughly investigate the oracle properties of the optimal subsample size and provide an in-depth comparison between different bagging variants.

math.ST↗

arXiv2023

Generalized equivalences between subsampling and ridge regularization

We establish precise structural and risk equivalences between subsampling and ridge regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic functionals of subsample ridge estimators, when fitted with different ridge regularization levels $λ$ and subsample aspect ratios $ψ$, are asymptotically equivalent along specific paths in the $(λ,ψ)$-plane (where $ψ$ is the ratio of the feature dimension to the subsample size). Our results only require bounded moment assumptions on feature and response distributions and allow for arbitrary joint distributions. Furthermore, we provide a data-dependent method to determine the equivalent paths of $(λ,ψ)$. An indirect implication of our equivalences is that optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio. This resolves a recent open problem raised by Nakkiran et al. for general data distributions under proportional asymptotics, assuming a mild regularity condition that maintains regression hardness through linearized signal-to-noise ratios.

math.ST↗

arXiv2023

Asymptotics of the Sketched Pseudoinverse

We take a random matrix theory approach to random sketching and show an asymptotic first-order equivalence of the regularized sketched pseudoinverse of a positive semidefinite matrix to a certain evaluation of the resolvent of the same matrix. We focus on real-valued regularization and extend previous results on an asymptotic equivalence of random matrices to the real setting, providing a precise characterization of the equivalence even under negative regularization, including a precise characterization of the smallest nonzero eigenvalue of the sketched matrix, which may be of independent interest. We then further characterize the second-order equivalence of the sketched pseudoinverse. We also apply our results to the analysis of the sketch-and-project method and to sketched ridge regression. Lastly, we prove that these results generalize to asymptotically free sketching matrices, obtaining the resulting equivalence for orthogonal sketching matrices and comparing our results to several common sketches used in practice.

math.NA↗

arXiv2023

Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation

We study subsampling-based ridge ensembles in the proportional asymptotics regime, where the feature size grows proportionally with the sample size such that their ratio converges to a constant. By analyzing the squared prediction risk of ridge ensembles as a function of the explicit penalty $λ$ and the limiting subsample aspect ratio $ϕ_s$ (the ratio of the feature size to the subsample size), we characterize contours in the $(λ, ϕ_s)$-plane at any achievable risk. As a consequence, we prove that the risk of the optimal full ridgeless ensemble (fitted on all possible subsamples) matches that of the optimal ridge predictor. In addition, we prove strong uniform consistency of generalized cross-validation (GCV) over the subsample sizes for estimating the prediction risk of ridge ensembles. This allows for GCV-based tuning of full ridgeless ensembles without sample splitting and yields a predictor whose risk matches optimal ridge risk.

math.ST↗

arXiv2022

Mitigating multiple descents: A model-agnostic framework for risk monotonization

Recent empirical and theoretical analyses of several commonly used prediction procedures reveal a peculiar risk behavior in high dimensions, referred to as double/multiple descent, in which the asymptotic risk is a non-monotonic function of the limiting aspect ratio of the number of features or parameters to the sample size. To mitigate this undesirable behavior, we develop a general framework for risk monotonization based on cross-validation that takes as input a generic prediction procedure and returns a modified procedure whose out-of-sample prediction risk is, asymptotically, monotonic in the limiting aspect ratio. As part of our framework, we propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting, respectively, and show that, under very mild assumptions, they provably achieve monotonic asymptotic risk behavior. Our results are applicable to a broad variety of prediction procedures and loss functions, and do not require a well-specified (parametric) model. We exemplify our framework with concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares prediction procedures. As one of the ingredients in our analysis, we also derive novel additive and multiplicative forms of oracle risk inequalities for split cross-validation that are of independent interest.

math.ST↗

arXiv2022

Objective frequentist uncertainty quantification for atmospheric CO$_2$ retrievals

The steadily increasing amount of atmospheric carbon dioxide (CO$_2$) is affecting the global climate system and threatening the long-term sustainability of Earth's ecosystem. In order to better understand the sources and sinks of CO$_2$, NASA operates the Orbiting Carbon Observatory-2 & 3 satellites to monitor CO$_2$ from space. These satellites make passive radiance measurements of the sunlight reflected off the Earth's surface in different spectral bands, which are then inverted in an ill-posed inverse problem to obtain estimates of the atmospheric CO$_2$ concentration. In this work, we propose a new CO$_2$ retrieval method that uses known physical constraints on the state variables and direct inversion of the target functional of interest to construct well-calibrated frequentist confidence intervals based on convex programming. We compare the method with the current operational retrieval procedure, which uses prior knowledge in the form of probability distributions to regularize the problem. We demonstrate that the proposed intervals consistently achieve the desired frequentist coverage, while the operational uncertainties are poorly calibrated in a frequentist sense both at individual locations and over a spatial region in a realistic simulation experiment. We also study the influence of specific nuisance state variables on the length of the proposed intervals and identify certain key variables that can greatly reduce the final uncertainty given additional deterministic or probabilistic constraints, and develop a principled framework to incorporate such information into our method.

stat.AP↗

arXiv2022

Uncertainty quantification for wide-bin unfolding: one-at-a-time strict bounds and prior-optimized confidence intervals

Unfolding is an ill-posed inverse problem in particle physics aiming to infer a true particle-level spectrum from smeared detector-level data. For computational and practical reasons, these spaces are typically discretized using histograms, and the smearing is modeled through a response matrix corresponding to a discretized smearing kernel of the particle detector. This response matrix depends on the unknown shape of the true spectrum, leading to a fundamental systematic uncertainty in the unfolding problem. To handle the ill-posed nature of the problem, common approaches regularize the problem either directly via methods such as Tikhonov regularization, or implicitly by using wide-bins in the true space that match the resolution of the detector. Unfortunately, both of these methods lead to a non-trivial bias in the unfolded estimator, thereby hampering frequentist coverage guarantees for confidence intervals constructed from these methods. We propose two new approaches to addressing the bias in the wide-bin setting through methods called One-at-a-time Strict Bounds (OSB) and Prior-Optimized (PO) intervals. The OSB intervals are a bin-wise modification of an existing guaranteed-coverage procedure, while the PO intervals are based on a decision-theoretic view of the problem. Importantly, both approaches provide well-calibrated frequentist confidence intervals even in constrained and rank-deficient settings. These methods are built upon a more general answer to the wide-bin bias problem, involving unfolding with fine bins first, followed by constructing confidence intervals for linear functionals of the fine-bin counts. We test and compare these methods to other available methodologies in a wide-bin deconvolution example and a realistic particle physics simulation of unfolding a steeply falling particle spectrum.

stat.AP↗

arXiv2021

Uplink-Downlink Duality Between Multiple-Access and Broadcast Channels with Compressing Relays

Uplink-downlink duality refers to the fact that under a sum-power constraint, the capacity regions of a Gaussian multiple-access channel and a Gaussian broadcast channel with Hermitian transposed channel matrices are identical. This paper generalizes this result to a cooperative cellular network, in which remote access-points are deployed as relays in serving the users under the coordination of a central processor (CP). In this model, the users and the relays are connected over noisy wireless links, while the relays and the CP are connected over noiseless but rate-limited fronthaul links. Based on a Lagrangian technique, this paper establishes a duality relationship between such a multiple-access relay channel and broadcast relay channel, under the assumption that the relays use compression-based strategies. Specifically, we show that under the same total transmit power constraint and individual fronthaul rate constraints, the achievable rate regions of the Gaussian multiple-access and broadcast relay channels are identical, when either independent compression or Wyner-Ziv and multivariate compression strategies are used. The key observations are that if the beamforming vectors at the relays are fixed, the sum-power minimization problems under the achievable rate and fronthaul constraints in both the uplink and the downlink can be transformed into either a linear programming or a semidefinite programming problem depending on the compression technique, and that the uplink and downlink problems are Lagrangian duals of each other. Moreover, the dual variables corresponding to the downlink rate constraints become the uplink powers; the dual variables corresponding to the downlink fronthaul constraints become the uplink quantization noises. This duality relationship enables an efficient algorithm for optimizing the downlink transmission and relaying strategies based on the uplink.

cs.IT↗

arXiv2019

Generalized Compression Strategy for the Downlink Cloud Radio Access Network

This paper studies the downlink of a cloud radio access network (C-RAN) in which a centralized processor (CP) communicates with mobile users through base stations (BSs) that are connected to the CP via finite-capacity fronthaul links. Information theoretically, the downlink of a C-RAN is modeled as a two-hop broadcast-relay network. Among the various transmission and relaying strategies for such model, this paper focuses on the compression strategy, in which the CP centrally encodes the signals to be broadcasted jointly by the BSs, then compresses and sends these signals to the BSs through the fronthaul links. This paper characterizes an achievable rate region for a generalized compression strategy with Marton's multicoding for broadcasting and multivariate compression for fronthaul transmission. We then compare this rate region with the distributed decode-forward (DDF) scheme, which achieves the capacity of the general relay networks to within a constant gap, and show that the difference lies in that DDF performs Marton's multicoding and multivariate compression jointly as opposed to successively as in the compression strategy. A main result of this paper is that under the assumption that the fronthaul links are subject to a \emph{sum} capacity constraint this difference is immaterial, so the successive encoding based compression strategy can already achieve the capacity region of the C-RAN to within a constant gap, where the gap is independent of the channel parameters and the power constraints at the BSs. For the special case of the Gaussian network, we further establish that under individual fronthaul constraints, the compression strategy achieves to within a constant gap to the \emph{sum} capacity of the C-RAN.

cs.IT↗

arXiv2018

Hybrid Data-Sharing and Compression Strategy for Downlink Cloud Radio Access Network

This paper studies transmission strategies for the downlink of a cloud radio access network, in which the base stations are connected to a centralized cloud-computing based processor with digital fronthaul or backhaul links. We provide a system-level performance comparison of two fundamentally different strategies, namely the data-sharing strategy and the compression strategy, that differ in the way the fronthaul/backhaul is utilized. It is observed that the performance of both strategies depends crucially on the available fronthaul or backhaul capacity. When the fronthaul/backhaul capacity is low, the datasharing strategy performs better, while under moderate-to-high fronthaul/backhaul capacity, the compression strategy is superior. Using insights from such a comparison, we propose a novel hybrid strategy, combining the data-sharing and compression strategies, that allows for better control over the fronthaul/backhaul capacity utilization. An optimization framework for the hybrid strategy is proposed. Numerical evidence demonstrates the performance gain of the hybrid strategy.

cs.IT↗

arXiv2018

Channel Diagonalization for Cloud Radio Access

The diagonalization of a conventional multiple-input multiple-output (MIMO) channel into parallel and independent subchannels via singular value decomposition (SVD) is a fundamental strategy that allows the MIMO channel capacity to be achieved using scalar channel codes. This letter establishes a similar channel diagonalization result for the uplink and the downlink of a cloud radio access network (C-RAN), in which a central processor (CP) is connected to a remote radio head (RRH) serving a single user via rate-limit digital fronthaul carrying the compressed baseband signal. Specifically, we show that the diagonalization of the MIMO channel between the RRH and the user via SVD and the subsequent independent and parallel quantization of scalar signals and channel coding in each of the subchannels is optimal. This letter establishes this fact using the majorization theory. Further, an uplink-downlink duality for the multiple-antenna C-RAN is identified for this single-user case.

cs.IT↗

arXiv2016

An uplink-downlink duality for cloud radio access network

Uplink-downlink duality refers to the fact that the Gaussian broadcast channel has the same capacity region as the dual Gaussian multiple-access channel under the same sumpower constraint. This paper investigates a similar duality relationship between the uplink and downlink of a cloud radio access network (C-RAN), where a central processor (CP) cooperatively serves multiple mobile users through multiple remote radio heads (RRHs) connected to the CP with finite-capacity fronthaul links. The uplink of such a C-RAN model corresponds to a multipleaccess relay channel; the downlink corresponds to a broadcast relay channel. This paper considers compression-based relay strategies in both uplink and downlink C-RAN, where the quantization noise levels are functions of the fronthaul link capacities. If the fronthaul capacities are infinite, the conventional uplinkdownlink duality applies. The main result of this paper is that even when the fronthaul capacities are finite, duality continues to hold for the case where independent compression is applied across each RRH in the sense that when the transmission and compression designs are jointly optimized, the achievable rate regions of the uplink and downlink remain identical under the same sum-power and individual fronthaul capacity constraints. As an application of the duality result, the power minimization problem in downlink C-RAN can be efficiently solved based on its uplink counterpart.

cs.IT↗

arXiv2013

Layered Constructions for Low-Delay Streaming Codes

We propose a new class of error correction codes for low-delay streaming communication. We consider an online setup where a source packet arrives at the encoder every $M$ channel uses, and needs to be decoded with a maximum delay of $T$ packets. We consider a sliding-window erasure channel --- $\cC(N,B,W)$ --- which introduces either up to $N$ erasures in arbitrary positions, or $B$ erasures in a single burst, in any window of length $W$. When $M=1$, the case where source-arrival and channel-transmission rates are equal, we propose a class of codes --- MiDAS codes --- that achieve a near optimal rate. Our construction is based on a {\em layered} approach. We first construct an optimal code for the $\cC(N=1,B,W)$ channel, and then concatenate an additional layer of parity-check symbols to deal with $N>1$. When $M > 1$, the case where source-arrival and channel-transmission rates are unequal, we characterize the capacity when $N=1$ and $W \ge M(T+1),$ and for $N>1$, we propose a construction based on a layered approach. Numerical simulations over Gilbert-Elliott and Fritchman channel models indicate significant gains in the residual loss probability over baseline schemes. We also discuss the connection between the error correction properties of the MiDAS codes and their underlying column distance and column span.

cs.IT↗