arXiv subjects

Ruth Heller

Publications and source records attributed to Ruth Heller.

At least 19 recordsLinked to original sources

arXiv2026

Selecting Informative Conformal Prediction Sets with an Optimized FCR-Controlled Approach

Conformal methods provide prediction sets for outcomes with confidence guarantees. We study their use in a selective inference setting, where inference is performed only when the prediction set is informative. The analyst may consider as informative, for example, cases with prediction sets that are sufficiently small, exclude null values, or satisfy other appropriate monotone constraints. Because inference is typically restricted to informative cases in practical applications, accounting for the resulting selection bias is crucial to maintaining false coverage rate (FCR) control. A general framework for constructing such informative conformal prediction sets while controlling the FCR on the selected sample was suggested in Gazin et al. (2025). In this work we focus on oracle-guided procedures. We derive the optimal decision policy under a suitable power objective in the oracle setting where the probability of belonging to each prediction set can be computed. In practice, of course, only estimated probabilities are available. We therefore introduce a calibration procedure that adjusts the oracle policy to maintain finite sample FCR control. We show that this approach can achieve substantially higher power than available alternatives. We demonstrate the effectiveness of our new methods for classification outcomes on both real and simulated data.

stat.ME↗

arXiv2025

The Bottom-Up Approach for Powerful Testing with FWER Control

We seek to design novel multiple testing procedures, which take into account a relevant notion of ''power'' or true discovery on the one hand, and allow computationally efficient test design and application on the other. Towards this end we characterize the optimal procedures that strongly control the family-wise error rate, for a range of power objectives measuring the success of multiple testing procedures in making true individual discoveries, and under a reasonable set of assumptions. While we cannot generally find these optimal solutions in practice, we propose the bottom-up approach, which constructs consonant closed testing procedures, while taking into account the overall power objective in designing the tests on every level of the closed testing hierarchy. This leads to a general recipe, yielding novel procedures which are computationally practical and demonstrate substantially improved power in both simulations and a real data study, compared to existing procedures.

stat.ME↗

arXiv2025

Online selective conformal inference: adaptive scores, convergence rates and optimality

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of Gibbs and Cand\`es (2021). For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average miscoverage getting close to a prespecified level $\alpha$ for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. OnlineSCI controls the false coverage proportion among selected times via a pathwise bound for arbitrary sequences, as well as the instantaneous error rate conditional on selection, up to a non-asymptotic remainder term, under stochastic assumptions. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {\it adaptive} capability. We show that the adaptive versions of OnlineSCI can converge to an optimal solution and provide an explicit convergence rate in several application settings, under model- and estimator-specific regularity conditions. The favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.

math.ST↗

arXiv2025

Exploration, Confirmation, and Replication in the Same Observational Study: A Two Team Cross-Screening Approach to Studying the Effect of Unwanted Pregnancy on Mothers' Later Life Outcomes

The long term consequences of unwanted pregnancies carried to term on mothers have not been much explored. We use data from the Wisconsin Longitudinal Study (WLS) and propose a novel approach, namely two team cross-screening, to study the possible effects of unwanted pregnancies carried to term on various aspects of mothers' later-life mental health, physical health, economic well-being and life satisfaction. Our method, unlike existing approaches to observational studies, enables the investigators to perform exploratory data analysis, confirmatory data analysis and replication in the same study. This is a valuable property when there is only a single data set available with unique strengths to perform exploratory, confirmatory and replication analysis. In two team cross-screening, the investigators split themselves into two teams and the data is split as well according to a meaningful covariate. Each team then performs exploratory data analysis on its part of the data to design an analysis plan for the other part of the data. The complete freedom of the teams in designing the analysis has the potential to generate new unanticipated hypotheses in addition to a prefixed set of hypotheses. Moreover, only the hypotheses that looked promising in the data each team explored are forwarded for analysis (thus alleviating the multiple testing problem). These advantages are demonstrated in our study of the effects of unwanted pregnancies on mothers' later life outcomes.

stat.ME↗

arXiv2024

Protocol for an Observational Study on the Effects of Paternal Alcohol Use Disorder on Children's Later Life Outcomes

The harmful effects of growing up with a parent with an alcohol use disorder have been closely examined in children and adolescents, and are reported to include mental and physical health problems, interpersonal difficulties, and a worsened risk of future substance use disorders. However, few studies have investigated how these impacts evolve into later life adulthood, leaving the ensuing long-term effects of interest. In this article, we provide the protocol for our observational study of the long-term consequences of growing up with a father who had an alcohol use disorder. We will use data from the Wisconsin Longitudinal Study to examine impacts on long-term economic success, interpersonal relationships, physical, and mental health. To reinforce our findings, we will conduct this investigation on two discrete subpopulations of individuals in our study, allowing us to analyze the replicability of our conclusions. We introduce a novel statistical design, called data turnover, to carry out this analysis. Data turnover allows a single group of statisticians and domain experts to work together to assess the strength of evidence gathered across multiple data splits, while incorporating both qualitative and quantitative findings from data exploration. We delineate our analysis plan using this new method and conclude with a brief discussion of some additional considerations for our study.

stat.AP↗

arXiv2024

Powerful batch conformal prediction for classification

In a split conformal framework with $K$ classes, a calibration sample of $n$ labeled examples is observed for inference on the label of a new unlabeled example. We explore the setting where a `batch' of $m$ independent such unlabeled examples is given, and the goal is to construct a batch prediction set with 1-$\alpha$ coverage. Unlike individual prediction sets, the batch prediction set is a collection of label vectors of size $m$, while the calibration sample consists of univariate labels. A natural approach is to apply the Bonferroni correction, which concatenates individual prediction sets at level $1-\alpha/m$. We propose a uniformly more powerful solution, based on specific combinations of conformal $p$-values that exploit the Simes inequality. We provide a general recipe for valid inference with any combinations of conformal $p$-values, and compare the performance of several useful choices. Intuitively, the pooled evidence of relatively `easy' examples within the batch can help provide narrower batch prediction sets. Additionally, we introduce a more computationally intensive method that aggregates batch scores and can be even more powerful. The theoretical guarantees are established when all examples are independent and identically distributed (iid), as well as more generally when iid is assumed only conditionally within each class. Notably, our results remain valid under label distribution shift, since the distribution of the labels need not be the same in the calibration sample and in the new batch. The effectiveness of the methods is highlighted through illustrative synthetic and real data examples.

stat.ME↗

arXiv2024

Direction Preferring Confidence Intervals

Confidence intervals (CIs) are instrumental in statistical analysis, providing a range estimate of the parameters. In modern statistics, selective inference is common, where only certain parameters are highlighted. However, this selective approach can bias the inference, leading some to advocate for the use of CIs over p-values. To increase the flexibility of confidence intervals, we introduce direction-preferring CIs, enabling analysts to focus on parameters trending in a particular direction. We present these types of CIs in two settings: First, when there is no selection of parameters; and second, for situations involving parameter selection, where we offer a conditional version of the direction-preferring CIs. Both of these methods build upon the foundations of Modified Pratt CIs, which rely on non-equivariant acceptance regions to achieve longer intervals in exchange for improved sign exclusions. We show that for selected parameters out of m > 1 initial parameters of interest, CIs aimed at controlling the false coverage rate, have higher power to determine the sign compared to conditional CIs. We also show that conditional confidence intervals control the marginal false coverage rate (mFCR) under any dependency.

stat.ME↗

arXiv2024

Selecting informative conformal prediction sets with false coverage rate control

In supervised learning, including regression and classification, conformal methods provide prediction sets for the outcome/label with finite sample coverage for any machine learning predictor. We consider here the case where such prediction sets come after a selection process. The selection process requires that the selected prediction sets be `informative' in a well defined sense. We consider both the classification and regression settings where the analyst may consider as informative only the sample with prediction sets small enough, excluding null values, or obeying other appropriate `monotone' constraints. We develop a unified framework for building such informative conformal prediction sets while controlling the false coverage rate (FCR) on the selected sample. While conformal prediction sets after selection have been the focus of much recent literature in the field, the new introduced procedures, called InfoSP and InfoSCOP, are to our knowledge the first ones providing FCR control for informative prediction sets. We show the usefulness of our resulting procedures on real and simulated data.

math.ST↗

arXiv2023

Simultaneous directional inference

We consider the problem of inference on the signs of $n>1$ parameters. We aim to provide $1-α$ post-hoc confidence bounds on the number of positive and negative (or non-positive) parameters. The guarantee is simultaneous, for all subsets of parameters. Our suggestion is as follows: start by using the data to select the direction of the hypothesis test for each parameter; then, adjust the $p$-values of the one-sided hypotheses for the selection, and use the adjusted $p$-values for simultaneous inference on the selected $n$ one-sided hypotheses. The adjustment is straightforward assuming that the $p$-values of one-sided hypotheses have densities with monotone likelihood ratio, and are mutually independent. We show that the bounds we provide are tighter (often by a great margin) than existing alternatives, and that they can be obtained by at most a polynomial time. We demonstrate the usefulness of our simultaneous post-hoc bounds in the evaluation of treatment effects across studies or subgroups. Specifically, we provide a tight lower bound on the number of studies which are beneficial, as well as on the number of studies which are harmful (or non-beneficial), and in addition conclude on the effect direction of individual studies, while guaranteeing that the probability of at least one wrong inference is at most 0.05.

stat.ME↗

arXiv2023

Replicability Across Multiple Studies

Meta-analysis is routinely performed in many scientific disciplines. This analysis is attractive since discoveries are possible even when all the individual studies are underpowered. However, the meta-analytic discoveries may be entirely driven by signal in a single study, and thus non-replicable. Although the great majority of meta-analyses carried out to date do not infer on the replicability of their findings, it is possible to do so. We provide a selective overview of analyses that can be carried out towards establishing replicability of the scientific findings. We describe methods for the setting where a single outcome is examined in multiple studies (as is common in systematic reviews of medical interventions), as well as for the setting where multiple studies each examine multiple features (as in genomics applications). We also discuss some of the current shortcomings and future directions.

stat.ME↗

arXiv2023

Protocol for an Observational Study on the Effects of Giving Births from Unintended Pregnancies on Later Life Physical and Mental Health

There has been increasing interest in studying the effect of giving births to unintended pregnancies on later life physical and mental health. In this article, we provide the protocol for our planned observational study on the long-term mental and physical health consequences for mothers who bear children resulting from unintended pregnancies. We aim to use the data from the Wisconsin Longitudinal Study (WLS) and examine the effect of births from unintended pregnancies on a broad range of outcomes, including mental depression, psychological well-being, physical health, alcohol usage, and economic well-being. To strengthen our causal findings, we plan to address our research questions on two subgroups, Catholics and non-Catholics, and discover the "replicable" outcomes for which the effect of unintended pregnancy is negative (or, positive) in both subgroups. Following the idea of non-random cross-screening, the data will be split according to whether the woman is Catholic or not, and then one part of the data will be used to select the hypotheses and design the corresponding tests for the second part of the data. In past use of cross-screening (automatic cross-screening) there was only one team of investigators that dealt with both parts of the data so that the investigators would need to decide on an analysis plan before looking at the data. In this protocol, we describe plans to carry out a novel flexible cross-screening in which there will be two teams of investigators with access only to one part of data and each team will use their part of the data to decide how to plan the analysis for the second team's data. In addition to the above replicability analysis, we also discuss the plan to test the global null hypothesis that is intended to identify the outcomes which are affected by unintended pregnancy for at least one of the two subgroups of Catholics and non-Catholics.

stat.AP↗

arXiv2022

Inference with approximate local false discovery rates

Efron's two-group model is widely used in large scale multiple testing. This model assumes that test statistics are mutually independent, however in realistic settings they are typically dependent, and taking the dependence into account can boost power. The general two-group model takes the dependence between the test statistics into account. Optimal policies in the general two-group model require calculation, for each hypothesis, of the probability that it is a true null given all test statistics, denoted local false discovery rate (locFDR). Unfortunately, calculating locFDRs under realistic dependence structures can be computationally prohibitive. We propose calculating approximate locFDRs based on a properly defined N-neighborhood for each hypothesis. We prove that by thresholding the approximate locFDRs with a fixed threshold, the marginal false discovery rate is controlled for any dependence structure. Furthermore, we prove that this is the optimal procedure in a restricted class of decision rules, where decision for each hypothesis is only guided by its N-neighborhood. We show through extensive simulations that our proposed method achieves substantial power gains compared to alternative practical approaches, while maintaining conceptual simplicity and computational feasibility. We demonstrate the utility of our method on a genome wide association study of height.

stat.ME↗

arXiv2022

Inferring on joint associations from marginal associations and a reference sample

We present a method to infer on joint regression coefficients obtained from marginal regressions using a reference panel. This type of scenario is common in genetic fine-mapping, where the estimated marginal associations are reported in genomewide association studies (GWAS), and a reference panel is used for inference on the association in a joint regression model. We show that ignoring the uncertainty due to the use of a reference panel instead of the original design matrix, can lead to a severe inflation of false discoveries and a lack of replicable findings. We derive the asymptotic distribution of the estimated coefficients in the joint regression model, and show how it can be used to produce valid inference. We address two settings: inference within regions that are pre-selected, as well as within regions that are selected based on the same data. By means of real data examples and simulations we demonstrate the usefulness of our suggested methodology.

stat.ME↗

arXiv2022

Optimal multiple testing and design in clinical trials

A central goal in designing clinical trials is to find the test that maximizes power (or equivalently minimizes required sample size) for finding a false null hypothesis subject to the constraint of type I error. When there is more than one test, such as in clinical trials with multiple endpoints, the issues of optimal design and optimal procedures become more complex. In this paper we address the question of how such optimal tests should be defined and how they can be found. We review different notions of power and how they relate to study goals, and also consider the requirements of type I error control and the nature of the procedures. This leads us to an explicit optimization problem with objective and constraints which describe its specific desiderata. We present a complete solution for deriving optimal procedures for two hypotheses, which have desired monotonicity properties, and are computationally simple. For some of the optimization formulations this yields optimal procedures that are identical to existing procedures, such as Hommel's procedure or the procedure of Bittman et al. (2009), while for others it yields completely novel and more powerful policies than existing ones. We demonstrate the nature of our novel policies and their improved power extensively in simulation and on the APEX study (Cohen et al., 2016).

stat.ME↗

arXiv2022

A statistical framework for efficient out of distribution detection in deep neural networks

Background. Commonly, Deep Neural Networks (DNNs) generalize well on samples drawn from a distribution similar to that of the training set. However, DNNs' predictions are brittle and unreliable when the test samples are drawn from a dissimilar distribution. This is a major concern for deployment in real-world applications, where such behavior may come at a considerable cost, such as industrial production lines, autonomous vehicles, or healthcare applications. Contributions. We frame Out Of Distribution (OOD) detection in DNNs as a statistical hypothesis testing problem. Tests generated within our proposed framework combine evidence from the entire network. Unlike previous OOD detection heuristics, this framework returns a $p$-value for each test sample. It is guaranteed to maintain the Type I Error (T1E - incorrectly predicting OOD for an actual in-distribution sample) for test data. Moreover, this allows to combine several detectors while maintaining the T1E. Building on this framework, we suggest a novel OOD procedure based on low-order statistics. Our method achieves comparable or better results than state-of-the-art methods on well-accepted OOD benchmarks, without retraining the network parameters or assuming prior knowledge on the test distribution -- and at a fraction of the computational cost.

cs.LG↗

arXiv2021

A procedure for multiple testing of partial conjunction hypotheses based on a hazard rate inequality

The partial conjunction null hypothesis is tested in order to discover a signal that is present in multiple studies. The standard approach of carrying out a multiple test procedure on the partial conjunction (PC) $p$-values can be extremely conservative. We suggest alleviating this conservativeness, by eliminating many of the conservative PC $p$-values prior to the application of a multiple test procedure. This leads to the following two step procedure: first, select the set with PC $p$-values below a selection threshold; second, within the selected set only, apply a family-wise error rate or false discovery rate controlling procedure on the conditional PC $p$-values. The conditional PC $p$-values are valid if the null p-values are uniform and the combining method is Fisher. The proof of their validity is based on a novel inequality in hazard rate order of partial sums of order statistics which may be of independent interest. We also provide the conditions for which the false discovery rate controlling procedures considered will be below the nominal level. We demonstrate the potential usefulness of our novel method, CoFilter (conditional testing after filtering), for analyzing multiple genome wide association studies of Crohn's disease.

stat.ME↗

arXiv2021

Quantifying replicability and consistency in systematic reviews

Systematic reviews of interventions are important tools for synthesizing evidence from multiple studies. They serve to increase power and improve precision, in the same way that larger studies can do, but also to establish the consistency of effects and replicability of results across studies which are not identical. In this work we suggest to incorporate replicability analysis tools to quantify the consistency and conflict. These are offered both for the fixed-effect and for the random-effects meta-analyses. We motivate and demonstrate our approach and its implications by examples from systematic reviews from the Cochrane library, and offer a way to incorporate our suggestions in their standard reporting system.

stat.ME↗

arXiv2020

Optimal and Maximin Procedures for Multiple Testing Problems

Multiple testing problems are a staple of modern statistical analysis. The fundamental objective of multiple testing procedures is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family-wise error rate (FWER) or false discovery rate (FDR). In this paper we formulate multiple testing of simple hypotheses as an infinite-dimensional optimization problem, seeking the most powerful rejection policy which guarantees strong control of the selected measure. In that sense, our approach is a generalization of the optimal Neyman-Pearson test for a single hypothesis. We show that for exchangeable hypotheses, for both FWER and FDR and relevant notions of power, these problems can be formulated as infinite linear programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives. We derive explicit optimal tests for FWER or FDR control for three independent normal means. We find that the power gain over natural competitors is substantial in all settings examined. Finally, we apply our optimal maximin rule to subgroup analyses in systematic reviews from the Cochrane library, leading to an increase in the number of findings while guaranteeing strong FWER control against the one sided alternative.

stat.ME↗