arXiv subjects

Yoav Benjamini

Publications and source records attributed to Yoav Benjamini.

At least 19 recordsLinked to original sources

arXiv2024

Direction Preferring Confidence Intervals

Confidence intervals (CIs) are instrumental in statistical analysis, providing a range estimate of the parameters. In modern statistics, selective inference is common, where only certain parameters are highlighted. However, this selective approach can bias the inference, leading some to advocate for the use of CIs over p-values. To increase the flexibility of confidence intervals, we introduce direction-preferring CIs, enabling analysts to focus on parameters trending in a particular direction. We present these types of CIs in two settings: First, when there is no selection of parameters; and second, for situations involving parameter selection, where we offer a conditional version of the direction-preferring CIs. Both of these methods build upon the foundations of Modified Pratt CIs, which rely on non-equivariant acceptance regions to achieve longer intervals in exchange for improved sign exclusions. We show that for selected parameters out of m > 1 initial parameters of interest, CIs aimed at controlling the false coverage rate, have higher power to determine the sign compared to conditional CIs. We also show that conditional confidence intervals control the marginal false coverage rate (mFCR) under any dependency.

stat.ME↗

arXiv2020

Time to adjust: Improving replicability in experimental psychology by adjustment for evident selective inference

The field of psychological sciences has been grappling with the replicability crisis. Various issues have been identified as potential sources of this problem. We bring to light a potential source that has largely been overlooked and demonstrate its significant contribution to the problem: the practice of multiple comparisons. We analyzed 88 papers from the Reproducibility Project in Psychology and found that multiple results are commonly reported in a single paper, ranging from 4 to 730 (M=77.7), without multiple comparison adjustments. We retroactively applied such an adjustment using a hierarchical FDR controlling procedure (TreeBH; Bogomolov et al., 2021). 21 of 88 results were deemed insignificant after adjustment. Twenty of these 21 results indeed failed to replicate, constituting over a third of the non-replicable findings, while maintaining 97% power. We propose that this should become a common practice as an essential means to increase replicability in experimental psychology.

stat.ME↗

arXiv2019

Testing the equality of multivariate means when $p>n$ by combining the Hoteling and Simes tests

We propose a method of testing the shift between mean vectors of two multivariate Gaussian random variables in a high-dimensional setting incorporating the possible dependency and allowing $p > n$. This method is a combination of two well-known tests: the Hotelling test and the Simes test. The tests are integrated by sampling several dimensions at each iteration, testing each using the Hotelling test, and combining their results using the Simes test. We prove that this procedure is valid asymptotically. This procedure can be extended to handle non-equal covariance matrices by plugging in the appropriate extension of the Hotelling test. Using a simulation study, we show that the proposed test is advantageous over state-of-the-art tests in many scenarios and robust to violation of the Gaussian assumption.

stat.ME↗

arXiv2019

Quantifying replicability and consistency in systematic reviews

Systematic reviews of interventions are important tools for synthesizing evidence from multiple studies. They serve to increase power and improve precision, in the same way that larger studies can do, but also to establish the consistency of effects and replicability of results across studies which are not identical. In this work we suggest to incorporate replicability analysis tools to quantify the consistency and conflict. These are offered both for the fixed-effect and for the random-effects meta-analyses. We motivate and demonstrate our approach and its implications by examples from systematic reviews from the Cochrane library, and offer a way to incorporate our suggestions in their standard reporting system.

stat.ME↗

arXiv2019

Confidence Intervals for Selected Parameters

Practical or scientific considerations often lead to selecting a subset of parameters as ``important.'' Inferences about those parameters often are based on the same data used to select them in the first place. That can make the reported uncertainties deceptively optimistic: confidence intervals that ignore selection generally have less than their nominal coverage probability. Controlling the probability that one or more intervals for selected parameters do not cover---the ``simultaneous over the selected'' (SoS) error rate---is crucial in many scientific problems. Intervals that control the SoS error rate can be constructed in ways that take advantage of knowledge of the selection rule. We construct SoS-controlling confidence intervals for parameters deemed the most ``important'' $k$ of $m$ shift parameters because they are estimated (by independent estimators) to be the largest. The new intervals improve substantially over Šidák intervals when $k$ is small compared to $m$, and approach the standard Bonferroni-corrected intervals when $k \approx m$. Standard, unadjusted confidence intervals for location parameters have the correct coverage probability for $k=1$, $m=2$ if, when the true parameters are zero, the estimators are exchangeable and symmetric.

stat.ME↗

arXiv2018

Testing hypotheses on a tree: new error rates and controlling strategies

We introduce a multiple testing procedure (TreeBH) which addresses the challenge of controlling error rates at multiple levels of resolution. Conceptually, we frame this problem as the selection of hypotheses which are organized hierarchically in a tree structure. We describe a fast algorithm for the proposed sequential procedure, and prove that it controls relevant error rates given certain assumptions on the dependence among the p-values. Through simulations, we demonstrate that TreeBH offers the desired guarantees under a range of dependency structures (including one similar to that encountered in genome-wide association studies) and that it has the potential of gaining power over alternative methods. We also introduce a modified version of TreeBH which we prove to control the relevant error rates under any dependency structure. We conclude with two case studies: we first analyze data collected as part of the Genotype-Tissue Expression (GTEx) project, which aims to characterize the genetic regulation of gene expression across multiple tissues in the human body, and secondly, data examining the relationship between the gut microbiome and colorectal cancer.

stat.ME↗

arXiv2015

Coping with Space Neophobia in Drosophila melanogaster: The Asymmetric Dynamics of Crossing a Doorway to the Untrodden

Insects exhibit remarkable cognitive skills in the field and several cognitive abilities have been demonstrated in Drosophila in the laboratory. By devising an ethologically relevant experimental setup that also allows comparison of behavior across remote taxonomic groups we sought to reduce the gap between the field and the laboratory, and reveal as yet undiscovered ethological phenomena within a wider phylogenetic perspective. We tracked individual flies that eclosed in a small (45mm) arena containing a piece of fruit, connected to a larger (130mm) arena by a wide (5mm) doorway. Using this setup we show that the widely open doorway initially functions as a barrier: the likelihood of entering the large arena increases gradually, requiring repeated approaches to the doorway, and even after entering the flies immediately return. Gradually the flies acquire the option to avoid returning, spending more relative time and performing relatively longer excursions in the large arena. The entire process may take up three successive days. This behavior constitutes coping with space neophobia, the avoidance of untrodden space. It appears to be the same as the neophobic doorway-crossing reported in mouse models of anxiety. In both mice and flies the moment-to-moment developmental dynamics of transition between trodden and untrodden terrain appear to be the same, and in mice it is taken to imply memory and, therefore, cognition. Recent claims have been made for a deep homology between the arthropod central complex and the vertebrate basal ganglia, two structures involved in navigation. The shared dynamics of space occupancy in flies and mice might indicate the existence of cognitive exploration also in the flies or else a convergent structure exhibiting the same developmental dynamics.

q-bio.NC↗

arXiv2015

Quantifying replicability in systematic reviews: the r-value

In order to assess the effect of a health care intervention, it is useful to look at an ensemble of relevant studies. The Cochrane Collaboration's admirable goal is to provide systematic reviews of all relevant clinical studies, in order to establish whether or not there is a conclusive evidence about a specific intervention. This is done mainly by conducting a meta-analysis: a statistical synthesis of results from a series of systematically collected studies. Health practitioners often interpret a significant meta-analysis summary effect as a statement that the treatment effect is consistent across a series of studies. However, the meta-analysis significance may be driven by an effect in only one of the studies. Indeed, in an analysis of two domains of Cochrane reviews we show that in a non-negligible fraction of reviews, the removal of a single study from the meta-analysis of primary endpoints makes the conclusion non-significant. Therefore, reporting the evidence towards replicability of the effect across studies in addition to the significant meta-analysis summary effect will provide credibility to the interpretation that the effect was replicated across studies. We suggest an objective, easily computed quantity, we term the r-value, that quantifies the extent of this reliance on single studies. We suggest adding the r-values to the main results and to the forest plots of systematic reviews.

stat.AP↗

arXiv2015

Many Phenotypes without Many False Discoveries: Error Controlling Strategies for Multi-Traits Association Studies

The genetic basis of multiple phenotypes such as gene expression, metabolite levels, or imaging features is often investigated by testing a large collection of hypotheses, probing the existence of association between each of the traits and hundreds of thousands of genotyped variants. Appropriate multiplicity adjustment is crucial to guarantee replicability of findings, and False Discovery Rate (FDR) is frequently adopted as a measure of global error. In the interest of interpretability, results are often summarized so that reporting focuses on variants discovered to be associated to some phenotypes. We show that applying FDR-controlling procedures on the entire collection of hypotheses fails to control the rate of false discovery of associated variants as well as the average rate of false discovery of phenotypes influenced by such variants. We propose a simple hierarchical testing procedure which allows control of both these error rates and provides a more reliable basis for the identification of variants with functional effects. We demonstrate the utility of this approach through simulation studies comparing various error rates and measures of power for genetic association studies of multiple traits. Finally, we apply the proposed method to identify genetic variants which impact flowering phenotypes in Arabdopsis thaliana, expanding the set of discoveries.

stat.AP↗

arXiv2015

Testing for replicability in a follow-up study when the primary study hypotheses are two-sided

When testing for replication of results from a primary study with two-sided hypotheses in a follow-up study, we are usually interested in discovering the features with discoveries in the same direction in the two studies. The direction of testing in the follow-up study for each feature can therefore be decided by the primary study. We prove that in this case the methods suggested in Heller, Bogomolov, and Benjamini (2014) for control over false replicability claims are valid. Specifically, we prove that if we input into the procedures in Heller, Bogomolov, and Benjamini (2014) the one-sided p-values in the directions favoured by the primary study, then we achieve directional control over the desired error measure (family-wise error rate or false discovery rate).

stat.ME↗

arXiv2014

Selective Correlations - the conditional estimators

The problem of Voodoo correlations is recognized in neuroimaging as the problem of estimating quantities of interest from the same data that was used to select them as interesting. In statistical terminology, the problem of inference following selection from the same data is that of selective inference. Motivated by the unwelcome side-effects of the recommended remedy- splitting the data. A method for constructing confidence intervals based on the correct post-selection distribution of the observations has been suggested recently. We utilize a similar approach in order to provide point estimates that account for a large part of the selection bias. We show via extensive simulations that the proposed estimator has favorable properties, namely, that it is likely to reduce estimation bias and the mean squared error compared to the direct estimator without sacrificing power to detect non-zero correlation as in the case of the data splitting approach. We show that both point estimates and confidence intervals are needed in order to get a full assessment of the uncertainty in the point estimates as both are integrated into the Confidence Calibration Plots proposed recently. The computation of the estimators is implemented in an accompanying software package.

stat.ME↗

arXiv2014

Deciding whether follow-up studies have replicated findings in a preliminary large-scale "omics' study"

We propose a formal method to declare that findings from a primary study have been replicated in a follow-up study. Our proposal is appropriate for primary studies that involve large-scale searches for rare true positives (i.e. needles in a haystack). Our proposal assigns an $r$-value to each finding; this is the lowest false discovery rate at which the finding can be called replicated. Examples are given and software is available.

stat.AP↗

arXiv2013

Another Argument in Favour of Wilcoxon's Signed Rank Test

The Wilcoxon Signed Rank test is typically called upon when testing whether a symmetric distribution has a specified centre and the Gaussianity is in question. As with all insurance policies it comes with a cost, even if small, in terms of power versus a t-test, when the distribution is indeed Gaussian. In this note we further show that even when the distribution tested is Gaussian there need not be power loss at all, if the alternative is of a mixture type rather than a shift. The signed rank test may turn out to be more powerful than the t-test, and the supposedly conservative strategy, might actually be the more powerful one. Drug testing and functional magnetic imaging are two such scenarios. Wilcoxon' signed rank test will typically be called upon by a researcher when testing for the location of a single population, using a small sample and Gaussianity is dubious. As all insurance policies, it will come with a cost-- power. It is well known, that under a Gaussian setup, the signed rank test is less powerful than, say, a t-test. The works of Pitman and others have reassured us that this power loss is surprisingly small. In this note we argue that the power loss might actually be smaller than typically assumed. In particular, if the deviation from the null Gaussian distribution is of a mixture type and not a shift type, the signed rank test is no longer dominated by the t-test and can actually be more powerful.

math.ST↗

arXiv2013

Revisiting Multi-Subject Random Effects in fMRI: Advocating Prevalence Estimation

Random Effects analysis has been introduced into fMRI research in order to generalize findings from the study group to the whole population. Generalizing findings is obviously harder than detecting activation in the study group since in order to be significant, an activation has to be larger than the inter-subject variability. Indeed, detected regions are smaller when using random effect analysis versus fixed effects. The statistical assumptions behind the classic random effects model are that the effect in each location is normally distributed over subjects, and "activation" refers to a non-null mean effect. We argue this model is unrealistic compared to the true population variability, where, due to functional plasticity and registration anomalies, at each brain location some of the subjects are active and some are not. We propose a finite-Gaussian--mixture--random-effect. A model that amortizes between-subject spatial disagreement and quantifies it using the "prevalence" of activation at each location. This measure has several desirable properties: (a) It is more informative than the typical active/inactive paradigm. (b) In contrast to the hypothesis testing approach (thus t-maps) which are trivially rejected for large sample sizes, the larger the sample size, the more informative the prevalence statistic becomes. In this work we present a formal definition and an estimation procedure of this prevalence. The end result of the proposed analysis is a map of the prevalence at locations with significant activation, highlighting activations regions that are common over many brains.

stat.AP↗

arXiv2011

Adjusting for selection bias in testing multiple families of hypotheses

In many large multiple testing problems the hypotheses are divided into families. Given the data, families with evidence for true discoveries are selected, and hypotheses within them are tested. Neither controlling the error-rate in each family separately nor controlling the error-rate over all hypotheses together can assure that an error-rate is controlled in the selected families. We formulate this concern about selective inference in its generality, for a very wide class of error-rates and for any selection criterion, and present an adjustment of the testing level inside the selected families that retains the average error-rate over the selected families.

math.ST↗

arXiv2010

High-throughput data analysis in behavior genetics

In recent years, a growing need has arisen in different fields for the development of computational systems for automated analysis of large amounts of data (high-throughput). Dealing with nonstandard noise structure and outliers, that could have been detected and corrected in manual analysis, must now be built into the system with the aid of robust methods. We discuss such problems and present insights and solutions in the context of behavior genetics, where data consists of a time series of locations of a mouse in a circular arena. In order to estimate the location, velocity and acceleration of the mouse, and identify stops, we use a nonstandard mix of robust and resistant methods: LOWESS and repeated running median. In addition, we argue that protection against small deviations from experimental protocols can be handled automatically using statistical methods. In our case, it is of biological interest to measure a rodent's distance from the arena's wall, but this measure is corrupted if the arena is not a perfect circle, as required in the protocol. The problem is addressed by estimating robustly the actual boundary of the arena and its center using a nonparametric regression quantile of the behavioral data, with the aid of a fast algorithm developed for that purpose.

stat.AP↗

arXiv2009

A simple forward selection procedure based on false discovery rate control

We propose the use of a new false discovery rate (FDR) controlling procedure as a model selection penalized method, and compare its performance to that of other penalized methods over a wide range of realistic settings: nonorthogonal design matrices, moderate and large pool of explanatory variables, and both sparse and nonsparse models, in the sense that they may include a small and large fraction of the potential variables (and even all). The comparison is done by a comprehensive simulation study, using a quantitative framework for performance comparisons in the form of empirical minimaxity relative to a "random oracle": the oracle model selection performance on data dependent forward selected family of potential models. We show that FDR based procedures have good performance, and in particular the newly proposed method, emerges as having empirical minimax performance. Interestingly, using FDR level of 0.05 is a global best.

stat.AP↗

arXiv2009

An adaptive step-down procedure with proven FDR control under independence

In this work we study an adaptive step-down procedure for testing $m$ hypotheses. It stems from the repeated use of the false discovery rate controlling the linear step-up procedure (sometimes called BH), and makes use of the critical constants $iq/[(m+1-i(1-q)]$, $i=1,...,m$. Motivated by its success as a model selection procedure, as well as by its asymptotic optimality, we are interested in its false discovery rate (FDR) controlling properties for a finite number of hypotheses. We prove this step-down procedure controls the FDR at level $q$ for independent test statistics. We then numerically compare it with two other procedures with proven FDR control under independence, both in terms of power under independence and FDR control under positive dependence.

math.ST↗