arXiv subjects

Ruth Heller

Publications and source records attributed to Ruth Heller.

34 records · Page 2Linked to original sources

arXiv2020

Optimal control of false discovery criteria in the two-group model

The highly influential two-group model in testing a large number of statistical hypotheses assumes that the test statistics are drawn independently from a mixture of a high probability null distribution and a low probability alternative. Optimal control of the marginal false discovery rate (mFDR), in the sense that it provides maximal power (expected true discoveries) subject to mFDR control, is known to be achieved by thresholding the local false discovery rate (locFDR), i.e., the probability of the hypothesis being null given the set of test statistics, with a fixed threshold. We address the challenge of controlling optimally the popular false discovery rate (FDR) or positive FDR (pFDR) rather than mFDR in the general two-group model, which also allows for dependence between the test statistics. These criteria are less conservative than the mFDR criterion, so they make more rejections in expectation. We derive their optimal multiple testing (OMT) policies, which turn out to be thresholding the locFDR with a threshold that is a function of the entire set of statistics. We develop an efficient algorithm for finding these policies, and use it for problems with thousands of hypotheses. We illustrate these procedures on gene expression studies.

stat.ME↗

arXiv2020

Testing for differential abundance in compositional counts data, with application to microbiome studies

Identifying which taxa in our microbiota are associated with traits of interest is important for advancing science and health. However, the identification is challenging because the measured vector of taxa counts (by amplicon sequencing) is compositional, so a change in the abundance of one taxon in the microbiota induces a change in the number of sequenced counts across all taxa. The data is typically sparse, with zero counts present either due to biological variance or limited sequencing depth (technical zeros). For low abundance taxa, the chance for technical zeros is non-negligible. We show that existing methods designed to identify differential abundance for compositional data may have an inflated number of false positives due to improper handling of the zero counts. We introduce a novel non-parametric approach which provides valid inference even when the fraction of zero counts is substantial. Our approach uses a set of reference taxa that are non-differentially abundant, which can be estimated from the data or from outside information. We show the usefulness of our approach via simulations, as well as on three different data sets: a Crohn's disease study, the Human Microbiome Project, and an experiment with 'spiked-in' bacteria.

q-bio.GN↗

arXiv2017

Post-selection estimation and testing following aggregated association tests

The practice of pooling several individual test statistics to form aggregate tests is common in many statistical application where individual tests may be underpowered. While selection by aggregate tests can serve to increase power, the selection process invalidates the individual test-statistics, making it difficult to identify the ones that drive the signal in follow-up inference. Here, we develop a general approach for valid inference following selection by aggregate testing. We present novel powerful post-selection tests for the individual null hypotheses which are exact for the normal model and asymptotically justified otherwise. Our approach relies on the ability to characterize the distribution of the individual test statistics after conditioning on the event of selection. We provide efficient algorithms for estimation of the post-selection maximum-likelihood estimates and suggest confidence intervals which rely on a novel switching regime for good coverage guarantees. We validate our methods via comprehensive simulation studies and apply them to data from the Dallas Heart Study, demonstrating that single variant association discovery following selection by an aggregated test is indeed possible in practice.

stat.ME↗

arXiv2016

Computing the Bergsma Dassios sign-covariance

Bergsma and Dassios (2014) introduced an independence measure which is zero if and only if two random variables are independent. This measure can be naively calculated in $O(n^4)$. Weihs et al. (2015) showed that it can be calculated in $O(n^2 \log n)$. In this note we will show that using the methods described in Heller et al. (2016), the measure can easily be calculated in only $O(n^2)$.

stat.ME↗

arXiv2016

Multivariate tests of association based on univariate tests

For testing two random vectors for independence, we consider testing whether the distance of one vector from a center point is independent from the distance of the other vector from a center point by a univariate test. In this paper we provide conditions under which it is enough to have a consistent univariate test of independence on the distances to guarantee that the power to detect dependence between the random vectors increases to one, as the sample size increases. These conditions turn out to be minimal. If the univariate test is distribution-free, the multivariate test will also be distribution-free. If we consider multiple center points and aggregate the center-specific univariate tests, the power may be further improved, and the resulting multivariate test may be distribution-free for specific aggregation methods (if the univariate test is distribution-free). We show that several multivariate tests recently proposed in the literature can be viewed as instances of this general approach.

stat.ME↗

arXiv2015

Consistent distribution-free $K$-sample and independence tests for univariate random variables

A popular approach for testing if two univariate random variables are statistically independent consists of partitioning the sample space into bins, and evaluating a test statistic on the binned data. The partition size matters, and the optimal partition size is data dependent. While for detecting simple relationships coarse partitions may be best, for detecting complex relationships a great gain in power can be achieved by considering finer partitions. We suggest novel consistent distribution-free tests that are based on summation or maximization aggregation of scores over all partitions of a fixed size. We show that our test statistics based on summation can serve as good estimators of the mutual information. Moreover, we suggest regularized tests that aggregate over all partition sizes, and prove those are consistent too. We provide polynomial-time algorithms, which are critical for computing the suggested test statistics efficiently. We show that the power of the regularized tests is excellent compared to existing tests, and almost as powerful as the tests based on the optimal (yet unknown in practice) partition size, in simulations as well as on a real data example.

stat.ME↗

arXiv2015

Quantifying replicability in systematic reviews: the r-value

In order to assess the effect of a health care intervention, it is useful to look at an ensemble of relevant studies. The Cochrane Collaboration's admirable goal is to provide systematic reviews of all relevant clinical studies, in order to establish whether or not there is a conclusive evidence about a specific intervention. This is done mainly by conducting a meta-analysis: a statistical synthesis of results from a series of systematically collected studies. Health practitioners often interpret a significant meta-analysis summary effect as a statement that the treatment effect is consistent across a series of studies. However, the meta-analysis significance may be driven by an effect in only one of the studies. Indeed, in an analysis of two domains of Cochrane reviews we show that in a non-negligible fraction of reviews, the removal of a single study from the meta-analysis of primary endpoints makes the conclusion non-significant. Therefore, reporting the evidence towards replicability of the effect across studies in addition to the significant meta-analysis summary effect will provide credibility to the interpretation that the effect was replicated across studies. We suggest an objective, easily computed quantity, we term the r-value, that quantifies the extent of this reliance on single studies. We suggest adding the r-values to the main results and to the forest plots of systematic reviews.

stat.AP↗

arXiv2015

Assessing replicability of findings across two studies of multiple features

Replicability analysis aims to identify the findings that replicated across independent studies that examine the same features. We provide powerful novel replicability analysis procedures for two studies for FWER and for FDR control on the replicability claims. The suggested procedures first select the promising features from each study solely based on that study, and then test for replicability only the features that were selected in both studies. We incorporate the plug-in estimates of the fraction of null hypotheses in one study among the selected hypotheses by the other study. Since the fraction of nulls in one study among the selected features from the other study is typically small, the power gain can be remarkable. We provide theoretical guarantees for the control of the appropriate error rates, as well as simulations that demonstrate the excellent power properties of the suggested procedures. We demonstrate the usefulness of our procedures on real data examples from two application fields: behavioural genetics and microarray studies.

stat.ME↗

arXiv2015

Testing for replicability in a follow-up study when the primary study hypotheses are two-sided

When testing for replication of results from a primary study with two-sided hypotheses in a follow-up study, we are usually interested in discovering the features with discoveries in the same direction in the two studies. The direction of testing in the follow-up study for each feature can therefore be decided by the primary study. We prove that in this case the methods suggested in Heller, Bogomolov, and Benjamini (2014) for control over false replicability claims are valid. Specifically, we prove that if we input into the procedures in Heller, Bogomolov, and Benjamini (2014) the one-sided p-values in the directions favoured by the primary study, then we achieve directional control over the desired error measure (family-wise error rate or false discovery rate).

stat.ME↗

arXiv2014

Consistent distribution-free tests of association between univariate random variables

We consider the problem of testing whether pairs of univariate random variables are associated. Few tests of independence exist that are consistent against all dependent alternatives and are distribution free. We propose novel tests that are consistent, distribution free, and have excellent power properties. The tests have simple form, and are surprisingly computationally efficient thanks to accompanying innovative algorithms we develop. Moreover, we show that one of the test statistics is a consistent estimator of the mutual information. We demonstrate the good power properties in simulations, and apply the tests to a microarray study where many pairs of genes are examined simultaneously for co-dependence.

stat.ME↗

arXiv2014

Deciding whether follow-up studies have replicated findings in a preliminary large-scale "omics' study"

We propose a formal method to declare that findings from a primary study have been replicated in a follow-up study. Our proposal is appropriate for primary studies that involve large-scale searches for rare true positives (i.e. needles in a haystack). Our proposal assigns an $r$-value to each finding; this is the lowest false discovery rate at which the finding can be called replicated. Examples are given and software is available.

stat.AP↗

arXiv2014

Replicability analysis for genome-wide association studies

The paramount importance of replicating associations is well recognized in the genome-wide associaton (GWA) research community, yet methods for assessing replicability of associations are scarce. Published GWA studies often combine separately the results of primary studies and of the follow-up studies. Informally, reporting the two separate meta-analyses, that of the primary studies and follow-up studies, gives a sense of the replicability of the results. We suggest a formal empirical Bayes approach for discovering whether results have been replicated across studies, in which we estimate the optimal rejection region for discovering replicated results. We demonstrate, using realistic simulations, that the average false discovery proportion of our method remains small. We apply our method to six type two diabetes (T2D) GWA studies. Out of 803 SNPs discovered to be associated with T2D using a typical meta-analysis, we discovered 219 SNPs with replicated associations with T2D. We recommend complementing a meta-analysis with a replicability analysis for GWA studies.

stat.ME↗

arXiv2014

False discovery rate controlling procedures for discrete tests

Benjamini and Hochberg (1995) proposed the false discovery rate (FDR) as an alternative to the family-wise error rate in multiple testing problems, and proposed a procedure to control the FDR. For discrete data this procedure may be highly conservative. We investigate alternative, more powerful, procedures that exploit the discreteness of the tests and have FDR levels closer in magnitude to the desired nominal level. Moreover, we develop a novel step-down procedure that dominates the step-down procedure of Benjamini and Liu (1999) for discrete data. We consider an application to pharmacovigilance spontaneous reporting systems, that serve for early detection of adverse reactions of marketed drugs.

stat.ME↗

arXiv2013

Discovering findings that replicate from a primary study of high dimension to a follow-up study

We consider the problem of identifying whether findings replicate from one study of high dimension to another, when the primary study guides the selection of hypotheses to be examined in the follow-up study as well as when there is no division of roles into the primary and the follow-up study. We show that existing meta-analysis methods are not appropriate for this problem, and suggest novel methods instead. We prove that our multiple testing procedures control for appropriate error-rates. The suggested FWER controlling procedure is valid for arbitrary dependence among the test statistics within each study. A more powerful procedure is suggested for FDR control. We prove that this procedure controls the FDR if the test statistics are independent within the primary study, and independent or have dependence of type PRDS in the follow-up study. For arbitrary dependence within the primary study, and either arbitrary dependence or dependence of type PRDS in the follow-up study, simple conservative modifications of the procedure control the FDR. We demonstrate the usefulness of these procedures via simulations and real data examples.

stat.ME↗

arXiv2012

Discussion of "Multiple Testing for Exploratory Research" by J. J. Goeman and A. Solari

Goeman and Solari [Statist. Sci. 26 (2011) 584-597, arXiv:1208.2841] have addressed the interesting topic of multiple testing for exploratory research, and provided us with nice suggestions for exploratory analysis. They defined properties that an inferential procedure should have for exploratory analysis: the procedure should be mild, flexible and post hoc. Their inferential procedure gives a lower bound on the number of false hypotheses among the selected hypotheses, and moreover whenever possible identifies elementary hypotheses that are false. The need to estimate a lower bound on the number of false hypotheses arises in various applications, and the partial conjunction approach was developed for this purpose in Biometrics 64 (2008) 1215-1222 (see also Philos. Trans. R. Soc. Lond. Ser. A 367 (2009) 4255-4271 for more details). For example, in a combined analysis of several studies that examine the same problem, it is of interest to give a lower bound on the number of studies in which the finding was reproduced. I will first address the relation between the method of Goeman and Solari and the partial conjunction approach. Then I will discuss possible extensions and address the issue of exploration in more general settings, where the local test may not be defined in advance or where the candidate hypotheses may not be known to begin with.

stat.ME↗

arXiv2012

A consistent multivariate test of association based on ranks of distances

We are concerned with the detection of associations between random vectors of any dimension. Few tests of independence exist that are consistent against all dependent alternatives. We propose a powerful test that is applicable in all dimensions and is consistent against all alternatives. The test has a simple form and is easy to implement. We demonstrate its good power properties in simulations and on examples.

stat.ME↗