arXiv subjects

Leonhard Held

Publications and source records attributed to Leonhard Held.

At least 37 records · Page 2Linked to original sources

arXiv2023

Power priors for replication studies

The ongoing replication crisis in science has increased interest in the methodology of replication studies. We propose a novel Bayesian analysis approach using power priors: The likelihood of the original study's data is raised to the power of $α$, and then used as the prior distribution in the analysis of the replication data. Posterior distribution and Bayes factor hypothesis tests related to the power parameter $α$ quantify the degree of compatibility between the original and replication study. Inferences for other parameters, such as effect sizes, dynamically borrow information from the original study. The degree of borrowing depends on the conflict between the two studies. The practical value of the approach is illustrated on data from three replication studies, and the connection to hierarchical modeling approaches explored. We generalize the known connection between normal power priors and normal hierarchical models for fixed parameters and show that normal power prior inferences with a beta prior on the power parameter $α$ align with normal hierarchical model inferences using a generalized beta prior on the relative heterogeneity variance $I^2$. The connection illustrates that power prior modeling is unnatural from the perspective of hierarchical modeling since it corresponds to specifying priors on a relative rather than an absolute heterogeneity scale.

stat.ME↗

arXiv2023

Bayesian Approaches to Designing Replication Studies

Replication studies are essential for assessing the credibility of claims from original studies. A critical aspect of designing replication studies is determining their sample size; a too small sample size may lead to inconclusive studies whereas a too large sample size may waste resources that could be allocated better in other studies. Here, we show how Bayesian approaches can be used for tackling this problem. The Bayesian framework allows researchers to combine the original data and external knowledge in a design prior distribution for the underlying parameters. Based on a design prior, predictions about the replication data can be made, and the replication sample size can be chosen to ensure a sufficiently high probability of replication success. Replication success may be defined by Bayesian or non-Bayesian criteria, and different criteria may also be combined to meet distinct stakeholders and enable conclusive inferences based on multiple analysis approaches. We investigate sample size determination in the normal-normal hierarchical model where analytical results are available and traditional sample size determination is a special case where the uncertainty on parameter values is not accounted for. We use data from a multisite replication project of social-behavioral experiments to illustrate how Bayesian approaches can help design informative and cost-effective replication studies. Our methods can be used through the R package BayesRepDesign.

stat.ME↗

arXiv2023

Normalized power priors always discount historical data

Power priors are used for incorporating historical data in Bayesian analyses by taking the likelihood of the historical data raised to the power $α$ as the prior distribution for the model parameters. The power parameter $α$ is typically unknown and assigned a prior distribution, most commonly a beta distribution. Here, we give a novel theoretical result on the resulting marginal posterior distribution of $α$ in case of the the normal and binomial model. Counterintuitively, when the current data perfectly mirror the historical data and the sample sizes from both data sets become arbitrarily large, the marginal posterior of $α$ does not converge to a point mass at $α= 1$ but approaches a distribution that hardly differs from the prior. The result implies that a complete pooling of historical and current data is impossible if a power prior with beta prior for $α$ is used.

stat.ME↗

arXiv2023

Assessing replicability with the sceptical p-value: Type-I error control and sample size planning

We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical $p$-value. A recalibration is proposed to obtain exact overall Type-I error control if the effect is null in both studies and additional bounds on the partial and conditional Type-I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two-trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two-trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

stat.ME↗

arXiv2023

Simulating and reporting frequentist operating characteristics of clinical trials that borrow external information

Borrowing of information from historical or external data to inform inference in a current trial is an expanding field in the era of precision medicine, where trials are often performed in small patient cohorts for practical or ethical reasons. Many approaches for borrowing from external data have been proposed. Even though these methods are mainly based on Bayesian approaches by incorporating external information into the prior for the current analysis, frequentist operating characteristics of the analysis strategy are of interest. In particular, type I error and power at a prespecified point alternative are in the focus. It is well-known that borrowing from external information may lead to the alteration of type I error rate. We propose a procedure to investigate and report the frequentist operating characteristics in this context. The approach evaluates type I error rate of the test with borrowing from external data and calibrates the test without borrowing to this type I error rate. On this basis, a fair comparison of power between the test with and without borrowing is achieved.

stat.ME↗

arXiv2022

Combining Evidence from Clinical Trials in Conditional or Accelerated Approval

Conditional (European Medicines Agency) or accelerated (U.S. Food and Drug Administration) approval of drugs allow earlier access to promising new treatments that address unmet medical needs. Certain post-marketing requirements must typically be met in order to obtain full approval, such as conducting a new post-market clinical trial. We study the applicability of the recently developed harmonic mean Chi-squared test to this conditional or accelerated approval framework. The proposed approach can be used both to support the design of the post-market trial and the analysis of the combined evidence provided by both trials. Other methods considered are the two-trials rule, Fisher's criterion and Stouffer's method. In contrast to some of the traditional methods, the harmonic mean Chi-squared test always requires a post-market clinical trial. If the p-value from the pre-market clinical trial is << 0.025, a smaller sample size for the post-market clinical trial is needed than with the two-trials rule. For illustration, we apply the harmonic mean Chi-squared test to a drug which received conditional (and later full) market licensing by the EMA. A simulation study is conducted to study the operating characteristics of the harmonic mean Chi-squared test and two-trials rule in more detail. We finally investigate the applicability of these two methods to compute the power at interim of an ongoing post-market trial. These results are expected to aid in the design and assessment of the required post-market studies in terms of the level of evidence required for full approval.

stat.ME↗

arXiv2022

Comparing Confidence Intervals for a Binomial Proportion with the Interval Score

There are over 55 different ways to construct a confidence respectively credible interval (CI) for the binomial proportion. Methods to compare them are necessary to decide which should be used in practice. The interval score has been suggested to compare prediction intervals. This score is a proper scoring rule that combines the coverage as a measure of calibration and the width as a measure of sharpness. We evaluate eleven CIs for the binomial proportion based on the expected interval score and propose a summary measure which can take into account different weighting of the underlying true proportion. Under uniform weighting, the expected interval score recommends the Wilson CI or Bayesian credible intervals with a uniform prior. If extremely low or high proportions receive more weight, the score recommends Bayesian credible intervals based on Jeffreys' prior. While more work is needed to theoretically justify the use of the interval score for the comparison of CIs, our results suggest that it constitutes a useful method to combine coverage and width in one measure. This novel approach could also be used in other applications.

stat.ME↗

arXiv2021

The sceptical Bayes factor for the assessment of replication success

Replication studies are increasingly conducted but there is no established statistical criterion for replication success. We propose a novel approach combining reverse-Bayes analysis with Bayesian hypothesis testing: a sceptical prior is determined for the effect size such that the original finding is no longer convincing in terms of a Bayes factor. This prior is then contrasted to an advocacy prior (the reference posterior of the effect size based on the original study), and replication success is declared if the replication data favour the advocacy over the sceptical prior at a higher level than the original data favoured the sceptical prior over the null hypothesis. The sceptical Bayes factor is the highest level where replication success can be declared. A comparison to existing methods reveals that the sceptical Bayes factor combines several notions of replicability: it ensures that both studies show sufficient evidence against the null and penalises incompatibility of their effect estimates. Analysis of asymptotic properties and error rates, as well as case studies from the Social Sciences Replication Project show the advantages of the method for the assessment of replicability.

stat.ME↗

arXiv2021

Reverse-Bayes methods for evidence assessment and research synthesis

It is now widely accepted that the standard inferential toolkit used by the scientific research community -- null-hypothesis significance testing (NHST) -- is not fit for purpose. Yet despite the threat posed to the scientific enterprise, there is no agreement concerning alternative approaches for evidence assessment. This lack of consensus reflects long-standing issues concerning Bayesian methods, the principal alternative to NHST. We report on recent work that builds on an approach to inference put forward over 70 years ago to address the well-known "Problem of Priors" in Bayesian analysis, by reversing the conventional prior-likelihood-posterior ("forward") use of Bayes's Theorem. Such Reverse-Bayes analysis allows priors to be deduced from the likelihood by requiring that the posterior achieve a specified level of credibility. We summarise the technical underpinning of this approach, and show how it opens up new approaches to common inferential challenges, such as assessing the credibility of scientific findings, setting them in appropriate context, estimating the probability of successful replications, and extracting more insight from NHST while reducing the risk of misinterpretation. We argue that Reverse-Bayes methods have a key role to play in making Bayesian methods more accessible and attractive for evidence assessment and research synthesis. As a running example we consider a recently published meta-analysis from several randomized controlled clinical trials investigating the association between corticosteroids and mortality in hospitalized patients with COVID-19.

stat.ME↗

arXiv2021

Power Calculations for Replication Studies

The reproducibility crisis has led to an increasing number of replication studies being conducted. Sample sizes for replication studies are often calculated using conditional power based on the effect estimate from the original study. However, this approach is not well suited as it ignores the uncertainty of the original result. Bayesian methods are used in clinical trials to incorporate prior information into power calculations. We propose to adapt this methodology to the replication framework and promote the use of predictive instead of conditional power in the design of replication studies. Moreover, we describe how extensions of the methodology to sequential clinical trials can be tailored to replication studies. Conditional and predictive power calculated at an interim analysis are compared and we argue that predictive power is a useful tool to decide whether to stop a replication study prematurely. A recent project on the replicability of social sciences is used to illustrate the properties of the different methods.

stat.ME↗

arXiv2021

The assessment of replication success based on relative effect size

Replication studies are increasingly conducted in order to confirm original findings. However, there is no established standard how to assess replication success and in practice many different approaches are used. The purpose of this paper is to refine and extend a recently proposed reverse-Bayes approach for the analysis of replication studies. We show how this method is directly related to the relative effect size, the ratio of the replication to the original effect estimate. This perspective leads to a new proposal to recalibrate the assessment of replication success, the golden level. The recalibration ensures that for borderline significant original studies replication success can only be achieved if the replication effect estimate is larger than the original one. Conditional power for replication success can then take any desired value if the original study is significant and the replication sample size is large enough. Compared to the standard approach to require statistical significance of both the original and replication study, replication success at the golden level offers uniform gains in project power and controls the Type-I error rate if the replication sample size is not smaller than the original one. An application to data from four large replication projects shows that the new approach leads to more appropriate inferences, as it penalizes shrinkage of the replication estimate compared to the original one, while ensuring that both effect estimates are sufficiently convincing on their own.

stat.ME↗

arXiv2020

A marginal moment matching approach for fitting endemic-epidemic models to underreported disease surveillance counts

Count data are often subject to underreporting, especially in infectious disease surveillance. We propose an approximate maximum likelihood method to fit count time series models from the endemic-epidemic class to underreported data. The approach is based on marginal moment matching where underreported processes are approximated through completely observed processes from the same class. Moreover, the form of the bias when underreporting is ignored or taken into account via multiplication factors is analysed. Notably, we show that this leads to a downward bias in model-based estimates of the effective reproductive number. A marginal moment matching approach can also be used to account for reporting intervals which are longer than the mean serial interval of a disease. The good performance of the proposed methodology is demonstrated in simulation studies. An extension to time-varying parameters and reporting probabilities is discussed and applied in a case study on weekly rotavirus gastroenteritis counts in Berlin, Germany.

stat.ME↗

arXiv2020

Endemic-epidemic models with discrete-time serial interval distributions for infectious disease prediction

Multivariate count time series models are an important tool for the analysis and prediction of infectious disease spread. We consider the endemic-epidemic framework, an autoregressive model class for infectious disease surveillance counts, and replace the default autoregression on counts from the previous time period with more flexible weighting schemes inspired by discrete-time serial interval distributions. We employ three different parametric formulations, each with an additional unknown weighting parameter estimated via a profile likelihood approach, and compare them to an unrestricted nonparametric approach. The new methods are illustrated in a univariate analysis of dengue fever incidence in San Juan, Puerto Rico, and a spatio-temporal study of viral gastroenteritis in the twelve districts of Berlin. We assess the predictive performance of the suggested models and several reference models at various forecast horizons. In both applications, the performance of the endemic-epidemic models is considerably improved by the proposed weighting schemes.

stat.AP↗

arXiv2020

The harmonic mean $χ^2$ test to substantiate scientific findings

Statistical methodology plays a crucial role in drug regulation. Decisions by the FDA or EMA are typically made based on multiple primary studies testing the same medical product, where the two-trials rule is the standard requirement, despite a number of shortcomings. A new approach is proposed for this task based on the (weighted) harmonic mean of the squared study-specific test statistics. Appropriate scaling ensures that, for any number of independent studies, the null distribution is a $χ^2$-distribution with one degree of freedom. This gives rise to a new method for combining one-sided $p$-values and calculating confidence intervals for the overall treatment effect. Further properties are discussed and a comparison with the two-trials rule is made, as well as with alternative research synthesis methods. An attractive feature of the new approach is that a claim of success requires each study to be convincing on its own to a certain degree depending on the overall significance level and the number of studies. A real example with 5 clinical trials investigating the effect of Carvedilol for the treatment of patients with moderate to severe heart failure patients is used to illustrate the methodology.

stat.ME↗

arXiv2019

A New Standard for the Analysis and Design of Replication Studies

A new standard is proposed for the evidential assessment of replication studies. The approach combines a specific reverse-Bayes technique with prior-predictive tail probabilities to define replication success. The method gives rise to a quantitative measure for replication success, called the sceptical p-value. The sceptical p-value integrates traditional significance of both the original and replication study with a comparison of the respective effect sizes. It incorporates the uncertainty of both the original and replication effect estimates and reduces to the ordinary p-value of the replication study if the uncertainty of the original effect estimate is ignored. The proposed framework can also be used to determine the power or the required replication sample size to achieve replication success. Numerical calculations highlight the difficulty to achieve replication success if the evidence from the original study is only suggestive. An application to data from the Open Science Collaboration project on the replicability of psychological science illustrates the proposed methodology.

stat.ME↗

arXiv2019

Efficient real-time monitoring of an emerging influenza epidemic: how feasible?

A prompt public health response to a new epidemic relies on the ability to monitor and predict its evolution in real time as data accumulate. The 2009 A/H1N1 outbreak in the UK revealed pandemic data as noisy, contaminated, potentially biased, and originating from multiple sources. This seriously challenges the capacity for real-time monitoring. Here we assess the feasibility of real-time inference based on such data by constructing an analytic tool combining an age-stratified SEIR transmission model with various observation models describing the data generation mechanisms. As batches of data become available, a sequential Monte Carlo (SMC) algorithm is developed to synthesise multiple imperfect data streams, iterate epidemic inferences and assess model adequacy amidst a rapidly evolving epidemic environment, substantially reducing computation time in comparison to standard MCMC, to ensure timely delivery of real-time epidemic assessments. In application to simulated data designed to mimic the 2009 A/H1N1 epidemic, SMC is shown to have additional benefits in terms of assessing predictive performance and coping with parameter non-identifiability.

stat.CO↗

arXiv2018

The Assessment of Intrinsic Credibility and a New Argument for p<0.005

The concept of intrinsic credibility has been recently introduced to check the credibility of "out of the blue" findings without any prior support. A significant result is deemed intrinsically credible if it is in conflict with a sceptical prior derived from the very same data that would make the effect non-significant. In this paper I propose to use Bayesian prior-predictive tail probabilities to assess intrinsic credibility. For the standard 5% significance level, this leads to a new p-value threshold that is remarkably close to the recently proposed p<0.005 standard. I also introduce the credibility ratio, the ratio of the upper to the lower limit of a standard confidence interval for the corresponding effect size. I show that the credibility ratio has to be smaller than 5.8 such that a significant finding is also intrinsically credible. Finally, a p-value for intrinsic credibility is proposed that is a simple function of the ordinary p-value and has a direct frequentist interpretation in terms of the probability of replicating an effect.

stat.ME↗

arXiv2018

Forecasting Based on Surveillance Data

Forecasting the future course of epidemics has always been one of the main goals of epidemic modelling. This chapter reviews statistical methods to quantify the accuracy of epidemic forecasts. We distinguish point and probabilistic forecasts and describe different methods to evaluate and compare the predictive performance across models. Two case studies demonstrate how to apply the different techniques to uni- and multivariate forecasts. We focus on forecasting count time series from routine public health surveillance: weekly counts of influenza-like illness in Switzerland, and age-stratified counts of norovirus gastroenteritis in Berlin, Germany. Data and code for all analyses are available in a supplementary R package.

stat.ME↗