arXiv subjects

David S. Robertson

Publications and source records attributed to David S. Robertson.

At least 19 recordsLinked to original sources

arXiv2026

Introducing precision-weighted bias as a performance measure to inform the inclusion of adaptive designs in meta-analysis

We propose a novel, intuitive measure of statistical performance: precision-weighted bias. Precision-weighted bias is defined as the unconditional bias of an estimator weighted by the degree of information (precision) it contains. Current guidelines, such as GRADE and CONSORT, often view the potential for increased bias in adaptive designs as a deterrent for the inclusion of such designs in systematic reviews. However, we demonstrate that the bias in a common-effect meta-analysis is approximately equal to the precision-weighted average of the precision-weighted biases of its constituent studies, rather than of their unweighted unconditional biases. Through simulation studies, we show that while adaptive designs may exhibit unweighted bias, they frequently have zero precision-weighted bias. Consequently, including these designs often results in a negligible change to the overall meta-analysis bias. These results suggest that precision-weighted bias is a superior indicator for determining whether to include an adaptive design in a meta-analysis. We recommend that precision-weighted bias be used as a standard complement to unweighted unconditional and conditional bias in simulation studies to support more inclusive and accurate evidence synthesis.

stat.ME↗

arXiv2026

Operationalizing Allocation Probability Tests: Practical Guidance on Optimized Implementation for Power and Robustness

Recently, a new testing approach for response-adaptive clinical trials was proposed based on the allocation probabilities (AP) rather than the outcome data. While original work on the AP test focused on binary and normal endpoints and demonstrated that significant efficiency gains are possible, many critical questions remain open regarding its practical implementation and upper limits. In this work, rather than simply proposing novel statistics, we seek to understand the maximum gain that can be obtained with the AP test by optimizing how these probabilities are used to define the test statistic. We expand the method's practical utility by applying it to survival endpoints (exponential distributions) and introducing a rigorous strategy for selecting the null hypothesis to properly calibrate type I error. Our simulation studies reveal that by optimizing the functional form of the AP test, investigators can achieve a substantial increase in power, approaching the theoretical maximum, without sacrificing the patient outcome goals of the design. Furthermore, we explicitly compare the method to a standard Bayesian decision rule, finding that the optimized AP test significantly outperforms traditional frequentist tests while maintaining strict error control. This work provides a missing practical framework for implementing robust and optimized AP tests in complex response-adaptive settings.

stat.ME↗

arXiv2025

Thompson, Ulam, or Gauss? Multi-criteria recommendations for posterior probability computation methods in Bayesian response-adaptive trials

Bayesian adaptive designs enable flexible clinical trials by adapting features based on accumulating data. Among these, Bayesian Response-Adaptive Randomization (BRAR) skews patient allocation towards more promising treatments based on interim data. Implementing BRAR requires the relatively quick evaluation of posterior probabilities. However, the limitations of existing closed-form solutions mean trials often rely on computationally intensive approximations which can impact accuracy and the scope of scenarios explored. While faster Gaussian approximations exist, their reliability is not guaranteed. Critically, the approximation method used is often poorly reported, and the literature lacks practical guidance for selecting and comparing these methods, particularly regarding the trade-offs between computational speed, inferential accuracy, and their implications for patient benefit. In this paper, we focus on BRAR trials with binary endpoints, developing a novel algorithm that efficiently and exactly computes these posterior probabilities, enabling a robust assessment of existing approximation methods in use. Leveraging these exact computations, we establish a comprehensive benchmark for evaluating approximation methods based on their computational speed, patient benefit, and inferential accuracy. Our comprehensive analysis, conducted through a range of simulations in the two-armed case and a re-analysis of the three-armed Established Status Epilepticus Treatment Trial, reveals that the exact calculation algorithm is often the fastest, even for up to 12 treatment arms. Furthermore, we demonstrate that commonly used approximation methods can lead to significant power loss and Type I error rate inflation. We conclude by providing practical guidance to aid practitioners in selecting the most appropriate computation method for various clinical trial settings.

stat.ME↗

arXiv2025

Informed Burn-In Decisions in RAR: Harmonizing Adaptivity and Inferential Precision Based on Study Setting

Response-Adaptive Randomization (RAR) is recognized for its potential to deliver improvements in patient benefit. However, the utility of RAR is contingent on regularization methods to mitigate early instability and preserve statistical integrity. A standard regularization approach is the ''burn-in'' period, an initial phase of equal randomization before treatment allocation adapts based on accrued data. The length of this burn-in is a critical design parameter, yet its selection remains unsystematic and improvised, as no established guideline exists. A poorly chosen length poses significant risks: one that is too short leads to high estimation bias and type-I error rate inflation, while one that is too long impedes the intended patient and power benefits of using adaptation. The challenge of selecting the burn-in generalizes to a fundamental question: what is the statistically appropriate timing for the first adaptation? We introduce the first systematic framework for determining burn-in length. This framework synthesizes core factors - total sample size, problem difficulty, and two novel metrics (reactivity and expected final allocation error) - into a single, principled formula. Simulation studies, grounded in real-world designs, demonstrate that lengths derived from our formula successfully stabilize the trial. The formula identifies a ''sweet spot'' that mitigates type-I error rate inflation and mean-squared error, preserving the advantages of higher power and patient benefit. This framework moves researchers from conjecture toward a systematic, reliable approach.

stat.ME↗

arXiv2025

Bayesian Optimal Phase II design with optimised stopping boundaries and response-adaptive randomisation

The Bayesian Optimal Phase II (BOP2) framework is a flexible trial design that can naturally facilitate complex adaptations due to its Bayesian setting. BOP2 uses equal randomisation and equally placed interim analyses in its design, but it is unclear whether these give the best operating characteristics. By incorporating Bayesian Response-Adaptive Randomisation (BRAR) and optimal interim analysis placement, we show that allocation to the best treatment and expected sample size can be improved with minimal impact on power. We discuss recommendations on implementing these adaptations, using simulation-based evidence, to give practical advice to practitioners. Reproducible code for the simulations is freely provided.

stat.AP↗

arXiv2025

Optimal weighted tests for replication studies and the two-trials rule

Replication studies for scientific research are an important part of ensuring the reliability and integrity of experimental findings. In the context of clinical trials, the concept of replication has been formalised by the 'two-trials' rule, where two pivotal studies are required to show positive results before a drug can be approved. In experiments testing multiple hypotheses simultaneously, control of the overall familywise error rate (FWER) is additionally required in many contexts. The well-known Bonferroni procedure controls the FWER, and a natural extension is to introduce weights into this procedure to reflect the a-priori importance of hypotheses or to maximise some measure of the overall power of the experiment. In this paper, we consider analysing a replication study using an optimal weighted Bonferroni procedure, with the weights based on the results of the original study that is being replicated and the optimality criterion being to maximise the disjunctive power of the trial (the power to reject at least one non-null hypothesis). We show that using the proposed procedure can lead to a substantial increase in the disjunctive power of the replication study, and is robust to changes in the effect sizes between the two studies.

stat.ME↗

arXiv2025

Is 1:1 Always Most Powerful? Why Careful Determination of Allocation Ratios Matters in Trial Design

The principle of allocating an equal number of patients to each arm in a randomized controlled trial remains widely believed to be optimal for maximising statistical power. However, this long-held belief only holds true if the treatment groups have equal outcome variances, a condition that is often not met or, is simply not assessed in practice. This paper reasserts the fact that a departure from a 1:1 ratio can maintain or improve statistical power while increasing the benefits to participants. The benefit is particularly self-evident for binary and time-to-event endpoints, where variances are determined by the assumed success or event rates. To illustrate this, we present two case studies: a small-scale metastatic melanoma trial with a binary endpoint and a larger trial evaluating virtual reality for pain reduction with a continuous endpoint. Our simulations compare equal randomisation, preplanned fixed unequal randomisation, and response-adaptive randomisation targeting Neyman allocation. Results show that unequal allocation can increase the proportion of patients receiving the superior treatment without reducing power, with modest power gains observed in both binary and continuous settings, highlighting the practical relevance of optimised allocation strategies across trial types and sizes.

stat.ME↗

arXiv2025

A comparison of approaches to incorporate patient-selected and patient-ranked outcomes in clinical trials

A key aspect of patient-focused drug development is identifying and measuring outcomes that are important to patients in clinical trials. Many medical conditions affect multiple symptom domains, and a consensus approach to determine the relative importance of the associated multiple outcomes ignores the heterogeneity in individual patient preferences. Patient-selected outcomes offer one way to incorporate individual patient preferences, as proposed in recent regulatory guidance for the treatment for migraine, where each patient selects their most bothersome migraine-associated symptom in addition to pain. Patient-ranked outcomes have also recently been proposed, which go further and consider the full ranking of the relative importance of all the outcomes. This can be assessed using a composite DOOR (Desirability of Outcome Ranking) endpoint. In this paper, we compare the advantages and disadvantages of using patient-selected versus patient-ranked outcomes in the context of a two-arm randomised controlled trial for multiple sclerosis. We compare the power and type I error rate by simulation, and discuss several other important considerations when using the two approaches.

stat.ME↗

arXiv2025

A burn-in(g) question: How long should an initial equal randomization stage be before Bayesian response-adaptive randomization?

Response-adaptive randomization (RAR) can increase participant benefit in clinical trials, but also complicates statistical analysis. The burn-in period (a non-adaptive initial stage) is commonly used to mitigate this disadvantage, yet guidance on its optimal duration is scarce. To address this critical gap, this paper introduces an exact evaluation approach to investigate how the burn-in length impacts statistical operating characteristics of two-arm binary Bayesian RAR (BRAR) designs. We show that (1) commonly used calibration and asymptotic tests show substantial type I error rate inflation for BRAR designs without a burn-in period, and increasing the total burn-in length to more than half the trial size reduces but does not fully mitigate type I error rate inflation, necessitating exact tests; (2) exact tests conditioning on total successes show the highest average and minimum power up to large burn-in lengths; (3) the burn-in length substantially influences power and participant benefit, which are often not maximized at the maximum or minimum possible burn-in length; (4) the test statistic influences the type I error rate and power; (5) estimation bias decreases quicker in the burn-in length for larger treatment effects and increases for larger trial sizes under the same burn-in length. Our approach is illustrated by re-designing the ARREST trial.

stat.ME↗

arXiv2024

Confidence intervals for adaptive trial designs I: A methodological review

Regulatory guidance notes the need for caution in the interpretation of confidence intervals (CIs) constructed during and after an adaptive clinical trial. Conventional CIs of the treatment effects are prone to undercoverage (as well as other undesirable properties) in many adaptive designs, because they do not take into account the potential and realised trial adaptations. This paper is the first in a two-part series that explores CIs for adaptive trials. It provides a comprehensive review of the methods to construct CIs for adaptive designs, while the second paper illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. We describe several classes of techniques for constructing CIs for adaptive clinical trials, before providing a systematic literature review of available methods, classified by the type of adaptive design. As part of this, we assess, through a proposed traffic light system, which of several desirable features of CIs (such as achieving nominal coverage and consistency with the hypothesis test decision) each of these methods holds.

stat.ME↗

arXiv2024

Confidence intervals for adaptive trial designs II: Case study and practical guidance

In adaptive clinical trials, the conventional confidence interval (CI) for a treatment effect is prone to undesirable properties such as undercoverage and potential inconsistency with the final hypothesis testing decision. Accordingly, as is stated in recent regulatory guidance on adaptive designs, there is the need for caution in the interpretation of CIs constructed during and after an adaptive clinical trial. However, it may be unclear which of the available CIs in the literature are preferable. This paper is the second in a two-part series that explores CIs for adaptive trials. Part I provided a methodological review of approaches to construct CIs for adaptive designs. In this paper (part II), we present an extended case study based around a two-stage group sequential trial, including a comprehensive simulation study of the proposed CIs for this setting. This facilitates an expanded description of considerations around what makes for an effective CI procedure following an adaptive trial. We show that the CIs can have notably different properties. Finally, we propose a set of guidelines for researchers around the choice of CIs and the reporting of CIs following an adaptive design.

stat.ME↗

arXiv2023

The use of restricted mean survival time to estimate treatment effect under model misspecification, a simulation study

The use of the non-parametric Restricted Mean Survival Time endpoint (RMST) has grown in popularity as trialists look to analyse time-to-event outcomes without the restrictions of the proportional hazards assumption. In this paper, we evaluate the power and type I error rate of the parametric and non-parametric RMST estimators when treatment effect is explained by multiple covariates, including an interaction term. Utilising the RMST estimator in this way allows the combined treatment effect to be summarised as a one-dimensional estimator, which is evaluated using a one-sided hypothesis Z-test. The estimators are either fully specified or misspecified, both in terms of unaccounted covariates or misspecified knot points (where trials exhibit crossing survival curves). A placebo-controlled trial of Gamma interferon is used as a motivating example to simulate associated survival times. When correctly specified, the parametric RMST estimator has the greatest power, regardless of the time of analysis. The misspecified RMST estimator generally performs similarly when covariates mirror those of the fitted case study dataset. However, as the magnitude of the unaccounted covariate increases, the associated power of the estimator decreases. In all cases, the non-parametric RMST estimator has the lowest power, and power remains very reliant on the time of analysis (with a later analysis time correlated with greater power).

stat.ME↗

arXiv2023

Online multiple hypothesis testing

Modern data analysis frequently involves large-scale hypothesis testing, which naturally gives rise to the problem of maintaining control of a suitable type I error rate, such as the false discovery rate (FDR). In many biomedical and technological applications, an additional complexity is that hypotheses are tested in an online manner, one-by-one over time. However, traditional procedures that control the FDR, such as the Benjamini-Hochberg procedure, assume that all p-values are available to be tested at a single time point. To address these challenges, a new field of methodology has developed over the past 15 years showing how to control error rates for online multiple hypothesis testing. In this framework, hypotheses arrive in a stream, and at each time point the analyst decides whether to reject the current hypothesis based both on the evidence against it, and on the previous rejection decisions. In this paper, we present a comprehensive exposition of the literature on online error rate control, with a review of key theory as well as a focus on applied examples. We also provide simulation results comparing different online testing algorithms and an up-to-date overview of the many methodological extensions that have been proposed.

stat.ME↗

arXiv2022

Point estimation for adaptive trial designs I: a methodological review

Recent FDA guidance on adaptive clinical trial designs defines bias as "a systematic tendency for the estimate of treatment effect to deviate from its true value", and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realised trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This paper is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this paper can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

stat.ME↗

arXiv2022

Point estimation for adaptive trial designs II: practical considerations and guidance

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This paper is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realisation they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy pre-specified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

stat.ME↗

arXiv2022

Some performance considerations when using multi-armed bandit algorithms in the presence of missing data

When comparing the performance of multi-armed bandit algorithms, the potential impact of missing data is often overlooked. In practice, it also affects their implementation where the simplest approach to overcome this is to continue to sample according to the original bandit algorithm, ignoring missing outcomes. We investigate the impact on performance of this approach to deal with missing data for several bandit algorithms through an extensive simulation study assuming the rewards are missing at random. We focus on two-armed bandit algorithms with binary outcomes in the context of patient allocation for clinical trials with relatively small sample sizes. However, our results apply to other applications of bandit algorithms where missing data is expected to occur. We assess the resulting operating characteristics, including the expected reward. Different probabilities of missingness in both arms are considered. The key finding of our work is that when using the simplest strategy of ignoring missing data, the impact on the expected performance of multi-armed bandit strategies varies according to the way these strategies balance the exploration-exploitation trade-off. Algorithms that are geared towards exploration continue to assign samples to the arm with more missing responses (which being perceived as the arm with less observed information is deemed more appealing by the algorithm than it would otherwise be). In contrast, algorithms that are geared towards exploitation would rapidly assign a high value to samples from the arms with a current high mean irrespective of the level observations per arm. Furthermore, for algorithms focusing more on exploration, we illustrate that the problem of missing responses can be alleviated using a simple mean imputation approach.

stat.ML↗

arXiv2022

Response-adaptive randomization in clinical trials: from myths to practical considerations

Response-Adaptive Randomization (RAR) is part of a wider class of data-dependent sampling algorithms, for which clinical trials are typically used as a motivating application. In that context, patient allocation to treatments is determined by randomization probabilities that change based on the accrued response data in order to achieve experimental goals. RAR has received abundant theoretical attention from the biostatistical literature since the 1930's and has been the subject of numerous debates. In the last decade, it has received renewed consideration from the applied and methodological communities, driven by well-known practical examples and its widespread use in machine learning. Papers on the subject present different views on its usefulness, and these are not easy to reconcile. This work aims to address this gap by providing a unified, broad and fresh review of methodological and practical issues to consider when debating the use of RAR in clinical trials.

stat.ME↗

arXiv2022

Online FDR Control for RNAseq Data

Motivation: While the analysis of a single RNA sequencing (RNAseq) dataset has been well described in the literature, modern research workflows often have additional complexity in that related RNAseq experiments are performed sequentially over time. The simplest and most widely used analysis strategy ignores the temporal aspects and analyses each dataset separately. However, this can lead to substantial inflation of the overall false discovery rate (FDR). We propose applying recently developed methodology for online hypothesis testing to analyse sequential RNAseq experiments in a principled way, guaranteeing FDR control at all times while never changing past decisions. Results: We show that standard offline approaches have variable control of FDR of related RNAseq experiments over time and a naively composed approach may improperly change historic decisions. We demonstrate that the online FDR algorithms are a principled way to guarantee control of FDR. Furthermore, in certain simulation scenarios, we observe empirically that online approaches have comparable power to offline approaches. Availability and Implementation: The onlineFDR package is freely available at http: //www.bioconductor.org/packages/onlineFDR. Additional code used for the simulation studies can be found at https://github.com/latlio/onlinefdr_rnaseq_simulation

q-bio.QM↗