Searcharxiv⌕ Search

arXiv subjects

Shigeyuki Matsui

Publications and source records attributed to Shigeyuki Matsui.

3 recordsLinked to original sources

Estimand-specific bootstrap confidence intervals after optimism correction for prediction models

Confidence intervals after bootstrap optimism correction are often reported for prediction models, but what they cover depends on the estimand. We distinguish two estimands: the target-population performance of the final model developed from the observed sample, $θ_{\mathrm{FM},n}$, and the average performance over repeated model development, $θ_{\mathrm{proc},n}$. Using first-order expansions of the estimation errors, we identify the bootstrap root that matches the error for $θ_{\mathrm{FM},n}$. We use this root in the nested error bootstrap (NEB) as a new reference method, and we show that the optimism error bootstrap (OEB), which needs no inner bootstrap, approximates NEB. We also show that the conventional two-stage bootstrap (TSB) and location-shifted bootstrap (LSB) reproduce the first-order variation for $θ_{\mathrm{proc},n}$. In simulations, NEB generally achieved coverage closer to the nominal level than TSB and LSB for the area under the receiver operating characteristic curve, Brier score, and log loss. OEB was close to NEB at moderate to high events per variable (EPV) but could give narrower intervals at low EPV. The distinction is particularly clear for calibration measures: first-order effects of perturbing the training and evaluation distributions cancel in the TSB root, yielding a degenerate first-order limit. In contrast, the NEB root has a nondegenerate first-order limit and yields asymptotically valid confidence intervals for $θ_{\mathrm{FM},n}$ under regularity conditions. Consistent with theory, TSB showed substantial undercoverage for calibration even at larger sample sizes. Our findings emphasize the need to construct confidence intervals for a specified estimand and performance measure.

stat.ME↗

Dose-finding design based on level set estimation in phase I cancer clinical trials

The primary objective of phase I cancer clinical trials is to evaluate the safety of a new experimental treatment and to find the maximum tolerated dose (MTD). We show that the MTD estimation problem can be regarded as a level set estimation (LSE) problem whose objective is to determine the regions where an unknown function value is above or below a given threshold. Then, we propose a novel dose-finding design in the framework of LSE. The proposed design determines the next dose on the basis of an acquisition function incorporating uncertainty in the posterior distribution of the dose-toxicity curve as well as overdose control. Simulation experiments show that the proposed LSE design achieves a higher accuracy in estimating the MTD and involves a lower risk of overdosing allocation compared to existing designs, thereby indicating that it provides an effective methodology for phase I cancer clinical trial design.

stat.ML↗

More Powerful Selective Kernel Tests for Feature Selection

Refining one's hypotheses in the light of data is a common scientific practice; however, the dependency on the data introduces selection bias and can lead to specious statistical analysis. An approach for addressing this is via conditioning on the selection procedure to account for how we have used the data to generate our hypotheses, and prevent information to be used again after selection. Many selective inference (a.k.a. post-selection inference) algorithms typically take this approach but will "over-condition" for sake of tractability. While this practice yields well calibrated statistic tests with controlled false positive rates (FPR), it can incur a major loss in power. In our work, we extend two recent proposals for selecting features using the Maximum Mean Discrepancy and Hilbert Schmidt Independence Criterion to condition on the minimal conditioning event. We show how recent advances in multiscale bootstrap makes conditioning on the minimal selection event possible and demonstrate our proposal over a range of synthetic and real world experiments. Our results show that our proposed test is indeed more powerful in most scenarios.

cs.LG↗