Beyond Publication-Bias Detection: Estimating Bias under Uncertainty in Heterogeneous Literatures
Meta-analysts routinely test for publication bias, but a nonsignificant test is inconclusive because it may be a Type 2 error. In a large factorial simulation, I show that publication-bias tests have low power even with 1,000 studies once effect sizes are heterogeneous and the data do not meet the model's assumptions. I argue that the goal should shift from detecting bias to estimating it with confidence intervals. A confidence interval does not only bound the hypothesis that bias is absent; its upper bound also indicates how much bias remains compatible with the data. When the interval is wide, the upper bound does not exclude a large amount of bias, even if the formal test is nonsignificant. I then show that confidence intervals from step-function selection models are too narrow, covering the true amount of bias only about half the time. In contrast, z-curve's confidence interval obtained by transforming the expected discovery rate into a selection weight parameter has nominal coverage. Z-curve is therefore a valuable tool for examining publication bias in meta-analyses, especially when heterogeneity is high. I illustrate the implications of bias estimation with a meta-analysis of social priming and a set of applied-psychology meta-analyses.