Post-hoc $\alpha$ Hypothesis Testing and the Post-hoc $p$-value

Abstract

In traditional hypothesis testing one must pre-specify the significance level $\alpha$ to bound the `size' of the test: its probability to falsely reject the hypothesis. Indeed, a data-dependent selection of $\alpha$ would generally distort the size, possibly making it larger than the specified level $\alpha$. We explore hypothesis testing with a data-dependent choice of $\alpha$ by guaranteeing that there is no such size distortion in expectation, even if the level $\alpha$ is arbitrarily selected based on the data. Unlike regular $p$-values, resulting `post-hoc $p$-values' allow us to `reject at level $p$' and still provide this guarantee. Interestingly, we find that $p$ is a post-hoc $p$-value if and only if $1/p$ is an $e$-value, a recently introduced measure of evidence. While often treated as different paradigms, this reveals $e$-values are simply $p$-values under a stronger error guarantee, thinly veiled by the reciprocal $p = 1/e$. Moreover, we extend classical optimal testing to optimal post-hoc testing. Finally, we apply our work to close Markov's inequality into a post-hoc $\alpha$ equality, and we study more general forms of post-hoc testing that require us to generalize beyond $e$-values.