Exoplaneteers Keep Overestimating Sigma Significances

Abstract

Astronomers, and in particular exoplaneteers, have a curious habit of expressing Bayes factors as frequentist sigma values. This is of course completely unnecessary and arguably rather ill-advised. Regardless, the practice is common - especially in the detection claims of chemical species within exoplanet atmospheres. The current canonical conversion strategy stems from a statistics paper from Sellke et al. (2001), who derived an upper bound on the Bayes factor between the test and null hypotheses, as a function of the $p$-value (or number of sigmas, $n_{\sigma}$). A common practice within the exoplanet atmosphere community is to numerically invert this formula, going from a Bayes factor to $n_\sigma$. This goes back to Benneke & Seager (2013) -- a highly cited paper that introduced Bayesian model comparison as a means of inferring the presence of specific chemical species -- in an attempt to calibrate the Bayes factors from their technique for a community that in 2013 was more familiar with frequentist sigma significances. However, as originally noted by Sellke et al. (2001), the conversion only provides an upper limit on $n_\sigma$, with the true value generally being lower. This can result in inflations of claimed detection significances, and this note strongly urges the community to stop converting to $n_\sigma$ at all and simply stick with Bayes factors.