On estimating the effective sample size of phylogenetic trees in an autocorrelated chain
Estimating the effective sample size (ESS) is fundamental in Bayesian phylogenetic inference to account for autocorrelation in MCMC samples. While methods for continuous parameters are well established, the discrete and high-dimensional nature of treespace poses substantial challenges. In this paper, we introduce a new approach based on Conditional Clade Distributions (CCDs) to emulate MCMC chains with known ESS bounds. We compare existing tree ESS estimators with novel approaches that utilize CCDs as tractable tree distributions, as well as a new probabilistic estimator based on differences in clade frequencies between independent chains. Using our emulated chains from both simulated and real datasets, we assess estimators for their ability to recover the true ESS (accuracy), their insensitivity to the removal of autocorrelated samples from the chain (stability), and their tolerance to combining multiple chains together (robustness). We further examine how multimodality and poor inter-mode chain mixing can compromise the reliability of tree ESS estimates. These findings indicate that our new CCD-based estimators perform comparably to existing approaches with respect to accuracy, stability, and robustness. In contrast, the probabilistic estimator and two previously recommended methods incur prohibitive computational costs for long chains. Together, these results establish a new framework for evaluating tree ESS estimators using CCD-based emulated MCMC chains and demonstrate their performance and current limitations in practical phylogenetic analyses. The expanded suite of tree ESS estimators is available in the user-friendly TreeStat2 software package for BEAST 2.