On estimating the effective sample size of phylogenetic trees in an autocorrelated chain
Estimating the effective sample size (ESS) is fundamental in Bayesian phylogenetic inference to properly account for autocorrelation in MCMC samples. While methods for continuous parameters are well established, the discrete and high-dimensional nature of treespace poses substantial challenges. Here, we compare existing tree ESS estimators with novel approaches that leverage tractable tree distributions, specifically Conditional Clade Distributions (CCDs), as well as a new probabilistic estimator based on clade frequency differences between independent chains. Using simulated chains with known ESS bounds, we assess estimator accuracy and evaluate their stability and robustness on simulated and real datasets. We further examine how multimodality in posterior distributions and poor mixing can substantially affect ESS estimates, highlighting the need for careful interpretation. Our CCD-based estimators perform comparably to existing approaches, with two methods showing lower variance by averaging across multiple estimates. In contrast, the probabilistic estimator and two previously recommended methods incur prohibitive computational costs for long chains. Together, these results provide guidance for reliable and efficient tree ESS estimation in complex phylogenetic analyses.