Takato Hashino | Searcharxiv

arXiv2026

Asymptotic bias of the plug-in Shannon entropy estimator under a regularly varying occupancy model

Estimating the Shannon entropy of discrete distributions with countably infinite support is a challenging problem. In this paper, we investigate the bias of the plug-in estimator $\hat{H}_n$ for the Shannon entropy $H(\boldsymbol{p})$ under an occupancy model whose frequency sequence exhibits regular variation with tail index $\alpha\in(0,1)$. Using Poissonization and the theory of regular variation, we establish the asymptotic relation $|\mathsf{E}[\hat H_n] - H(\boldsymbol{p})| \sim n^{\alpha-1}L(n)C_\alpha$, where $L$ is a slowly varying function and $C_\alpha$ is an explicit constant depending only on $\alpha$ that admits an integral representation. Our result shows that the asymptotic behavior of the bias of the plug-in estimator under power-law frequency distributions is determined by the tail behavior of the underlying distribution.

math.PR↗

arXiv2026

Estimating the Shannon Entropy Using the Pitman--Yor Process

The Shannon entropy is a fundamental measure for quantifying diversity and model complexity in fields such as information theory, ecology, and genetics. However, many existing studies assume that the number of species is known, an assumption that is often unrealistic in practice. In recent years, efforts have been made to relax this restriction. Motivated by these developments, this study proposes an entropy estimation method based on the Pitman--Yor process, a representative approach in Bayesian nonparametrics. By approximating the true distribution as an infinite-dimensional process, the proposed method enables stable estimation even when the number of observed species is smaller than the true number of species. This approach provides a principled way to deal with the uncertainty in species diversity and enhances the reliability and robustness of entropy-based diversity assessment. In addition, we investigate the convergence property of the Shannon entropy for regularly varying distributions and use this result to establish the consistency of the proposed estimator. Finally, we demonstrate the effectiveness of the proposed method through numerical experiments.

stat.ME↗