The Harmonic Entropy Estimator: Minimax Optimality and Semiparametric Efficiency for Infinite Alphabets

Abstract

This paper considers the estimation of Shannon entropy for discrete distributions with countably infinite support. While minimax rates for finite-support distributions are established, infinite-support distributions present distinct challenges regarding bias control as probabilities vanish. We address this by introducing the \textit{harmonic entropy estimator}, a statistic derived from an exact algebraic identity relating the expectation of harmonic-transformed binomial counts to the logarithm of underlying success probabilities. We establish two main results characterizing the statistical limits of this problem. First, for the class of distributions with at least quadratically decaying tails ($p_j\lesssim j^{-2}$), we prove that the estimator achieves the parametric $L_2$-minimax convergence rate of order $1/n$. Second, under the stronger condition $p_j =o(j^{-2})$, we demonstrate that the estimator is semiparametrically efficient, converging to a normal distribution with variance matching the asymptotic efficiency bound $\textrm{Var}[\log p(X)]$. These results unify entropy estimation theory for finite-variance distributions, and provide a simple, one-step estimator with sharp theoretical guarantees.