Finite-Resolution Information from Collision Statistics

Abstract

Collision statistics provide a finite-resolution view of information by measuring how often a fixed number of independent samples fall on the same state. These directly countable quantities form the basis of integer-order R\'enyi entropies. Here, we use low-order R\'enyi entropies to approximate Shannon entropy and mutual information, while characterizing what is necessarily lost when only finitely many collision moments are used. We derive interpolation-error bounds showing that approximation error is controlled by the shape of the R\'enyi entropy path near the Shannon point. We also separate this deterministic error from finite-sample estimation error: for fixed collision order, increasing sample size improves estimation of the finite-resolution target but does not eliminate its deterministic difference from Shannon entropy or mutual information. Finally, we show that finite collision moments do not generally identify Shannon entropy, and that increasing collision order shifts sensitivity toward high-probability events. Numerical experiments illustrate the approximation--estimation tradeoff and compare collision-based approximations with plug-in and Miller--Madow estimators. The framework links collision counts, R\'enyi entropy, Shannon limits, and mutual information through a finite-resolution view of information, clarifying when low-order coincidence structure is informative and when irreducible information is lost.