Entropy Estimates from Insufficient Samplings

P. Grassberger

Home/Archive/arXiv

arXiv · physics/0307138

Entropy Estimates from Insufficient Samplings

P. Grassberger

View original source ↗Read source document ↗Export metadata ↓

Abstract

We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1 points in each box) they have exponentially small biases. In the low sampling regime the errors increase but are still much smaller than for most other estimators. One advantage is that our main estimators are given analytically, with explicitly known analytical formulas for the biases.

Explore related subjects

physics.data-an physics.comp-ph

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

P. Grassberger. 2008-01-04. Entropy Estimates from Insufficient Samplings. https://arxiv.org/abs/physics/0307138

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A Probability Model for Pentagonal Prism Dice Rolls

In the realm of probability theory and game design, the study of dice probabilities plays a crucial role. The featured dice are usually fair, so their probabilities can be predicted without rolling the dice. Pentagonal prismatic dice are not fair. Hence, the probabilities are more complicated. Our mathematical model is naturally derived from both the geometric properties of the dice and empirical evidence. Dice of varying sizes but with constant volume were designed. Using a 3D printer with 100 percent infill, the dice were printed and prepared. A machine that simulates dropping dice through a dice tower onto a hard surface was used to obtain about 41,000 results. From these data, a statistical model was constructed that gives probabilities of dice with varying height-to-radius ratios. The formula for the model is based on the centroid solid angle, but with two adjustment parameters due to varying energy and other requirements to transition between different resting aspects. Within the range of height-to-radius ratios tested, the data are within reasonable statistical error of the model.

physics.data-an↗

arXiv2026

Accelerating fits in high-energy physics and beyond

Data analysis in particle physics often involves optimizing a loss function that quantifies how well a theoretical model describes the observed data, in order to estimate the parameters of the model. This is referred to as fitting. As datasets grow larger and more expensive analysis methods are used, the time required to perform this parameter estimation can be a limiting factor. This work explores techniques for accelerating the fitting process, and measures their performance on a set of models representative of a variety of physics fits. We compare the speed and precision of variants of well-known optimization algorithms, namely Newton's method, BFGS, and the Migrad algorithm (commonly used in particle physics fits). We observe significant speedups when using some algorithms: most notably, some variants of BFGS are about ten times faster than Migrad, while achieving the same precision. For quasi-Newtonian algorithms like BFGS and Migrad, we find that the initial value of the Hessian matrix has a large impact on the runtime. We additionally discuss methods for accelerating the evaluation of the loss function and its derivatives, using algebraic simplifications, automatic differentiation, and modern hardware features. We explore techniques for achieving good precision with 32-bit operations, which can achieve significantly higher throughput on the GPU. Lastly we explore efficient parallelization strategies for the CPU and GPU. All of the techniques and algorithms in this paper are implemented in our new FastFit library, which was used for all of our benchmarks.

physics.data-an↗

arXiv2026

Elementwise Adaptive Random Search for High-Dimensional Constrained Continuous Global Optimization

We propose a new stochastic method for high-dimensional constrained continuous global optimization without convexity assumptions. The algorithm combines global exploration and local refinement through coordinate-wise adaptive perturbations, whose scale is proportional to the current iterate. We show that global exploration reaches any feasible region with positive probability, while local search benefits from strong concentration properties, enabling efficient decline near optima. Their combination yields convergence to a global minimizer under mild regularity conditions. Applications to LASSO sparsity improvement as well as similar BPDN (Basis Pursuit DeNoising) problems, and to neural network pruning, illustrate the practical effectiveness of the approach.

physics.data-an↗