Searcharxiv⌕ Search

arXiv subjects

Mokshay Madiman

Publications and source records attributed to Mokshay Madiman.

At least 37 records · Page 2Linked to original sources
arXiv2020

The Differential Entropy of Mixtures: New Bounds and Applications

Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of probability distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the entropy of the mixture and the weighted sum of entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of "skew" $f$-divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities.

cs.IT↗
arXiv2020

Concentration of information content for convex measures

We establish sharp exponential deviation estimates of the information content as well as a sharp bound on the varentropy for the class of convex measures on Euclidean spaces. This generalizes a similar development for log-concave measures in the recent work of Fradelizi, Madiman and Wang (2016). In particular, our results imply that convex measures in high dimensions are concentrated in an annulus between two convex sets (as in the log-concave case) despite their possibly having much heavier tails. Various tools and consequences are developed, including a sharp comparison result for Rényi entropies, inequalities of Kahane-Khinchine type for convex measures that extend those of Koldobsky, Pajor and Yaskin (2008) for log-concave measures, and an extension of Berwald's inequality (1947).

math.PR↗
arXiv2019

Two remarks on generalized entropy power inequalities

This note contributes to the understanding of generalized entropy power inequalities. Our main goal is to construct a counter-example regarding monotonicity and entropy comparison of weighted sums of independent identically distributed log-concave random variables. We also present a complex analogue of a recent dependent entropy power inequality of Hao and Jog, and give a very simple proof.

cs.IT↗
arXiv2018

Combinatorial Entropy Power Inequalities: A Preliminary Study of the Stam region

We initiate the study of the Stam region, defined as the subset of the positive orthant in $\mathbb{R}^{2^n-1}$ that arises from considering entropy powers of subset sums of $n$ independent random vectors in a Euclidean space of finite dimension. We show that the class of fractionally superadditive set functions provides an outer bound to the Stam region, resolving a conjecture of A. R. Barron and the first author. On the other hand, the entropy power of a sum of independent random vectors is not supermodular in any dimension. We also develop some qualitative properties of the Stam region, showing for instance that its closure is a logarithmically convex cone.

math.PR↗
arXiv2018

A combinatorial approach to small ball inequalities for sums and differences

Small ball inequalities have been extensively studied in the setting of Gaussian processes and associated Banach or Hilbert spaces. In this paper, we focus on studying small ball probabilities for sums or differences of independent, identically distributed random elements taking values in very general sets. Depending on the setting--abelian or nonabelian groups, or vector spaces, or Banach spaces--we provide a collection of inequalities relating different small ball probabilities that are sharp in many cases of interest. We prove these distribution-free probabilistic inequalities by showing that underlying them are inequalities of extremal combinatorial nature, related among other things to classical packing problems such as the kissing number problem. Applications are given to moment inequalities.

math.PR↗
arXiv2018

The convexification effect of Minkowski summation

Let us define for a compact set $A \subset \mathbb{R}^n$ the sequence $$ A(k) = \left\{\frac{a_1+\cdots +a_k}{k}: a_1, \ldots, a_k\in A\right\}=\frac{1}{k}\Big(\underset{k\ {\rm times}}{\underbrace{A + \cdots + A}}\Big). $$ It was independently proved by Shapley, Folkman and Starr (1969) and by Emerson and Greenleaf (1969) that $A(k)$ approaches the convex hull of $A$ in the Hausdorff distance induced by the Euclidean norm as $k$ goes to $\infty$. We explore in this survey how exactly $A(k)$ approaches the convex hull of $A$, and more generally, how a Minkowski sum of possibly different compact sets approaches convexity, as measured by various indices of non-convexity. The non-convexity indices considered include the Hausdorff distance induced by any norm on $\mathbb{R}^n$, the volume deficit (the difference of volumes), a non-convexity index introduced by Schneider (1975), and the effective standard deviation or inner radius. After first clarifying the interrelationships between these various indices of non-convexity, which were previously either unknown or scattered in the literature, we show that the volume deficit of $A(k)$ does not monotonically decrease to 0 in dimension 12 or above, thus falsifying a conjecture of Bobkov et al. (2011), even though their conjecture is proved to be true in dimension 1 and for certain sets $A$ with special structure. On the other hand, Schneider's index possesses a strong monotonicity property along the sequence $A(k)$, and both the Hausdorff distance and effective standard deviation are eventually monotone (once $k$ exceeds $n$). Along the way, we obtain new inequalities for the volume of the Minkowski sum of compact sets, falsify a conjecture of Dyn and Farkhi (2004), demonstrate applications of our results to combinatorial discrepancy theory, and suggest some questions worthy of further investigation.

math.FA↗
arXiv2017

Deviation inequalities for convex functions motivated by the Talagrand conjecture

Motivated by Talagrand's conjecture on regularization properties of the natural semigroup on the Boolean hypercube, and in particular its continuous analogue involving regularization properties of the Ornstein-Uhlenbeck semigroup acting on in-tegrable functions, we explore deviation inequalities for log-semiconvex functions under Gaussian measure.

math.PR↗
arXiv2017

Rogozin's convolution inequality for locally compact groups

General extensions of an inequality due to Rogozin, concerning the essential supremum of a convolution of probability density functions on the real line, are obtained. While a weak version of the inequality is proved in the very general context of Polish $σ$-compact groups, particular attention is paid to the group \(\mathbb{R}^d\), where the result can combined with rearrangement inequalities for certain linear images for a strong generalization. As a consequence, we obtain a unification and sharpening of both the \(\infty\)-Renyi entropy power inequality for sums of independent random vectors, due to Bobkov and Chistyakov, and the bounds on marginals of projections of product measures due to Rudelson and Vershynin (matching and extending the sharp improvement of Livshyts, Paouris and Pivovarov). The proof is elementary and relies on a characterization of extreme points of a class of probability measures in the general setting of Polish measure spaces, as well as the development of a generalization of Ball's cube slicing bounds for products of \(d\)-dimensional Euclidean balls (where the "co-dimension 1" case had been recently settled by Brzezinski).

math.PR↗
arXiv2016

Forward and Reverse Entropy Power Inequalities in Convex Geometry

The entropy power inequality, which plays a fundamental role in information theory and probability, may be seen as an analogue of the Brunn-Minkowski inequality. Motivated by this connection to Convex Geometry, we survey various recent developments on forward and reverse entropy power inequalities not just for the Shannon-Boltzmann entropy but also more generally for Rényi entropy. In the process, we discuss connections between the so-called functional (or integral) and probabilistic (or entropic) analogues of some classical inequalities in geometric functional analysis

cs.IT↗
arXiv2016

The norm of the Fourier transform on compact or discrete abelian groups

We calculate the norm of the Fourier operator from $L^p(X)$ to $L^q(\hat{X})$ when $X$ is an infinite locally compact abelian group that is, furthermore, compact or discrete. This subsumes the sharp Hausdorff-Young inequality on such groups. In particular, we identify the region in $(p,q)$-space where the norm is infinite, generalizing a result of Fournier, and setting up a contrast with the case of finite abelian groups, where the norm was determined by Gilbert and Rzeszotnik. As an application, uncertainty principles on such groups expressed in terms of Rényi entropies are discussed.

math.CA↗
arXiv2016

Entropies of weighted sums in cyclic groups and an application to polar codes

In this note, the following basic question is explored: in a cyclic group, how are the Shannon entropies of the sum and difference of i.i.d. random variables related to each other? For the integer group, we show that they can differ by any real number additively, but not too much multiplicatively; on the other hand, for $\mathbb{Z}/3\mathbb{Z}$, the entropy of the difference is always at least as large as that of the sum. These results are closely related to the study of more-sum-than-difference (i.e. MSTD) sets in additive combinatorics. We also investigate polar codes for $q$-ary input channels using non-canonical kernels to construct the generator matrix, and present applications of our results to constructing polar codes with significantly improved error probability compared to the canonical construction.

cs.IT↗
arXiv2016

Do Minkowski averages get progressively more convex?

Let us define, for a compact set $A \subset \mathbb{R}^n$, the Minkowski averages of $A$: $$ A(k) = \left\{\frac{a_1+\cdots +a_k}{k} : a_1, \ldots, a_k\in A\right\}=\frac{1}{k}\Big(\underset{k\ {\rm times}}{\underbrace{A + \cdots + A}}\Big). $$ We study the monotonicity of the convergence of $A(k)$ towards the convex hull of $A$, when considering the Hausdorff distance, the volume deficit and a non-convexity index of Schneider as measures of convergence. For the volume deficit, we show that monotonicity fails in general, thus disproving a conjecture of Bobkov, Madiman and Wang. For Schneider's non-convexity index, we prove that a strong form of monotonicity holds, and for the Hausdorff distance, we establish that the sequence is eventually nonincreasing.

math.MG↗
arXiv2015

Optimal Concentration of Information Content For Log-Concave Densities

An elementary proof is provided of sharp bounds for the varentropy of random vectors with log-concave densities, as well as for deviations of the information content from its mean. These bounds significantly improve on the bounds obtained by Bobkov and Madiman ({\it Ann. Probab.}, 39(4):1528--1543, 2011).

math.PR↗
arXiv2015

Entropy bounds on abelian groups and the Ruzsa divergence

Over the past few years, a family of interesting new inequalities for the entropies of sums and differences of random variables has been developed by Ruzsa, Tao and others, motivated by analogous results in additive combinatorics. The present work extends these earlier results to the case of random variables taking values in $\mathbb{R}^n$ or, more generally, in arbitrary locally compact and Polish abelian groups. We isolate and study a key quantity, the Ruzsa divergence between two probability distributions, and we show that its properties can be used to extend the earlier inequalities to the present general setting. The new results established include several variations on the theme that the entropies of the sum and the difference of two independent random variables severely constrain each other. Although the setting is quite general, the result are already of interest (and new) for random vectors in $\mathbb{R}^n$. In that special case, quantitative bounds are provided for the stability of the equality conditions in the entropy power inequality; a reverse entropy power inequality for log-concave random vectors is proved; an information-theoretic analog of the Rogers-Shephard inequality for convex bodies is established; and it is observed that some of these results lead to new inequalities for the determinants of positive-definite matrices. Moreover, by considering the multiplicative subgroups of the complex plane, one obtains new inequalities for the differential entropies of products and ratios of nonzero, complex-valued random variables.

cs.IT↗
arXiv2014

Beyond the entropy power inequality, via rearrangements

A lower bound on the Rényi differential entropy of a sum of independent random vectors is demonstrated in terms of rearrangements. For the special case of Boltzmann-Shannon entropy, this lower bound is better than that given by the entropy power inequality. Several applications are discussed, including a new proof of the classical entropy power inequality and an entropy inequality involving symmetrization of Lévy processes.

cs.IT↗
arXiv2013

Contribution to the theory of Pitman estimators

New inequalities are proved for the variance of the Pitman estimators (minimum variance equivariant estimators) of θconstructed from samples of fixed size from populations F(x-θ). The inequalities are closely related to the classical Stam inequality for the Fisher information, its analog in small samples, and a powerful variance drop inequality. The only condition required is finite variance of F; even the absolute continuity of F is not assumed. As corollaries of the main inequalities for small samples, one obtains alternate proofs of known properties of the Fisher information, as well as interesting new observations like the fact that the variance of the Pitman estimator based on a sample of size n scaled by n monotonically decreases in n. Extensions of the results to the polynomial versions of the Pitman estimators and a multivariate location parameter are given. Also, the search for characterization of equality conditions for one of the inequalities leads to a Cauchy-type functional equation for independent random variables, and an interesting new behavior of its solutions is described.

math.ST↗