On Pinsker's Type Inequalities and Csiszar's f-divergences. Part I: Second and Fourth-Order Inequalities

Abstract

We study conditions on $f$ under which an $f$-divergence $D_f$ will satisfy $D_f \geq c_f V^2$ or $D_f \geq c_{2,f} V^2 + c_{4,f} V^4$, where $V$ denotes variational distance and the coefficients $c_f$, $c_{2,f}$ and $c_{4,f}$ are {\em best possible}. As a consequence, we obtain lower bounds in terms of $V$ for many well known distance and divergence measures. For instance, let $D_{(α)} (P,Q) = [α(α-1)]^{-1} [\int q^α p^{1-α} d μ-1]$ and ${\cal I}_α(P,Q) = (α-1)^{-1} \log [\int p^αq^{1-α} d μ]$ be respectively the {\em relative information of type} ($1-α$) and {\em Rényi's information gain of order} $α$. We show that $D_{(α)} \geq {1/2} V^2 + {1/72} (α+1)(2-α) V^4$ whenever $-1 \leq α\leq 2$, $α\not= 0,1$ and that ${\cal I}_α = \fracα{2} V^2 + {1/36} α(1 + 5 α- 5 α^2) V^4$ for $0 < α< 1$. Pinsker's inequality $D \geq {1/2} V^2$ and its extension $D \geq {1/2} V^2 + {1/36} V^4$ are special cases of each one of these.