arXiv subjects

Sergio Verdú

Publications and source records attributed to Sergio Verdú.

At least 19 recordsLinked to original sources

arXiv2019

Second-Order Converses via Reverse Hypercontractivity

A strong converse shows that no procedure can beat the asymptotic (as blocklength $n\to\infty$) fundamental limit of a given information-theoretic problem for any fixed error probability. A second-order converse strengthens this conclusion by showing that the asymptotic fundamental limit cannot be exceeded by more than $O(\tfrac{1}{\sqrt{n}})$. While strong converses are achieved in a broad range of information-theoretic problems by virtue of the "blowing-up method"---a powerful methodology due to Ahlswede, Gács and Körner (1976) based on concentration of measure---this method is fundamentally unable to attain second-order converses and is restricted to finite-alphabet settings. Capitalizing on reverse hypercontractivity of Markov semigroups and functional inequalities, this paper develops the "smoothing-out" method, an alternative to the blowing-up approach that does not rely on finite alphabets and that leads to second-order converses in a variety of information-theoretic problems that were out of reach of previous methods.

cs.IT↗

arXiv2019

Chaining Mutual Information and Tightening Generalization Bounds

Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the hypotheses, (ii) exploiting the dependence between the algorithm's input and output. Progress on the first point was made with the chaining method, originating from the work of Kolmogorov, and used in the VC-dimension bound. More recently, progress on the second point was made with the mutual information method by Russo and Zou '15. Yet, these two methods are currently disjoint. In this paper, we introduce a technique to combine the chaining and mutual information methods, to obtain a generalization bound that is both algorithm-dependent and that exploits the dependencies between the hypotheses. We provide an example in which our bound significantly outperforms both the chaining and the mutual information bounds. As a corollary, we tighten Dudley's inequality when the learning algorithm chooses its output from a small subset of hypotheses with high probability.

cs.LG↗

arXiv2019

Sharp Bounds for Mutual Covering

A fundamental tool in network information theory is the covering lemma, which lower bounds the probability that there exists a pair of random variables, among a give number of independently generated candidates, falling within a given set. We use a weighted sum trick and Talagrand's concentration inequality to prove new mutual covering bounds. We identify two interesting applications: 1) When the probability of the set under the given joint distribution is bounded away from 0 and 1, the covering probability converges to 1 \emph{doubly} exponentially fast in the blocklength, which implies that the covering lemma does not induce penalties on the error exponents in the applications to coding theorems. 2) Using Hall's marriage lemma, we show that the maximum difference between the probability of the set under the joint distribution and the covering probability equals half the minimum total variation distance between the joint distribution and any distribution that can be simulated by selecting a pair from the candidates. Thus we use the mutual covering bound to derive the exact error exponent in the joint distribution simulation problem. In both applications, the determination of the exact exponential (or doubly exponential) behavior relies crucially on the sharp concentration inequality used in the proof of the mutual covering lemma.

cs.IT↗

arXiv2018

Improved Bounds on Lossless Source Coding and Guessing Moments via Rényi Measures

This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probability are also established, characterizing the exact locus of their attainable values. The bounds on optimal guessing moments serve to improve non-asymptotic bounds on the cumulant generating function of the codeword lengths for fixed-to-variable optimal lossless source coding without prefix constraints. Non-asymptotic bounds on the reliability function of discrete memoryless sources are derived as well. Relying on these techniques, lower bounds on the cumulant generating function of the codeword lengths are derived, by means of the smooth Rényi entropy, for source codes that allow decoding errors.

cs.IT↗

arXiv2018

The Error Probability of Generalized Perfect Codes via the Meta-Converse

We introduce a definition of perfect and quasi-perfect codes for symmetric channels parametrized by an auxiliary output distribution. This notion generalizes previous definitions of perfect and quasi-perfect codes and encompasses maximum distance separable codes. The error probability of these codes, whenever they exist, is shown to coincide with the estimate provided by the meta-converse lower bound. We illustrate how the proposed definition naturally extends to cover almost-lossless source-channel coding and lossy compression.

cs.IT↗

arXiv2018

Minimax Rényi Redundancy

The redundancy for universal lossless compression of discrete memoryless sources in Campbell's setting is characterized as a minimax Rényi divergence, which is shown to be equal to the maximal $α$-mutual information via a generalized redundancy-capacity theorem. Special attention is placed on the analysis of the asymptotics of minimax Rényi divergence, which is determined up to a term vanishing in blocklength.

cs.IT↗

arXiv2017

Arimoto-Rényi Conditional Entropy and Bayesian $M$-ary Hypothesis Testing

This paper gives upper and lower bounds on the minimum error probability of Bayesian $M$-ary hypothesis testing in terms of the Arimoto-Rényi conditional entropy of an arbitrary order $α$. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy ($α=1$) is demonstrated. In particular, in the case where $M$ is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing $M$ to be infinite, a lower bound on the Arimoto-Rényi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-Rényi conditional entropy for both positive and negative $α$. Furthermore, we give upper bounds on the minimum error probability as functions of the Rényi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output when averaged over a random coding ensemble.

cs.IT↗

arXiv2017

$E_γ$-Resolvability

The conventional channel resolvability refers to the minimum rate needed for an input process to approximate the channel output distribution in total variation distance. In this paper we study $E_γ$-resolvability, in which total variation is replaced by the more general $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. Let $Q_{\sf X|U}$ be a random transformation, $n$ be an integer, and $E\in(0,+\infty)$. We show that in the asymptotic setting where $γ=\exp(nE)$, a (nonnegative) randomness rate above $\inf_{Q_{\sf U}: D(Q_{\sf X}\|{π}_{\sf X})\le E} \{D(Q_{\sf X}\|{π}_{\sf X})+I(Q_{\sf U},Q_{\sf X|U})-E\}$ is sufficient to approximate the output distribution ${π}_{\sf X}^{\otimes n}$ using the channel $Q_{\sf X|U}^{\otimes n}$, where $Q_{\sf U}\to Q_{\sf X|U}\to Q_{\sf X}$, and is also necessary in the case of finite $\mathcal{U}$ and $\mathcal{X}$. In particular, a randomness rate of $\inf_{Q_{\sf U}}I(Q_{\sf U},Q_{\sf X|U})-E$ is always sufficient. We also study the convergence of the approximation error under the high probability criteria in the case of random codebooks. Moreover, by developing simple bounds relating $E_γ$ and other distance measures, we are able to determine the exact linear growth rate of the approximation errors measured in relative entropy and smooth Rényi divergences for a fixed-input randomness rate. The new resolvability result is then used to derive 1) a one-shot upper bound on the probability of excess distortion in lossy compression, which is exponentially tight in the i.i.d.~setting, 2) a one-shot version of the mutual covering lemma, and 3) a lower bound on the size of the eavesdropper list to include the actual message and a lower bound on the eavesdropper false-alarm probability in the wiretap channel problem, which is (asymptotically) ensemble-tight.

cs.IT↗

arXiv2017

Optimal Universal Lossless Compression with Side Information

This paper presents conditional versions of Lempel-Ziv (LZ) algorithm for settings where compressor and decompressor have access to the same side information. We propose a fixed-length-parsing LZ algorithm with side information, motivated by the Willems algorithm, and prove the optimality for any stationary processes. In addition, we suggest strategies to improve the algorithm which lower the data compression rate. A modification of a variable-length-parsing LZ algorithm with side information is proposed and proved to be asymptotically optimal for any stationary and ergodic processes.

cs.IT↗

arXiv2017

Secret Key Generation with Limited Interaction

A basic two-terminal secret key generation model is considered, where the interactive communication rate between the terminals may be limited, and in particular may not be enough to achieve the maximum key rate. We first prove a multi-letter characterization of the key-communication rate region (where the number of auxiliary random variables depend on the number of rounds of the communication), and then provide an equivalent but simpler characterization in terms of concave envelopes in the case of unlimited number of rounds. Two extreme cases are given special attention. First, in the regime of very low communication rates, the \emph{key bits per interaction bit} (KBIB) is expressed with a new "symmetric strong data processing constant", which has a concave envelope characterization analogous to that of the conventional strong data processing constant. The symmetric strong data processing constant can be upper bounded by the supremum of the maximal correlation coefficient over a set of distributions, which allows us to determine the KBIB for binary symmetric sources, and conclude, in particular, that the interactive scheme is not more efficient than the one-way scheme at least in the low communication-rate regime. Second, a new characterization of the \emph{minimum interaction rate needed for achieving the maximum key rate} (MIMK) is given, and we resolve a conjecture by Tyagi regarding the MIMK for (possibly nonsymmetric) binary sources. We also propose a new conjecture for binary symmetric sources that the interactive scheme is not more efficient than the one-way scheme at any communication rate.

cs.IT↗

arXiv2017

Joint source-channel coding with feedback

This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce $k$ source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint.

cs.IT↗

arXiv2016

$f$-divergence Inequalities

This paper develops systematic approaches to obtain $f$-divergence inequalities, dealing with pairs of probability measures defined on arbitrary alphabets. Functional domination is one such approach, where special emphasis is placed on finding the best possible constant upper bounding a ratio of $f$-divergences. Another approach used for the derivation of bounds among $f$-divergences relies on moment inequalities and the logarithmic-convexity property, which results in tight bounds on the relative entropy and Bhattacharyya distance in terms of $χ^2$ divergences. A rich variety of bounds are shown to hold under boundedness assumptions on the relative information. Special attention is devoted to the total variation distance and its relation to the relative information and relative entropy, including "reverse Pinsker inequalities," as well as on the $E_γ$ divergence, which generalizes the total variation distance. Pinsker's inequality is extended for this type of $f$-divergence, a result which leads to an inequality linking the relative entropy and relative information spectrum. Integral expressions of the Rényi divergence in terms of the relative information spectrum are derived, leading to bounds on the Rényi divergence in terms of either the variational distance or relative entropy.

cs.IT↗

arXiv2016

$f$-Divergence Inequalities via Functional Domination

This paper considers derivation of $f$-divergence inequalities via the approach of functional domination. Bounds on an $f$-divergence based on one or several other $f$-divergences are introduced, dealing with pairs of probability measures defined on arbitrary alphabets. In addition, a variety of bounds are shown to hold under boundedness assumptions on the relative information. The journal paper, which includes more approaches for the derivation of f-divergence inequalities and proofs, is available on the arXiv at https://arxiv.org/abs/1508.00335, and it has been published in the IEEE Trans. on Information Theory, vol. 62, no. 11, pp. 5973-6006, November 2016.

cs.IT↗

arXiv2016

Nonasymptotic noisy lossy source coding

This paper shows new general nonasymptotic achievability and converse bounds and performs their dispersion analysis for the lossy compression problem in which the compressor observes the source through a noisy channel. While this problem is asymptotically equivalent to a noiseless lossy source coding problem with a modified distortion function, nonasymptotically there is a noticeable gap in how fast their minimum achievable coding rates approach the common rate-distortion function, as evidenced both by the refined asymptotic analysis (dispersion) and the numerical results. The size of the gap between the dispersions of the noisy problem and the asymptotically equivalent noiseless problem depends on the stochastic variability of the channel through which the compressor observes the source.

cs.IT↗

arXiv2016

Key Capacity for Product Sources with Application to Stationary Gaussian Processes

We show that for product sources, rate splitting is optimal for secret key agreement using limited one-way communication between two terminals. This yields an alternative information-theoretic-converse-style proof of the tensorization property of a strong data processing inequality originally studied by Erkip and Cover and amended recently by Anantharam et al. We derive a water-filling solution of the communication-rate--key-rate tradeoff for a wide class of discrete memoryless vector Gaussian sources which subsumes the case without an eavesdropper. Moreover, we derive an explicit formula for the maximum secret key per bit of communication for all discrete memoryless vector Gaussian sources using a tensorization property and a variation on the enhanced channel technique of Weingarten et al. Finally, a one-shot information spectrum achievability bound for key generation is proved from which we characterize the communication-rate--key-rate tradeoff for stationary Gaussian processes.

cs.IT↗

arXiv2015

Channels with cost constraints: strong converse and dispersion

This paper shows the strong converse and the dispersion of memoryless channels with cost constraints and performs refined analysis of the third order term in the asymptotic expansion of the maximum achievable channel coding rate, showing that it is equal to $\frac 1 2 \frac {\log n}{n}$ in most cases of interest. The analysis is based on a non-asymptotic converse bound expressed in terms of the distribution of a random variable termed the $\mathsf b$-tilted information density, which plays a role similar to that of the $\mathsf d$-tilted information in lossy source coding. We also analyze the fundamental limits of lossy joint-source-channel coding over channels with cost constraints.

cs.IT↗

arXiv2015

Variable-length compression allowing errors

This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability $ε$, for lossless compression. We give non-asymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit which is quite accurate for all but small blocklengths: $$(1 - ε) k H(\mathsf S) - \sqrt{\frac{k V(\mathsf S)}{2 π} } e^{- \frac {(Q^{-1}(ε))^2} 2 }$$ where $Q^{-1}(\cdot)$ is the functional inverse of the standard Gaussian complementary cdf, and $V(\mathsf S)$ is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of $1 - ε$, but this asymptotic limit is approached from below, i.e. larger source dispersions and shorter blocklengths are beneficial. Variable-length lossy compression under an excess distortion constraint is shown to exhibit similar properties.

cs.IT↗

arXiv2015

Resolvability in Eγ with Applications to Lossy Compression and Wiretap Channels

We study the amount of randomness needed for an input process to approximate a given output distribution of a channel in the $E_γ$ distance. A general one-shot achievability bound for the precision of such an approximation is developed. In the i.i.d.~setting where $γ=\exp(nE)$, a (nonnegative) randomness rate above $\inf_{Q_{\sf U}: D(Q_{\sf X}||π_{\sf X})\le E} \{D(Q_{\sf X}||π_{\sf X})+I(Q_{\sf U},Q_{\sf X|U})-E\}$ is necessary and sufficient to asymptotically approximate the output distribution $π_{\sf X}^{\otimes n}$ using the channel $Q_{\sf X|U}^{\otimes n}$, where $Q_{\sf U}\to Q_{\sf X|U}\to Q_{\sf X}$. The new resolvability result is then used to derive a one-shot upper bound on the error probability in the rate distortion problem, and a lower bound on the size of the eavesdropper list to include the actual message in the wiretap channel problem. Both bounds are asymptotically tight in i.i.d.~settings.

cs.IT↗