arXiv subjects

Yury Polyanskiy

Publications and source records attributed to Yury Polyanskiy.

At least 109 records · Page 6Linked to original sources

arXiv2019

Energy efficient coded random access for the wireless uplink

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free and uncoordinated uplink for densely packed wireless nodes. Specifically, we study random-access codes, previously introduced for the AWGN multiple-access channel (MAC) by Polyanskiy'2017, in the practically more relevant case of users subject to Rayleigh fading, when channel gains are unknown to the decoder. We propose a random coding achievability bound, which we analyze both non-asymptotically (at finite blocklength) and asymptotically. As a candidate practical solution, we propose an explicit sparse-graph based coding scheme together with an alternating belief-propagation decoder. The latter's performance is found to be surprisingly close to the finite-blocklength bounds. Our main findings are twofold. First, just like in the AWGN MAC we see that jointly decoding large number of users leads to a surprising phase transition effect, where at spectral efficiencies below a critical threshold (5-15 bps/Hz depending on reliability) a perfect multi-user interference cancellation is possible. Second, while the presence of Rayleigh fading significantly increases the minimal required energy-per-bit $E_b/N_0$ (from about 0-2 dB to about 8-11 dB), the inherent randomization introduced by the channel makes it much easier to attain the optimal performance via iterative schemes. In all, it is hoped that a principled definition of the random-access model together with our information-theoretic analysis will open the road towards unified benchmarking and comparison performance of various random-access solutions, such as the currently discussed candidates (MUSA, SCMA, RSMA) for the 5G/6G.

cs.IT↗

arXiv2019

Estimating Differential Entropy under Gaussian Convolutions

This paper studies the problem of estimating the differential entropy $h(S+Z)$, where $S$ and $Z$ are independent $d$-dimensional random variables with $Z\sim\mathcal{N}(0,σ^2 \mathrm{I}_d)$. The distribution of $S$ is unknown, but $n$ independently and identically distributed (i.i.d) samples from it are available. The question is whether having access to samples of $S$ as opposed to samples of $S+Z$ can improve estimation performance. We show that the answer is positive. More concretely, we first show that despite the regularizing effect of noise, the number of required samples still needs to scale exponentially in $d$. This result is proven via a random-coding argument that reduces the question to estimating the Shannon entropy on a $2^{O(d)}$-sized alphabet. Next, for a fixed $d$ and $n$ large enough, it is shown that a simple plugin estimator, given by the differential entropy of the empirical distribution from $S$ convolved with the Gaussian density, achieves the loss of $O\left((\log n)^{d/4}/\sqrt{n}\right)$. Note that the plugin estimator amounts here to the differential entropy of a $d$-dimensional Gaussian mixture, for which we propose an efficient Monte Carlo computation algorithm. At the same time, estimating $h(S+Z)$ via popular differential entropy estimators (based on kernel density estimation (KDE) or k nearest neighbors (kNN) techniques) applied to samples from $S+Z$ would only attain much slower rates of order $O(n^{-1/d})$, despite the smoothness of $P_{S+Z}$. As an application, which was in fact our original motivation for the problem, we estimate information flows in deep neural networks and discuss Tishby's Information Bottleneck and the compression conjecture, among others.

math.ST↗

arXiv2019

Estimating Information Flow in Deep Neural Networks

We study the flow of information and the evolution of internal representations during deep neural network (DNN) training, aiming to demystify the compression aspect of the information bottleneck theory. The theory suggests that DNN training comprises a rapid fitting phase followed by a slower compression phase, in which the mutual information $I(X;T)$ between the input $X$ and internal representations $T$ decreases. Several papers observe compression of estimated mutual information on different DNN models, but the true $I(X;T)$ over these networks is provably either constant (discrete $X$) or infinite (continuous $X$). This work explains the discrepancy between theory and experiments, and clarifies what was actually measured by these past works. To this end, we introduce an auxiliary (noisy) DNN framework for which $I(X;T)$ is a meaningful quantity that depends on the network's parameters. This noisy framework is shown to be a good proxy for the original (deterministic) DNN both in terms of performance and the learned representations. We then develop a rigorous estimator for $I(X;T)$ in noisy DNNs and observe compression in various models. By relating $I(X;T)$ in the noisy DNN to an information-theoretic communication problem, we show that compression is driven by the progressive clustering of hidden representations of inputs from the same class. Several methods to directly monitor clustering of hidden representations, both in noisy and deterministic DNNs, are used to show that meaningful clusters form in the $T$ space. Finally, we return to the estimator of $I(X;T)$ employed in past works, and demonstrate that while it fails to capture the true (vacuous) mutual information, it does serve as a measure for clustering. This clarifies the past observations of compression and isolates the geometric clustering of hidden representations as the true phenomenon of interest.

cs.LG↗

arXiv2019

Improved log-Sobolev inequalities, hypercontractivity and uncertainty principle on the hypercube

Log-Sobolev inequalities (LSIs) upper-bound entropy via a multiple of the Dirichlet form (i.e. norm of a gradient). In this paper we prove a family of entropy-energy inequalities for the binary hypercube which provide a non-linear comparison between the entropy and the Dirichlet form and improve on the usual LSIs for functions with small support. These non-linear LSIs, in turn, imply a new version of the hypercontractivity for such functions. As another consequence, we derive a sharp form of the uncertainty principle for the hypercube: a function whose energy is concentrated on a set of small size, and whose Fourier energy is concentrated on a small Hamming ball must be zero. The tradeoff between the sizes that we derive is asymptotically optimal. This new uncertainty principle implies a new estimate on the size of Fourier coefficients of sparse Boolean functions. We observe that an analogous (asymptotically optimal) uncertainty principle in the Euclidean space follows from the sharp form of Young's inequality due to Beckner. This hints that non-linear LSIs augment Young's inequality (which itself is sharp for finite groups).

math.PR↗

arXiv2019

Communication Complexity of Estimating Correlations

We characterize the communication complexity of the following distributed estimation problem. Alice and Bob observe infinitely many iid copies of $ρ$-correlated unit-variance (Gaussian or $\pm1$ binary) random variables, with unknown $ρ\in[-1,1]$. By interactively exchanging $k$ bits, Bob wants to produce an estimate $\hatρ$ of $ρ$. We show that the best possible performance (optimized over interaction protocol $Π$ and estimator $\hat ρ$) satisfies $\inf_{Π\hatρ}\sup_ρ\mathbb{E} [|ρ-\hatρ|^2] = \tfrac{1}{k} (\frac{1}{2 \ln 2} + o(1))$. Curiously, the number of samples in our achievability scheme is exponential in $k$; by contrast, a naive scheme exchanging $k$ samples achieves the same $Ω(1/k)$ rate but with a suboptimal prefactor. Our protocol achieving optimal performance is one-way (non-interactive). We also prove the $Ω(1/k)$ bound even when $ρ$ is restricted to any small open sub-interval of $[-1,1]$ (i.e. a local minimax lower bound). Our proof techniques rely on symmetric strong data-processing inequalities and various tensorization techniques from information-theoretic interactive common-randomness extraction. Our results also imply an $Ω(n)$ lower bound on the information complexity of the Gap-Hamming problem, for which we show a direct information-theoretic proof.

cs.IT↗

arXiv2019

A simple bound on the BER of the MAP decoder for massive MIMO systems

The deployment of massive MIMO systems has revived much of the interest in the study of the large-system performance of multiuser detection systems. In this paper, we prove a non-trivial upper bound on the bit-error rate (BER) of the MAP detector for BPSK signal transmission and equal-power condition. In particular, our bound is approximately tight at high-SNR. The proof is simple and relies on Gordon's comparison inequality. Interestingly, we show that under the assumption that Gordon's inequality is tight, the resulting BER prediction matches that of the replica method under the replica symmetry (RS) ansatz. Also, we prove that, when the ratio of receive to transmit antennas exceeds $0.9251$, the replica prediction matches the matched filter lower bound (MFB) at high-SNR. We corroborate our results by numerical evidence.

cs.IT↗

arXiv2018

Comparison of Channels: Criteria for Domination by a Symmetric Channel

This paper studies the basic question of whether a given channel $V$ can be dominated (in the precise sense of being more noisy) by a $q$-ary symmetric channel. The concept of "less noisy" relation between channels originated in network information theory (broadcast channels) and is defined in terms of mutual information or Kullback-Leibler divergence. We provide an equivalent characterization in terms of $χ^2$-divergence. Furthermore, we develop a simple criterion for domination by a $q$-ary symmetric channel in terms of the minimum entry of the stochastic matrix defining the channel $V$. The criterion is strengthened for the special case of additive noise channels over finite Abelian groups. Finally, it is shown that domination by a symmetric channel implies (via comparison of Dirichlet forms) a logarithmic Sobolev inequality for the original channel.

cs.IT↗

arXiv2018

Hypercontractivity of spherical averages in Hamming space

Consider the linear space of functions on the binary hypercube and the linear operator $S_δ$ acting by averaging a function over a Hamming sphere of radius $δn$ around every point. It is shown that this operator has a dimension-independent bound on the norm $L_p \to L_2$ with $p = 1+(1-2δ)^2$. This result evidently parallels a classical estimate of Bonami and Gross for $L_p \to L_q$ norms for the operator of convolution with a Bernoulli noise. The estimate for $S_δ$ is harder to obtain since the latter is neither a part of a semigroup, nor a tensor power. The result is shown by a detailed study of the eigenvalues of $S_δ$ and $L_p\to L_2$ norms of the Fourier multiplier operators $Π_a$ with symbol equal to a characteristic function of the Hamming sphere of radius $a$ (in the notation common in boolean analysis $Π_a f=f^{=a}$, where $f^{=a}$ is a degree-$a$ component of function $f$). A sample application of the result is given: Any set $A\subset \FF_2^n$ with the property that $A+A$ contains a large portion of some Hamming sphere (counted with multiplicity) must have cardinality a constant multiple of $2^n$.

math.PR↗

arXiv2018

Coherent multiple-antenna block-fading channels at finite blocklength

In this paper we consider a channel model that is often used to describe the mobile wireless scenario: multiple-antenna additive white Gaussian noise channels subject to random (fading) gain with full channel state information at the receiver. Dynamics of the fading process are approximated by a piecewise-constant process (frequency non-selective isotropic block fading). This work addresses the finite blocklength fundamental limits of this channel model. Specifically, we give a formula for the channel dispersion -- a quantity governing the delay required to achieve capacity. Multiplicative nature of the fading disturbance leads to a number of interesting technical difficulties that required us to enhance traditional methods for finding channel dispersion. Alas, one difficulty remains: the converse (impossibility) part of our result holds under an extra constraint on the growth of the peak-power with blocklength. Our results demonstrate, for example, that while capacities of $n_t\times n_r$ and $n_r \times n_t$ antenna configurations coincide (under fixed received power), the coding delay can be quite sensitive to this switch. For example, at the received SNR of $20$ dB the $16\times 100$ system achieves capacity with codes of length (delay) which is only $60\%$ of the length required for the $100\times 16$ system. Another interesting implication is that for the MISO channel, the dispersion-optimal coding schemes require employing orthogonal designs such as Alamouti's scheme -- a surprising observation considering the fact that Alamouti's scheme was designed for reducing demodulation errors, not improving coding rate. Finding these dispersion-optimal coding schemes naturally gives a criteria for producing orthogonal design-like inputs in dimensions where orthogonal designs do not exist.

cs.IT↗

arXiv2018

Beta-Beta Bounds: Finite-Blocklength Analog of the Golden Formula

It is well known that the mutual information between two random variables can be expressed as the difference of two relative entropies that depend on an auxiliary distribution, a relation sometimes referred to as the golden formula. This paper is concerned with a finite-blocklength extension of this relation. This extension consists of two elements: 1) a finite-blocklength channel-coding converse bound by Polyanskiy and Verdú (2014), which involves the ratio of two Neyman-Pearson $β$ functions (beta-beta converse bound); and 2) a novel beta-beta channel-coding achievability bound, expressed again as the ratio of two Neyman-Pearson $β$ functions. To demonstrate the usefulness of this finite-blocklength extension of the golden formula, the beta-beta achievability and converse bounds are used to obtain a finite-blocklength extension of Verdú's (2002) wideband-slope approximation. The proof parallels the derivation of the latter, with the beta-beta bounds used in place of the golden formula. The beta-beta (achievability) bound is also shown to be useful in cases where the capacity-achieving output distribution is not a product distribution due to, e.g., a cost constraint or structural constraints on the codebook, such as orthogonality or constant composition. As an example, the bound is used to characterize the channel dispersion of the additive exponential-noise channel and to obtain a finite-blocklength achievability bound (the tightest to date) for multiple-input multiple-output Rayleigh-fading channels with perfect channel state information at the receiver.

cs.IT↗

arXiv2018

List-decodable zero-rate codes

We consider list-decoding in the zero-rate regime for two cases: the binary alphabet and the spherical codes in Euclidean space. Specifically, we study the maximal $τ\in [0,1]$ for which there exists an arrangement of $M$ balls of relative Hamming radius $τ$ in the binary hypercube (of arbitrary dimension) with the property that no point of the latter is covered by $L$ or more of them. As $M\to \infty$ the maximal $τ$ decreases to a well-known critical value $τ_L$. In this work, we prove several results on the rate of this convergence. For the binary case, we show that the rate is $Θ(M^{-1})$ when $L$ is even, thus extending the classical results of Plotkin and Levenshtein for $L=2$. For $L=3$ the rate is shown to be $Θ(M^{-\tfrac{2}{3}})$. For the similar question about spherical codes, we prove the rate is $Ω(M^{-1})$ and $O(M^{-\tfrac{2L}{L^2-L+2}})$.

cs.IT↗

arXiv2018

Broadcasting on Bounded Degree DAGs

We study the following generalization of the well-known model of broadcasting on trees. Consider an infinite directed acyclic graph (DAG) with a unique source node $X$. Let the collection of nodes at distance $k$ from $X$ be called the $k$th layer. At time zero, the source node is given a bit. At time $k\geq 1$, each node in the $(k-1)$th layer inspects its inputs and sends a bit to its descendants in the $k$th layer. Each bit is flipped with a probability of error $δ\in \left(0,\frac{1}{2}\right)$ in the process of transmission. The goal is to be able to recover the original bit with probability of error better than $\frac{1}{2}$ from the values of all nodes at an arbitrarily deep layer $k$. Besides its natural broadcast interpretation, the DAG broadcast is a natural model of noisy computation. Some special cases of the model represent information flow in biological networks, and other cases represent noisy finite automata models. We show that there exist DAGs with bounded degree and layers of size $ω(\log(k))$ that permit recovery provided $δ$ is sufficiently small and find the critical $δ$ for the DAGs constructed. Our result demonstrates a doubly-exponential advantage for storing a bit in bounded degree DAGs compared to trees. On the negative side, we show that if the DAG is a two-dimensional regular grid, then recovery is impossible for any $δ\in \left(0,\frac{1}{2}\right)$ provided all nodes use either AND or XOR for their processing functions.

cs.IT↗

arXiv2018

Almost Optimal Scaling of Reed-Muller Codes on BEC and BSC Channels

Consider a binary linear code of length $N$, minimum distance $d_{\text{min}}$, transmission over the binary erasure channel with parameter $0 < ε< 1$ or the binary symmetric channel with parameter $0 < ε< \frac12$, and block-MAP decoding. It was shown by Tillich and Zemor that in this case the error probability of the block-MAP decoder transitions "quickly" from $δ$ to $1-δ$ for any $δ>0$ if the minimum distance is large. In particular the width of the transition is of order $O(1/\sqrt{d_{\text{min}}})$. We strengthen this result by showing that under suitable conditions on the weight distribution of the code, the transition width can be as small as $Θ(1/N^{\frac12-κ})$, for any $κ>0$, even if the minimum distance of the code is not linear. This condition applies e.g., to Reed-Mueller codes. Since $Θ(1/N^{\frac12})$ is the smallest transition possible for any code, we speak of "almost" optimal scaling. We emphasize that the width of the transition says nothing about the location of the transition. Therefore this result has no bearing on whether a code is capacity-achieving or not. As a second contribution, we present a new estimate on the derivative of the EXIT function, the proof of which is based on the Blowing-Up Lemma.

cs.IT↗

arXiv2017

Strong Data Processing Inequalities for Input Constrained Additive Noise Channels

This paper quantifies the intuitive observation that adding noise reduces available information by means of non-linear strong data processing inequalities. Consider the random variables $W\to X\to Y$ forming a Markov chain, where $Y=X+Z$ with $X$ and $Z$ real-valued, independent and $X$ bounded in $L_p$-norm. It is shown that $I(W;Y) \le F_I(I(W;X))$ with $F_I(t) 0$, if and only if $Z$ has a density whose support is not disjoint from any translate of itself. A related question is to characterize for what couplings $(W,X)$ the mutual information $I(W;Y)$ is close to maximum possible. To that end we show that in order to saturate the channel, i.e. for $I(W;Y)$ to approach capacity, it is mandatory that $I(W;X)\to\infty$ (under suitable conditions on the channel). A key ingredient for this result is a deconvolution lemma which shows that post-convolution total variation distance bounds the pre-convolution Kolmogorov-Smirnov distance. Explicit bounds are provided for the special case of the additive Gaussian noise channel with quadratic cost constraint. These bounds are shown to be order-optimal. For this case simplified proofs are provided leveraging Gaussian-specific tools such as the connection between information and estimation (I-MMSE) and Talagrand's information-transportation inequality.

cs.IT↗

arXiv2017

Defect tolerance: fundamental limits and examples

This paper addresses the problem of adding redundancy to a collection of physical objects so that the overall system is more robust to failures. In contrast to its information counterpart, which can exploit parity to protect multiple information symbols from a single erasure, physical redundancy can only be realized through duplication and substitution of objects. We propose a bipartite graph model for designing defect-tolerant systems in which defective objects are replaced by judiciously connected redundant objects. The fundamental limits of this model are characterized under various asymptotic settings and both asymptotic and finite-size systems that approach these limits are constructed. Among other results, we show that simple modular redundancy is in general suboptimal. As we develop, this combinatorial problem of defect tolerant system design has a natural interpretation as one of graph coloring, and the analysis is significantly different from that traditionally used in information redundancy for error-control codes.

cs.IT↗

arXiv2017

Remote Estimation of the Wiener Process over a Channel with Random Delay

In this paper, we consider a problem of sampling a Wiener process, with samples forwarded to a remote estimator via a channel that consists of a queue with random delay. The estimator reconstructs a real-time estimate of the signal from causally received samples. Motivated by recent research on age-of-information, we study the optimal sampling strategy that minimizes the mean square estimation error subject to a sampling frequency constraint. We prove that the optimal sampling strategy is a threshold policy, and find the optimal threshold. This threshold is determined by the sampling frequency constraint and how much the Wiener process varies during the channel delay. An interesting consequence is that even in the absence of the sampling frequency constraint, the optimal strategy is not zero-wait sampling in which a new sample is taken once the previous sample is delivered; rather, it is optimal to wait for a non-zero amount of time after the previous sample is delivered, and then take the next sample. Further, if the sampling times are independent of the observed Wiener process, the optimal sampling problem reduces to an age-of-information optimization problem that has been recently solved. Our comparisons show that the estimation error of the optimal sampling policy is much smaller than those of age-optimal sampling, zero-wait sampling, and classic uniform sampling.

cs.IT↗

arXiv2017

Joint source-channel coding with feedback

This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce $k$ source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint.

cs.IT↗

arXiv2016

Strong data-processing inequalities for channels and Bayesian networks

The data-processing inequality, that is, $I(U;Y) \le I(U;X)$ for a Markov chain $U \to X \to Y$, has been the method of choice for proving impossibility (converse) results in information theory and many other disciplines. Various channel-dependent improvements (called strong data-processing inequalities, or SDPIs) of this inequality have been proposed both classically and more recently. In this note we first survey known results relating various notions of contraction for a single channel. Then we consider the basic extension: given SDPI for each constituent channel in a Bayesian network, how to produce an end-to-end SDPI? Our approach is based on the (extract of the) Evans-Schulman method, which is demonstrated for three different kinds of SDPIs, namely, the usual Ahslwede-Gács type contraction coefficients (mutual information), Dobrushin's contraction coefficients (total variation), and finally the $F_I$-curve (the best possible non-linear SDPI for a given channel). Resulting bounds on the contraction coefficients are interpreted as probability of site percolation. As an example, we demonstrate how to obtain SDPI for an $n$-letter memoryless channel with feedback given an SDPI for $n=1$. Finally, we discuss a simple observation on the equivalence of a linear SDPI and comparison to an erasure channel (in the sense of "less noisy" order). This leads to a simple proof of a curious inequality of Samorodnitsky (2015), and sheds light on how information spreads in the subsets of inputs of a memoryless channel.

cs.IT↗