arXiv subjects

S. Hamed Hassani

Publications and source records attributed to S. Hamed Hassani.

At least 19 recordsLinked to original sources

arXiv2020

Binary Linear Codes with Optimal Scaling: Polar Codes with Large Kernels

We prove that, for the binary erasure channel (BEC), the polar-coding paradigm gives rise to codes that not only approach the Shannon limit but do so under the best possible scaling of their block length as a~function of the gap to capacity. This result exhibits the first known family of binary codes that attain both optimal scaling and quasi-linear complexity of encoding and decoding. Our proof is based on the construction and analysis of binary polar codes with large kernels. When communicating reliably at rates within $\varepsilon > 0$ of capacity, the code length $n$ often scales as $O(1/\varepsilon^μ)$, where the constant $μ$ is called the scaling exponent. It is known that the optimal scaling exponent is $μ=2$, and it is achieved by random linear codes. The scaling exponent of conventional polar codes (based on the $2\times 2$ kernel) on the BEC is $μ=3.63$. This falls far short of the optimal scaling guaranteed by random codes. Our main contribution is a rigorous proof of the following result: for the BEC, there exist $\ell\times\ell$ binary kernels, such that polar codes constructed from these kernels achieve scaling exponent $μ(\ell)$ that tends to the optimal value of $2$ as $\ell$ grows. We furthermore characterize precisely how large $\ell$ needs to be as a function of the gap between $μ(\ell)$ and $2$. The resulting binary codes maintain the recursive structure of conventional polar codes, and thereby achieve construction complexity $O(n)$ and encoding/decoding complexity $O(n\log n)$.

cs.IT↗

arXiv2019

A New Coding Paradigm for the Primitive Relay Channel

We consider the primitive relay channel, where the source sends a message to the relay and to the destination, and the relay helps the communication by transmitting an additional message to the destination via a separate channel. Two well-known coding techniques have been introduced for this setting: decode-and-forward and compress-and-forward. In decode-and-forward, the relay completely decodes the message and sends some information to the destination; in compress-and-forward, the relay does not decode, and it sends a compressed version of the received signal to the destination using Wyner-Ziv coding. In this paper, we present a novel coding paradigm that provides an improved achievable rate for the primitive relay channel. The idea is to combine compress-and-forward and decode-and-forward via a chaining construction. We transmit over pairs of blocks: in the first block, we use compress-and-forward; and in the second block, we use decode-and-forward. More specifically, in the first block, the relay does not decode, it compresses the received signal via Wyner-Ziv, and it sends only part of the compression to the destination. In the second block, the relay completely decodes the message, it sends some information to the destination, and it also sends the remaining part of the compression coming from the first block. By doing so, we are able to strictly outperform both compress-and-forward and decode-and-forward. Note that the proposed coding scheme can be implemented with polar codes. As such, it has the typical attractive properties of polar coding schemes, namely, quasi-linear encoding and decoding complexity, and error probability that decays at super-polynomial speed. As a running example, we take into account the special case of the erasure relay channel, and we provide a comparison between the rates achievable by our proposed scheme and the existing upper and lower bounds.

cs.IT↗

arXiv2018

How to Achieve the Capacity of Asymmetric Channels

We survey coding techniques that enable reliable transmission at rates that approach the capacity of an arbitrary discrete memoryless channel. In particular, we take the point of view of modern coding theory and discuss how recent advances in coding for symmetric channels help provide more efficient solutions for the asymmetric case. We consider, in more detail, three basic coding paradigms. The first one is Gallager's scheme that consists of concatenating a linear code with a non-linear mapping so that the input distribution can be appropriately shaped. We explicitly show that both polar codes and spatially coupled codes can be employed in this scenario. Furthermore, we derive a scaling law between the gap to capacity, the cardinality of the input and output alphabets, and the required size of the mapper. The second one is an integrated scheme in which the code is used both for source coding, in order to create codewords distributed according to the capacity-achieving input distribution, and for channel coding, in order to provide error protection. Such a technique has been recently introduced by Honda and Yamamoto in the context of polar codes, and we show how to apply it also to the design of sparse graph codes. The third paradigm is based on an idea of Böcherer and Mathar, and separates the two tasks of source coding and channel coding by a chaining construction that binds together several codewords. We present conditions for the source code and the channel code, and we describe how to combine any source code with any channel code that fulfill those conditions, in order to provide capacity-achieving schemes for asymmetric channels. In particular, we show that polar codes, spatially coupled codes, and homophonic codes are suitable as basic building blocks of the proposed coding strategy.

cs.IT↗

arXiv2017

Partitioned List Decoding of Polar Codes: Analysis and Improvement of Finite Length Performance

Polar codes represent one of the major recent breakthroughs in coding theory and, because of their attractive features, they have been selected for the incoming 5G standard. As such, a lot of attention has been devoted to the development of decoding algorithms with good error performance and efficient hardware implementation. One of the leading candidates in this regard is represented by successive-cancellation list (SCL) decoding. However, its hardware implementation requires a large amount of memory. Recently, a partitioned SCL (PSCL) decoder has been proposed to significantly reduce the memory consumption. In this paper, we examine the paradigm of PSCL decoding from both theoretical and practical standpoints: (i) by changing the construction of the code, we are able to improve the performance at no additional computational, latency or memory cost, (ii) we present an optimal scheme to allocate cyclic redundancy checks (CRCs), and (iii) we provide an upper bound on the list size that allows MAP performance.

cs.IT↗

arXiv2017

Construction of Polar Codes with Sublinear Complexity

Consider the problem of constructing a polar code of block length $N$ for the transmission over a given channel $W$. Typically this requires to compute the reliability of all the $N$ synthetic channels and then to include those that are sufficiently reliable. However, we know from [1], [2] that there is a partial order among the synthetic channels. Hence, it is natural to ask whether we can exploit it to reduce the computational burden of the construction problem. We show that, if we take advantage of the partial order [1], [2], we can construct a polar code by computing the reliability of roughly a fraction $1/\log^{3/2} N$ of the synthetic channels. In particular, we prove that $N/\log^{3/2} N$ is a lower bound on the number of synthetic channels to be considered and such a bound is tight up to a multiplicative factor $\log\log N$. This set of roughly $N/\log^{3/2} N$ synthetic channels is universal, in the sense that it allows one to construct polar codes for any $W$, and it can be identified by solving a maximum matching problem on a bipartite graph. Our proof technique consists of reducing the construction problem to the problem of computing the maximum cardinality of an antichain for a suitable partially ordered set. As such, this method is general and it can be used to further improve the complexity of the construction problem in case a new partial order on the synthetic channels of polar codes is discovered.

cs.IT↗

arXiv2017

Uniform Deviation Bounds for Unbounded Loss Functions like k-Means

Uniform deviation bounds limit the difference between a model's expected loss and its loss on an empirical sample uniformly for all models in a learning problem. As such, they are a critical component to empirical risk minimization. In this paper, we provide a novel framework to obtain uniform deviation bounds for loss functions which are *unbounded*. In our main application, this allows us to obtain bounds for $k$-Means clustering under weak assumptions on the underlying distribution. If the fourth moment is bounded, we prove a rate of $\mathcal{O}\left(m^{-\frac12}\right)$ compared to the previously known $\mathcal{O}\left(m^{-\frac14}\right)$ rate. Furthermore, we show that the rate also depends on the kurtosis - the normalized fourth moment which measures the "tailedness" of a distribution. We further provide improved rates under progressively stronger assumptions, namely, bounded higher moments, subgaussianity and bounded support.

stat.ML↗

arXiv2017

Capacity-Achieving Rate-Compatible Polar Codes for General Channels

We present a rate-compatible polar coding scheme that achieves the capacity of any family of channels. Our solution generalizes the previous results [1], [2] that provide capacity-achieving rate-compatible polar codes for a degraded family of channels. The motivation for our extension comes from the fact that in many practical scenarios, e.g., MIMO systems and non-Gaussian interference, the channels cannot be ordered by degradation. The main technical contribution of this paper consists in removing the degradation condition. To do so, we exploit the ideas coming from the construction of universal polar codes. Our scheme possesses the usual attractive features of polar codes: low complexity code construction, encoding, and decoding; super-polynomial scaling of the error probability with the block length; and absence of error floors. On the negative side, the scaling of the gap to capacity with the block length is slower than in standard polar codes, and we prove an upper bound on the scaling exponent.

cs.IT↗

arXiv2016

Unified Scaling of Polar Codes: Error Exponent, Scaling Exponent, Moderate Deviations, and Error Floors

Consider the transmission of a polar code of block length $N$ and rate $R$ over a binary memoryless symmetric channel $W$ and let $P_e$ be the block error probability under successive cancellation decoding. In this paper, we develop new bounds that characterize the relationship of the parameters $R$, $N$, $P_e$, and the quality of the channel $W$ quantified by its capacity $I(W)$ and its Bhattacharyya parameter $Z(W)$. In previous work, two main regimes were studied. In the error exponent regime, the channel $W$ and the rate $R<I(W)$ are fixed, and it was proved that the error probability $P_e$ scales roughly as $2^{-\sqrt{N}}$. In the scaling exponent approach, the channel $W$ and the error probability $P_e$ are fixed and it was proved that the gap to capacity $I(W)-R$ scales as $N^{-1/μ}$. Here, $μ$ is called scaling exponent and this scaling exponent depends on the channel $W$. A heuristic computation for the binary erasure channel (BEC) gives $μ=3.627$ and it was shown that, for any channel $W$, $3.579 \le μ\le 5.702$. Our contributions are as follows. First, we provide the tighter upper bound $μ\le 4.714$ valid for any $W$. With the same technique, we obtain $μ\le 3.639$ for the case of the BEC, which approaches very closely its heuristically derived value. Second, we develop a trade-off between the gap to capacity $I(W)-R$ and the error probability $P_e$ as functions of the block length $N$. In other words, we consider a moderate deviations regime in which we study how fast both quantities, as functions of the block length $N$, simultaneously go to $0$. Third, we prove that polar codes are not affected by error floors. To do so, we fix a polar code of block length $N$ and rate $R$. Then, we vary the channel $W$ and we show that the error probability $P_e$ scales as the Bhattacharyya parameter $Z(W)$ raised to a power that scales roughly like $\sqrt{N}$.

cs.IT↗

arXiv2016

Near-optimal Bayesian Active Learning with Correlated and Noisy Tests

We consider the Bayesian active learning and experimental design problem, where the goal is to learn the value of some unknown target variable through a sequence of informative, noisy tests. In contrast to prior work, we focus on the challenging, yet practically relevant setting where test outcomes can be conditionally dependent given the hidden target variable. Under such assumptions, common heuristics, such as greedily performing tests that maximize the reduction in uncertainty of the target, often perform poorly. In this paper, we propose ECED, a novel, computationally efficient active learning algorithm, and prove strong theoretical guarantees that hold with correlated, noisy tests. Rather than directly optimizing the prediction error, at each step, ECED picks the test that maximizes the gain in a surrogate objective, which takes into account the dependencies between tests. Our analysis relies on an information-theoretic auxiliary function to track the progress of ECED, and utilizes adaptive submodularity to attain the near-optimal bound. We demonstrate strong empirical performance of ECED on two problem instances, including a Bayesian experimental design task intended to distinguish among economic theories of how people make risky decisions, and an active preference learning task via pairwise comparisons.

cs.LG↗

arXiv2014

Alignment of Polarized Sets

Arıkan's polar coding technique is based on the idea of synthesizing $n$ channels from the $n$ instances of the physical channel by a simple linear encoding transformation. Each synthesized channel corresponds to a particular input to the encoder. For large $n$, the synthesized channels become either essentially noiseless or almost perfectly noisy, but in total carry as much information as the original $n$ channels. Capacity can therefore be achieved by transmitting messages over the essentially noiseless synthesized channels. Unfortunately, the set of inputs corresponding to reliable synthesized channels is poorly understood, in particular how the set depends on the underlying physical channel. In this work, we present two analytic conditions sufficient to determine if the reliable inputs corresponding to different discrete memoryless channels are aligned or not, i.e. if one set is contained in the other. Understanding the alignment of the polarized sets is important as it is directly related to universality properties of the induced polar codes, which are essential in particular for network coding problems. We demonstrate the performance of our conditions on a few examples for wiretap and broadcast channels. Finally we show that these conditions imply that the simple quantum polar coding scheme of Renes et al. [Phys. Rev. Lett. 109, 050504 (2012)] requires entanglement assistance for general channels, but also show such assistance to be unnecessary in many cases of interest.

cs.IT↗

arXiv2014

Achieving Marton's Region for Broadcast Channels Using Polar Codes

This paper presents polar coding schemes for the 2-user discrete memoryless broadcast channel (DM-BC) which achieve Marton's region with both common and private messages. This is the best achievable rate region known to date, and it is tight for all classes of 2-user DM-BCs whose capacity regions are known. To accomplish this task, we first construct polar codes for both the superposition as well as the binning strategy. By combining these two schemes, we obtain Marton's region with private messages only. Finally, we show how to handle the case of common information. The proposed coding schemes possess the usual advantages of polar codes, i.e., they have low encoding and decoding complexity and a super-polynomial decay rate of the error probability. We follow the lead of Goela, Abbe, and Gastpar, who recently introduced polar codes emulating the superposition and binning schemes. In order to align the polar indices, for both schemes, their solution involves some degradedness constraints that are assumed to hold between the auxiliary random variables and the channel outputs. To remove these constraints, we consider the transmission of $k$ blocks and employ a chaining construction that guarantees the proper alignment of the polarized indices. The techniques described in this work are quite general, and they can be adopted to many other multi-terminal scenarios whenever there polar indices need to be aligned.

cs.IT↗

arXiv2014

Scaling Exponent of List Decoders with Applications to Polar Codes

Motivated by the significant performance gains which polar codes experience under successive cancellation list decoding, their scaling exponent is studied as a function of the list size. In particular, the error probability is fixed and the trade-off between block length and back-off from capacity is analyzed. A lower bound is provided on the error probability under $\rm MAP$ decoding with list size $L$ for any binary-input memoryless output-symmetric channel and for any class of linear codes such that their minimum distance is unbounded as the block length grows large. Then, it is shown that under $\rm MAP$ decoding, although the introduction of a list can significantly improve the involved constants, the scaling exponent itself, i.e., the speed at which capacity is approached, stays unaffected for any finite list size. In particular, this result applies to polar codes, since their minimum distance tends to infinity as the block length increases. A similar result is proved for genie-aided successive cancellation decoding when transmission takes place over the binary erasure channel, namely, the scaling exponent remains constant for any fixed number of helps from the genie. Note that since genie-aided successive cancellation decoding might be strictly worse than successive cancellation list decoding, the problem of establishing the scaling exponent of the latter remains open.

cs.IT↗

arXiv2014

From Polar to Reed-Muller Codes: a Technique to Improve the Finite-Length Performance

We explore the relationship between polar and RM codes and we describe a coding scheme which improves upon the performance of the standard polar code at practical block lengths. Our starting point is the experimental observation that RM codes have a smaller error probability than polar codes under MAP decoding. This motivates us to introduce a family of codes that "interpolates" between RM and polar codes, call this family ${\mathcal C}_{\rm inter} = \{C_α : α\in [0, 1]\}$, where $C_α \big |_{α= 1}$ is the original polar code, and $C_α \big |_{α= 0}$ is an RM code. Based on numerical observations, we remark that the error probability under MAP decoding is an increasing function of $α$. MAP decoding has in general exponential complexity, but empirically the performance of polar codes at finite block lengths is boosted by moving along the family ${\mathcal C}_{\rm inter}$ even under low-complexity decoding schemes such as, for instance, belief propagation or successive cancellation list decoder. We demonstrate the performance gain via numerical simulations for transmission over the erasure channel as well as the Gaussian channel.

cs.IT↗

arXiv2014

Finite-Length Scaling of Polar Codes

Consider a binary-input memoryless output-symmetric channel $W$. Such a channel has a capacity, call it $I(W)$, and for any $R 0$, then the required block-length $N$ scales in terms of the rate $R < I(W)$ as $N \geq \fracα{(I(W)-R)^{\underlineμ}}$, where $α$ is a positive constant that depends on $P_{\rm e}$ and $I(W)$, and $\underlineμ = 3.579$. Also, we show that with the same requirement on the sum of Bhattacharyya parameters, the block-length scales in terms of the rate like $N \leq \fracβ{(I(W)-R)^{\overlineμ}}$, where $β$ is a constant that depends on $P_{\rm e}$ and $I(W)$, and $\overlineμ=6$.

cs.IT↗

arXiv2013

Universal Polar Codes

Polar codes, invented by Arikan in 2009, are known to achieve the capacity of any binary-input memoryless output-symmetric channel. One of the few drawbacks of the original polar code construction is that it is not universal. This means that the code has to be tailored to the channel if we want to transmit close to capacity. We present two "polar-like" schemes which are capable of achieving the compound capacity of the whole class of binary-input memoryless output-symmetric channels with low complexity. Roughly speaking, for the first scheme we stack up $N$ polar blocks of length $N$ on top of each other but shift them with respect to each other so that they form a "staircase." Coding then across the columns of this staircase with a standard Reed-Solomon code, we can achieve the compound capacity using a standard successive decoder to process the rows (the polar codes) and in addition a standard Reed-Solomon erasure decoder to process the columns. Compared to standard polar codes this scheme has essentially the same complexity per bit but a block length which is larger by a factor $O(N \log_2(N)/ε)$, where $ε$ is the gap to capacity. For the second scheme we first show how to construct a true polar code which achieves the compound capacity for a finite number of channels. We achieve this by introducing special "polarization" steps which "align" the good indices for the various channels. We then show how to exploit the compactness of the space of binary-input memoryless output-symmetric channels to reduce the compound capacity problem for this class to a compound capacity problem for a finite set of channels. This scheme is similar in spirit to standard polar codes, but the price for universality is a considerably larger blocklength. We close with what we consider to be some interesting open problems.

cs.IT↗

arXiv2013

The Least Degraded and the Least Upgraded Channel with respect to a Channel Family

Given a family of binary-input memoryless output-symmetric (BMS) channels having a fixed capacity, we derive the BMS channel having the highest (resp. lowest) capacity among all channels that are degraded (resp. upgraded) with respect to the whole family. We give an explicit characterization of this channel as well as an explicit formula for the capacity of this channel.

cs.IT↗

arXiv2013

The Space of Solutions of Coupled XORSAT Formulae

The XOR-satisfiability (XORSAT) problem deals with a system of $n$ Boolean variables and $m$ clauses. Each clause is a linear Boolean equation (XOR) of a subset of the variables. A $K$-clause is a clause involving $K$ distinct variables. In the random $K$-XORSAT problem a formula is created by choosing $m$ $K$-clauses uniformly at random from the set of all possible clauses on $n$ variables. The set of solutions of a random formula exhibits various geometrical transitions as the ratio $\frac{m}{n}$ varies. We consider a {\em coupled} $K$-XORSAT ensemble, consisting of a chain of random XORSAT models that are spatially coupled across a finite window along the chain direction. We observe that the threshold saturation phenomenon takes place for this ensemble and we characterize various properties of the space of solutions of such coupled formulae.

cond-mat.dis-nn↗

arXiv2012

On the Construction of Polar Codes

We consider the problem of efficiently constructing polar codes over binary memoryless symmetric (BMS) channels. The complexity of designing polar codes via an exact evaluation of the polarized channels to find which ones are "good" appears to be exponential in the block length. In \cite{TV11}, Tal and Vardy show that if instead the evaluation if performed approximately, the construction has only linear complexity. In this paper, we follow this approach and present a framework where the algorithms of \cite{TV11} and new related algorithms can be analyzed for complexity and accuracy. We provide numerical and analytical results on the efficiency of such algorithms, in particular we show that one can find all the "good" channels (except a vanishing fraction) with almost linear complexity in block-length (except a polylogarithmic factor).

cs.IT↗