Searcharxiv⌕ Search

arXiv subjects

Neri Merhav

Publications and source records attributed to Neri Merhav.

At least 109 records · Page 6Linked to original sources

Universal Decoding for Gaussian Intersymbol Interference Channels

A universal decoding procedure is proposed for the intersymbol interference (ISI) Gaussian channels. The universality of the proposed decoder is in the sense of being independent of the various channel parameters, and at the same time, attaining the same random coding error exponent as the optimal maximum-likelihood (ML) decoder, which utilizes full knowledge of these unknown parameters. The proposed decoding rule can be regarded as a frequency domain version of the universal maximum mutual information (MMI) decoder. Contrary to previously suggested universal decoders for ISI channels, our proposed decoding metric can easily be evaluated.

cs.IT↗

List decoding - random coding exponents and expurgated exponents

Some new results are derived concerning random coding error exponents and expurgated exponents for list decoding with a deterministic list size $L$. Two asymptotic regimes are considered, the fixed list-size regime, where $L$ is fixed independently of the block length $n$, and the exponential list-size, where $L$ grows exponentially with $n$. We first derive a general upper bound on the list-decoding average error probability, which is suitable for both regimes. This bound leads to more specific bounds in the two regimes. In the fixed list-size regime, the bound is related to known bounds and we establish its exponential tightness. In the exponential list-size regime, we establish the achievability of the well known sphere packing lower bound. Relations to guessing exponents are also provided. An immediate byproduct of our analysis in both regimes is the universality of the maximum mutual information (MMI) list decoder in the error exponent sense. Finally, we consider expurgated bounds at low rates, both using Gallager's approach and the Csiszár-Körner-Marton approach, which is, in general better (at least for $L=1$). The latter expurgated bound, which involves the notion of {\it multi-information}, is also modified to apply to continuous alphabet channels, and in particular, to the Gaussian memoryless channel, where the expression of the expurgated bound becomes quite explicit.

cs.IT↗

Zero-Delay and Causal Secure Source Coding

We investigate the combination between causal/zero-delay source coding and information-theoretic secrecy. Two source coding models with secrecy constraints are considered. We start by considering zero-delay perfectly secret lossless transmission of a memoryless source. We derive bounds on the key rate and coding rate needed for perfect zero-delay secrecy. In this setting, we consider two models which differ by the ability of the eavesdropper to parse the bit-stream passing from the encoder to the legitimate decoder into separate messages. We also consider causal source coding with a fidelity criterion and side information at the decoder and the eavesdropper. Unlike the zero-delay setting where variable-length coding is traditionally used but might leak information on the source through the length of the codewords, in this setting, since delay is allowed, block coding is possible. We show that in this setting, separation of encryption and causal source coding is optimal.

cs.IT↗

Asymptotically optimal decision rules for joint detection and source coding

The problem of joint detection and lossless source coding is considered. We derive asymptotically optimal decision rules for deciding whether or not a sequence of observations has emerged from a desired information source, and to compress it if has. In particular, our decision rules asymptotically minimize the cost of compression in the case that the data has been classified as `desirable', subject to given constraints on the two kinds of the probability of error. In another version of this performance criterion, the constraint on the false alarm probability is replaced by the a constraint on the cost of compression in the false alarm event. We then analyze the asymptotic performance of these decision rules and demonstrate that they may exhibit certain phase transitions. We also derive universal decision rules for the case where the underlying sources (under either hypothesis or both) are unknown, and training sequences from each source may or may not be available. Finally, we discuss how our framework can be extended in several directions.

cs.IT↗

Codeword or noise? Exact random coding exponents for slotted asynchronism

We consider the problem of slotted asynchronous coded communication, where in each time frame (slot), the transmitter is either silent or transmits a codeword from a given (randomly selected) codebook. The task of the decoder is to decide whether transmission has taken place, and if so, to decode the message. We derive the optimum detection/decoding rule in the sense of the best trade-off among the probabilities of decoding error, false alarm, and misdetection. For this detection/decoding rule, we then derive single-letter characterizations of the exact exponential rates of these three probabilities for the average code in the ensemble.

cs.IT↗

Statistical Physics: a Short Course for Electrical Engineering Students

This is a set of lecture notes of a course on statistical physics and thermodynamics, which is oriented, to a certain extent, towards electrical engineering students. The main body of the lectures is devoted to statistical physics, whereas much less emphasis is given to the thermodynamics part. In particular, the idea is to let the most important results of thermodynamics (most notably, the laws of thermodynamics) to be obtained as conclusions from the derivations in statistical physics. Beyond the variety of central topics in statistical physics that are important to the general scientific education of the EE student, special emphasis is devoted to subjects that are vital to the engineering education concretely. These include, first of all, quantum statistics, like the Fermi-Dirac distribution, as well as diffusion processes, which are both fundamental for deep understanding of semiconductor devices. Another important issue for the EE student is to understand mechanisms of noise generation and stochastic dynamics in physical systems, most notably, in electric circuitry. Accordingly, the fluctuation-dissipation theorem of statistical mechanics, which is the theoretical basis for understanding thermal noise processes in systems, is presented from a signals--and--systems point of view, in a way that would hopefully be understandable and useful for an engineering student, and well connected to other courses in the electrcial engineering curriculum like courses on random priocesses. The quantum regime, in this context, is important too and hence provided as well. Finally, we touch very briefly upon some relationships between statistical mechanics and information theory, which is the theoretical basis for communications engineering, and demonstrate how statistical-mechanical approach can be useful in order for the study of information-theoretic problems.

cond-mat.stat-mech↗

Analysis of Mismatched Estimation Errors Using Gradients of Partition Functions

We consider the problem of signal estimation (denoising) from a statistical-mechanical perspective, in continuation to a recent work on the analysis of mean-square error (MSE) estimation using a direct relationship between optimum estimation and certain partition functions. The paper consists of essentially two parts. In the first part, using the aforementioned relationship, we derive single-letter expressions of the mismatched MSE of a codeword (from a randomly selected code), corrupted by a Gaussian vector channel. In the second part, we provide several examples to demonstrate phase transitions in the behavior of the MSE. These examples enable us to understand more deeply and to gather intuition regarding the roles of the real and the mismatched probability measures in creating these phase transitions.

cs.IT↗

Erasure/list exponents for Slepian-Wolf decoding

We analyze random coding error exponents associated with erasure/list Slepian-Wolf decoding using two different methods and then compare the resulting bounds. The first method follows the well known techniques of Gallager and Forney and the second method is based on a technique of distance enumeration, or more generally, type class enumeration, which is rooted in the statistical mechanics of a disordered system that is related to the random energy model (REM). The second method is guaranteed to yield exponent functions which are at least as tight as those of the first method, and it is demonstrated that for certain combinations of coding rates and thresholds, the bounds of the second method are strictly tighter than those of the first method, by an arbitrarily large factor. In fact, the second method may even yield an infinite exponent at regions where the first method gives finite values. We also discuss the option of variable-rate Slepian-Wolf encoding and demonstrate how it can improve on the resulting exponents.

cs.IT↗

On the data processing theorem in the semi-deterministic setting

Data processing lower bounds on the expected distortion are derived in the finite-alphabet semi-deterministic setting, where the source produces a deterministic, individual sequence, but the channel model is probabilistic, and the decoder is subjected to various kinds of limitations, e.g., decoders implementable by finite-state machines, with or without counters, and with or without a restriction of common reconstruction with high probability. Some of our bounds are given in terms of the Lempel-Ziv complexity of the source sequence or the reproduction sequence. We also demonstrate how some analogous results can be obtained for classes of linear encoders and linear decoders in the continuous alphabet case.

cs.IT↗

Another look at expurgated bounds and their statistical-mechanical interpretation

We revisit the derivation of expurgated error exponents using a method of type class enumeration, which is inspired by statistical-mechanical methods, and which has already been used in the derivation of random coding exponents in several other scenarios. We compare our version of the expurgated bound to both the one by Gallager and the one by Csiszar, Korner and Marton (CKM). For expurgated ensembles of fixed composition codes over finite alphabets, our basic expurgated bound coincides with the CKM expurgated bound, which is in general tighter than Gallager's bound, but with equality for the optimum type class of codewords. Our method, however, extends beyond fixed composition codes and beyond finite alphabets, where it is natural to impose input constraints (e.g., power limitation). In such cases, the CKM expurgated bound may not apply directly, and our bound is in general tighter than Gallager's bound. In addition, while both the CKM and the Gallager expurgated bounds are based on Bhattacharyya bound for bounding the pairwise error probabilities, our bound allows the more general Chernoff distance measure, thus giving rise to additional improvement using the Chernoff parameter as a degree of freedom to be optimized.

cs.IT↗

Zero-Delay and Causal Single-User and Multi-User Lossy Source Coding with Decoder Side Information

We consider zero-delay single-user and multi-user source coding with average distortion constraint and decoder side information. The zero-delay constraint translates into causal (sequential) encoder and decoder pairs as well as the use of instantaneous codes. For the single-user setting, we show that optimal performance is attained by time sharing at most two scalar encoder-decoder pairs, that use zero-error side information codes. Side information lookahead is shown to useless in this setting. We show that the restriction to causal encoding functions is the one that causes the performance degradation, compared to unrestricted systems, and not the sequential decoders or instantaneous codes. Furthermore, we show that even without delay constraints, if either the encoder or decoder are restricted a-priori to be scalar, the performance loss cannot be compensated by the other component, which can be scalar as well without further loss. Finally, we show that the multi-terminal source coding problem can be solved in the zero-delay regime and the rate-distortion region is given.

cs.IT↗

Exponential error bounds on parameter modulation-estimation for discrete memoryless channels

We consider the problem of modulation and estimation of a random parameter $U$ to be conveyed across a discrete memoryless channel. Upper and lower bounds are derived for the best achievable exponential decay rate of a general moment of the estimation error, $\bE|\hat{U}-U|^ρ$, $ρ\ge 0$, when both the modulator and the estimator are subjected to optimization. These exponential error bounds turn out to be intimately related to error exponents of channel coding and to channel capacity. While in general, there is some gap between the upper and the lower bound, they asymptotically coincide both for very small and for very large values of the moment power $ρ$. This means that our achievability scheme, which is based on simple quantization of $U$ followed by channel coding, is nearly optimum in both limits. Some additional properties of the bounds are discussed and demonstrated, and finally, an extension to the case of a multidimensional parameter vector is outlined, with the principal conclusion that our upper and lower bound asymptotically coincide also for a high dimensionality.

cs.IT↗

Universal decoding for arbitrary channels relative to a given class of decoding metrics

We consider the problem of universal decoding for arbitrary unknown channels in the random coding regime. For a given random coding distribution and a given class of metric decoders, we propose a generic universal decoder whose average error probability is, within a sub-exponential multiplicative factor, no larger than that of the best decoder within this class of decoders. Since the optimum, maximum likelihood (ML) decoder of the underlying channel is not necessarily assumed to belong to the given class of decoders, this setting suggests a common generalized framework for: (i) mismatched decoding, (ii) universal decoding for a given family of channels, and (iii) universal coding and decoding for deterministic channels using the individual-sequence approach. The proof of our universality result is fairly simple, and it is demonstrated how some earlier results on universal decoding are obtained as special cases. We also demonstrate how our method extends to more complicated scenarios, like incorporation of noiseless feedback, and the multiple access channel.

cs.IT↗

Average redundancy of the Shannon code for Markov sources

It is known that for memoryless sources, the average and maximal redundancy of fixed-to-variable length codes, such as the Shannon and Huffman codes, exhibit two modes of behavior for long blocks. It either converges to a limit or it has an oscillatory pattern, depending on the irrationality or rationality, respectively, of certain parameters that depend on the source. In this paper, we extend these findings, concerning the Shannon code, to the case of a Markov source, which is considerably more involved. While this dichotomy, of convergent vs. oscillatory behavior, is well known in other contexts (including renewal theory, ergodic theory, local limit theorems and large deviations of discrete distributions), in information theory (e.g., in redundancy analysis) it was recognized relatively recently. To the best of our knowledge, no results of this type were reported thus far for Markov sources. We provide a precise characterization of the convergent vs. oscillatory behavior of the Shannon code redundancy for a class of irreducible, periodic and aperiodic, Markov sources. These findings are obtained by analytic methods, such as Fourier/Fejer series analysis and spectral analysis of matrices.

cs.IT↗

On Real-Time and Causal Secure Source Coding

We investigate two source coding problems with secrecy constraints. In the first problem we consider real--time fully secure transmission of a memoryless source. We show that although classical variable--rate coding is not an option since the lengths of the codewords leak information on the source, the key rate can be as low as the average Huffman codeword length of the source. In the second problem we consider causal source coding with a fidelity criterion and side information at the decoder and the eavesdropper. We show that when the eavesdropper has degraded side information, it is optimal to first use a causal rate distortion code and then encrypt its output with a key.

cs.IT↗

On optimum parameter modulation-estimation from a large deviations perspective

We consider the problem of jointly optimum modulation and estimation of a real-valued random parameter, conveyed over an additive white Gaussian noise (AWGN) channel, where the performance metric is the large deviations behavior of the estimator, namely, the exponential decay rate (as a function of the observation time) of the probability that the estimation error would exceed a certain threshold. Our basic result is in providing an exact characterization of the fastest achievable exponential decay rate, among all possible modulator-estimator (transmitter-receiver) pairs, where the modulator is limited only in the signal power, but not in bandwidth. This exponential rate turns out to be given by the reliability function of the AWGN channel. We also discuss several ways to achieve this optimum performance, and one of them is based on quantization of the parameter, followed by optimum channel coding and modulation, which gives rise to a separation-based transmitter, if one views this setting from the perspective of joint source-channel coding. This is in spite of the fact that, in general, when error exponents are considered, the source-channel separation theorem does not hold true. We also discuss several observations, modifications and extensions of this result in several directions, including other channels, and the case of multidimensional parameter vectors. One of our findings concerning the latter, is that there is an abrupt threshold effect in the dimensionality of the parameter vector: below a certain critical dimension, the probability of excess estimation error may still decay exponentially, but beyond this value, it must converge to unity.

cs.IT↗

Perfectly secure encryption of individual sequences

In analogy to the well-known notion of finite--state compressibility of individual sequences, due to Lempel and Ziv, we define a similar notion of "finite-state encryptability" of an individual plaintext sequence, as the minimum asymptotic key rate that must be consumed by finite-state encrypters so as to guarantee perfect secrecy in a well-defined sense. Our main basic result is that the finite-state encryptability is equal to the finite-state compressibility for every individual sequence. This is in parallelism to Shannon's classical probabilistic counterpart result, asserting that the minimum required key rate is equal to the entropy rate of the source. However, the redundancy, defined as the gap between the upper bound (direct part) and the lower bound (converse part) in the encryption problem, turns out to decay at a different rate (in fact, much slower) than the analogous redundancy associated with the compression problem. We also extend our main theorem in several directions, allowing: (i) availability of side information (SI) at the encrypter/decrypter/eavesdropper, (ii) lossy reconstruction at the decrypter, and (iii) the combination of both lossy reconstruction and SI, in the spirit of the Wyner--Ziv problem.

cs.IT↗

Data processing inequalities based on a certain structured class of information measures with application to estimation theory

We study data processing inequalities that are derived from a certain class of generalized information measures, where a series of convex functions and multiplicative likelihood ratios are nested alternately. While these information measures can be viewed as a special case of the most general Zakai-Ziv generalized information measure, this special nested structure calls for attention and motivates our study. Specifically, a certain choice of the convex functions leads to an information measure that extends the notion of the Bhattacharyya distance (or the Chernoff divergence): While the ordinary Bhattacharyya distance is based on the (weighted) geometric mean of two replicas of the channel's conditional distribution, the more general information measure allows an arbitrary number of such replicas. We apply the data processing inequality induced by this information measure to a detailed study of lower bounds of parameter estimation under additive white Gaussian noise (AWGN) and show that in certain cases, tighter bounds can be obtained by using more than two replicas. While the resulting lower bound may not compete favorably with the best bounds available for the ordinary AWGN channel, the advantage of the new lower bound, relative to the other bounds, becomes significant in the presence of channel uncertainty, like unknown fading. This different behavior in the presence of channel uncertainty is explained by the convexity property of the information measure.

cs.IT↗