arXiv subjects

Neri Merhav

Publications and source records attributed to Neri Merhav.

At least 55 records · Page 3Linked to original sources

arXiv2020

Error Exponents in the Bee Identification Problem

We derive various error exponents in the bee identification problem under two different decoding rules. Under naïve decoding, which decodes each bee independently of the others, we analyze a general discrete memoryless channel and a relatively wide family of stochastic decoders. Upper and lower bounds to the random coding error exponent are derived and proved to be equal at relatively high coding rates. Then, we propose a lower bound on the error exponent of the typical random code, which improves upon the random coding exponent at low coding rates. We also derive a third bound, which is related to expurgated codes, which turns out to be strictly higher than the other bounds, also at relatively low rates. We show that the universal maximum mutual information decoder is optimal with respect to the typical random code and the expurgated code. Moving further, we derive error exponents under optimal decoding, the relatively wide family of symmetric channels, and the maximum likelihood decoder. We first propose a random coding lower bound, and then, an improved bound which stems from an expurgation process. We show numerically that our second bound strictly improves upon the random coding bound at an intermediate range of coding rates, where a bound derived in a previous work no longer holds.

cs.IT↗

arXiv2020

On More General Distributions of Random Binning for Slepian-Wolf Encoding

Traditionally, ensembles of Slepian-Wolf (SW) codes are defined such that every bin of each $n$-vector of each source is randomly drawn under the uniform distribution across the sets $\{0,1,\ldots,2^{nR_X}-1\}$ and $\{0,1,\ldots,2^{nR_Y}-1\}$, where $R_X$ and $R_Y$ are the coding rates of the two sources, $X$ and $Y$, respectively. In a few more recent works, where only one source, say, $X$, is compressed and the other one, $Y$, serves as side information available at the decoder, the scope is extended to variable-rate S-W (VRSW) codes, where the rate is allowed to depend on the type class of the source string, but still, the random-binning distribution is assumed uniform within the corresponding, type-dependent, bin index set. In this expository work, we investigate the role of the uniformity of the random binning distribution from the perspective of the trade-off between the reliability (defined in terms of the error exponent) and the compression performance (measured from the viewpoint of the source coding exponent). To this end, we study a much wider class of random-binning distributions, which includes the ensemble of VRSW codes as a special case, but it also goes considerably beyond. We first show that, with the exception of some pathological cases, the smaller ensemble, of VRSW codes, is as good as the larger ensemble in terms the trade-off between the error exponent and the source coding exponent. Notwithstanding this finding, the wider class of ensembles proposed is motivated in two ways. The first is that it outperforms VRSW codes in the above-mentioned pathological cases, and the second is that it allows robustness: in the event of a system failure that causes unavailability of the compressed bit-stream from one of the sources, it still allows reconstruction of the other source within some controllable distortion.

cs.IT↗

arXiv2020

Optimal Work Extraction and the Minimum Description Length Principle

We discuss work extraction from classical information engines (e.g., Szilárd) with $N$-particles, $q$ partitions, and initial arbitrary non-equilibrium states. In particular, we focus on their {\em optimal} behaviour, which includes the measurement of a set of quantities $Φ$ with a feedback protocol that extracts the maximal average amount of work. We show that the optimal non-equilibrium state to which the engine should be driven before the measurement is given by the normalised maximum-likelihood probability distribution of a statistical model that admits $Φ$ as sufficient statistics. Furthermore, we show that the minimax universal code redundancy $\mathcal{R}^*$ associated to this model, provides an upper bound to the work that the demon can extract on average from the cycle, in units of $k_{\rm B}T$. We also find that, in the limit of $N$ large, the maximum average extracted work cannot exceed $H[Φ]/2$, i.e. one half times the Shannon entropy of the measurement. Our results establish a connection between optimal work extraction in stochastic thermodynamics and optimal universal data compression, providing design principles for optimal information engines. In particular, they suggest that: (i) optimal coding is thermodynamically efficient, and (ii) it is essential to drive the system into a critical state in order to achieve optimal performance.

cond-mat.stat-mech↗

arXiv2020

Universal Decoding for Asynchronous Slepian-Wolf Encoding

We consider the problem of (almost) lossless source coding of two correlated memoryless sources using separate encoders and a joint decoder, that is, Slepian-Wolf (S-W) coding. In our setting, the encoding and decoding are asynchronous, i.e., there is a certain relative delay between the two sources. Neither the source parameters nor the relative delay are known to the encoders and the decoder. Since we assume that both encoders implement standard random binning, which does not require such knowledge anyway, the focus of this work is on the decoder. Our main contribution is in proposing a universal decoder, that independent of the unknown source parameters and the relative delay, and at the same time, is asymptotically as good as the optimal maximum a posteriori probability (MAP) decoder in the sense of the random coding error exponent achieved.Consequently, the achievable rate region is also the same as if the source parameters and the delay were known to the decoder.

cs.IT↗

arXiv2020

The MMI Decoder is Asymptotically Optimal for the Typical Random Code and for the Expurgated Code

We provide two results concerning the optimality of the maximum mutual information (MMI) decoder. First, we prove that the error exponents of the typical random codes under the optimal maximum likelihood (ML) decoder and the MMI decoder are equal. As a corollary to this result, we also show that the error exponents of the expurgated codes under the ML and the MMI decoders are equal. These results strengthen the well known result due to Csiszár and Körner, according to which, these decoders achieve equal random coding error exponents, since the error exponents of the typical random code and the expurgated code are strictly higher than the random coding error exponents, at least at low coding rates. While the universal optimality of the MMI decoder, in the random-coding error exponent sense, is easily proven by commuting the expectation over the channel noise and the expectation over the ensemble, when it comes to typical and expurgated exponents, this commutation can no longer be carried out. Therefore, the proof of the universal optimality of the MMI decoder must be completely different and it turns out to be highly non-trivial.

cs.IT↗

arXiv2020

Some Useful Integral Representations for Information-Theoretic Analyses

This work is an extension of our earlier article, where a well-known integral representation of the logarithmic function was explored, and was accompanied with demonstrations of its usefulness in obtaining compact, easily-calculable, exact formulas for quantities that involve expectations of the logarithm of a positive random variable. Here, in the same spirit, we derive an exact integral representation (in one or two dimensions) of the moment of a nonnegative random variable, or the sum of such independent random variables, where the moment order is a general positive noninteger real (also known as fractional moments). The proposed formula is applied to a variety of examples with an information-theoretic motivation, and it is shown how it facilitates their numerical evaluations. In particular, when applied to the calculation of a moment of the sum of a large number, $n$, of nonnegative random variables, it is clear that integration over one or two dimensions, as suggested by our proposed integral representation, is significantly easier than the alternative of integrating over $n$ dimensions, as needed in the direct calculation of the desired moment.

cs.IT↗

arXiv2020

An Integral Representation of the Logarithmic Function with Applications in Information Theory

We explore a well-known integral representation of the logarithmic function, and demonstrate its usefulness in obtaining compact, easily-computable exact formulas for quantities that involve expectations and higher moments of the logarithm of a positive random variable (or the logarithm of a sum of positive random variables). The integral representation of the logarithm is proved useful in a variety of information-theoretic applications, including universal lossless data compression, entropy and differential entropy evaluations, and the calculation of the ergodic capacity of the single-input, multiple-output (SIMO) Gaussian channel with random parameters (known to both transmitter and receiver). This integral representation and its variants are anticipated to serve as a useful tool in additional applications, as a rigorous alternative to the popular (but non-rigorous) replica method (at least in some situations).

cs.IT↗

arXiv2019

Large Deviations Behavior of the Logarithmic Error Probability of Random Codes

This work studies the deviations of the error exponent of the constant composition code ensemble around its expectation, known as the error exponent of the typical random code (TRC). In particular, it is shown that the probability of randomly drawing a codebook whose error exponent is smaller than the TRC exponent is exponentially small; upper and lower bounds for this exponent are given, which coincide in some cases. In addition, the probability of randomly drawing a codebook whose error exponent is larger than the TRC exponent is shown to be double-exponentially small; upper and lower bounds to the double-exponential exponent are given. The results suggest that codebooks whose error exponent is larger than the error exponent of the TRC are extremely rare. The key ingredient in the proofs is a new large deviations result of type class enumerators with dependent variables.

cs.IT↗

arXiv2019

Noisy Guesses

We consider the problem of guessing a random, finite-alphabet, secret $n$-vector, where the guesses are transmitted via a noisy channel. We provide a single-letter formula for the best achievable exponential growth rate of the $ρ$--th moment of the number of guesses, as a function of $n$. This formula exhibits a fairly clear insight concerning the penalty due to the noise. We describe two different randomized schemes that achieve the optimal guessing exponent. One of them is fully universal in the sense of being independent of source (that governs the vector to be guessed), the channel (that corrupts the guesses), and the moment power $ρ$. Interestingly, it turns out that, in general, the optimal guessing exponent function exhibits a phase transition when it is examined either as a function of the channel parameters, or as a function of $ρ$: as long as the channel is not too distant (in a certain sense to be defined precisely) from the identity channel (i.e., the clean channel), or equivalently, as long $ρ$ is larger than a certain critical value, $ρ_{\mbox{\tiny c}}$, there is no penalty at all in the guessing exponent, compared to the case of noiseless guessing.

cs.IT↗

arXiv2019

A Lagrange-Dual Lower Bound to the Error Exponent Function of the Typical Random Code

A Lagrange-dual (Gallager-style) lower bound is derived for the error exponent function of the typical random code (TRC) pertaining to the i.i.d. random coding ensemble and mismatched stochastic likelihood decoding. While the original expression, derived from the method of types (the Csiszar-style expression) involves minimization over probability distributions defined on the channel input--output alphabets, the new Lagrange-dual formula involves optimization of five parameters, independently of the alphabet sizes. For both stochastic and deterministic mismatched decoding (including maximum likelihood decoding as a special case),we provide a rather comprehensive discussion on the insight behind the various ingredients of this formula and describe how its behavior varies as the coding rate exhausts the relevant range. Among other things, it is demonstrated that this expression simultaneously generalizes both the expurgated error exponent function (at zero rate) and the classical random coding exponent function at high rates, where it also meets the sphere--packing bound.

cs.IT↗

arXiv2019

Guessing Individual Sequences: Generating Randomized Guesses Using Finite-State Machines

Motivated by earlier results on universal randomized guessing, we consider an individual-sequence approach to the guessing problem: in this setting, the goal is to guess a secret, individual (deterministic) vector $x^n=(x_1,\ldots,x_n)$, by using a finite-state machine that sequentially generates randomized guesses from a stream of purely random bits. We define the finite-state guessing exponent as the asymptotic normalized logarithm of the minimum achievable moment of the number of randomized guesses, generated by any finite-state machine, until $x^n$ is guessed successfully. We show that the finite-state guessing exponent of any sequence is intimately related to its finite-state compressibility (due to Lempel and Ziv), and it is asymptotically achieved by the decoder of (a certain modified version of) the 1978 Lempel-Ziv data compression algorithm (a.k.a. the LZ78 algorithm), fed by purely random bits. The results are also extended to the case where the guessing machine has access to a side information sequence, $y^n=(y_1,\ldots,y_n)$, which is also an individual sequence.

cs.IT↗

arXiv2019

Weak-Noise Modulation-Estimation of Vector Parameters

We address the problem of modulating a parameter onto a power-limited signal, transmitted over a discrete-time Gaussian channel and estimating this parameter at the receiver. Continuing an earlier work, where the optimal trade-off between the weak-noise estimation performance and the outage probability (threshold-effect breakdown) was studied for a single (scalar) parameter, here we extend the derivation of the weak-noise estimation performance to the case of a multi-dimensional vector parameter. This turns out to be a non-trivial extension, that provides a few insights and it has some interesting implications, which are discussed in depth. Several modifications and extensions of the basic setup are also studied and discussed.

cs.IT↗

arXiv2019

Error Exponents of Typical Random Trellis Codes

In continuation to an earlier work, where error exponents of typical random codes were studied in the context of general block coding, with no underlying structure, here we carry out a parallel study on typical random, time-varying trellis codes for general discrete memoryless channels, focusing on a certain range of low rates. By analyzing an upper bound to the error probability of the typical random trellis code, using the method of types, we first derive a Csiszar-style error exponent formula (with respect to the constraint length), which allows to easily identify and characterize properties of good codes and dominant error events. We also derive a Gallager-style form of this error exponent, which turns out to be related to the expurgated error exponent. The main result is further extended to channels with memory and mismatch.

cs.IT↗

arXiv2018

Error Exponents of Typical Random Codes for the Colored Gaussian Channel

The error exponent of the typical random code is defined as the asymptotic normalized expectation of the logarithm of the probability of error, as opposed to the traditional definition of the random coding exponent as the normalized logarithm of the expectation of the probability of error with respect to a given ensemble of codes. For a certain ensemble of independent codewords, with a given power spectrum, and a generalized stochastic mismatched decoder, we characterize the error exponent the typical random codes (TRC) for the colored Gaussian channel, with emphasis on the range of low rates, where the TRC error exponent differs in value from the ordinary random coding error exponent. The error exponent formula, which is exponentially tight at some range of low rates, is presented as the maximum of a certain function with respect to one parameter only (in the spirit of Gallager's formulas) in the case of matched decoding, and two parameters in the case of mismatched decoding. Several aspects of the main results are discussed. These include: general properties, a parametric representation, critical rates, phase transitions, optimal input spectrum (water pouring), and comparison to the random coding exponent.

cs.IT↗

arXiv2018

Universal Randomized Guessing with Application to Asynchronous Decentralized Brute-Force Attacks

Consider the problem of guessing the realization of a random vector $\textbf{X}$ by repeatedly submitting queries (guesses) of the form "Is $\textbf{X}$ equal to $\textbf{x}$?" until an affirmative answer is obtained. In this setup, a key figure of merit is the number of queries required until the right vector is identified, a number that is termed the \emph{guesswork}. Typically, one wishes to devise a guessing strategy which minimizes a certain guesswork moment. In this work, we study a universal, decentralized scenario where the guesser does not know the distribution of $\textbf{X}$, and is not allowed to use a strategy which prepares a list of words to be guessed in advance, or even remember which words were already used. Such a scenario is useful, for example, if bots within a Botnet carry out a brute-force attack in order to guess a password or decrypt a message, yet cannot coordinate the guesses between them or even know how many bots actually participate in the attack. We devise universal decentralized guessing strategies, first, for memoryless sources, and then generalize them for finite-state sources. In each case, we derive the guessing exponent, and then prove its asymptotic optimality by deriving a compatible converse bound. The strategies are based on randomized guessing using a universal distribution. We also extend the results to guessing with side information. Finally, for all above scenarios, we design efficient algorithms in order to sample from the universal distributions, resulting in strategies which do not depend on the source distribution, are efficient to implement, and can be used asynchronously by multiple agents.

cs.IT↗

arXiv2018

False-Accept/False-Reject Trade-offs in Biometric Authentication Systems

Biometric authentication systems, based on secret key generation, work as follows. In the enrollment stage, an individual provides a biometric signal that is mapped into a secret key and a helper message, the former being prepared to become available to the system at a later time (for authentication), and the latter is stored in a public database. When an authorized user requests authentication, claiming his/her identity as one of the subscribers, he/she has to provide a biometric signal again, and then the system, which retrieves also the helper message of the claimed subscriber, produces an estimate of the secret key, that is finally compared to the secret key of the claimed user. In case of a match, the authentication request is approved, otherwise, it is rejected. Evidently, there is an inherent tension between two desired, but conflicting, properties of the helper message encoder: on the one hand, the encoding should be informative enough concerning the identity of the real subscriber, in order to approve him/her in the authentication stage, but on the other hand, it should not be too informative, as otherwise, unauthorized imposters could easily fool the system and gain access. A good encoder should then trade off the two kinds of errors: the false reject (FR) error and the false accept (FA) error. In this work, we investigate trade-offs between the random coding FR error exponent and the best achievable FA error exponent. We compare two types of ensembles of codes: fixed-rate codes and variable-rate codes, and we show that the latter class of codes offers considerable improvement compared to the former. In doing this, we characterize the optimal rate functions for both types of codes. We also examine privacy leakage constraints for both fixed-rate codes and variable-rate codes.

cs.IT↗

arXiv2018

Codewords With Memory Improve Achievable Rate Regions of the Memoryless Gaussian Interference Channel

The two-user Gaussian interference channel (GIC) has been extensively studied in the literature during the last four decades. The full characterization of the capacity region of the GIC is a long-standing open problem, except the case of strong or very strong interference. For general GIC's, many inner bounds have been provided over the years, among of them, the Han-Kobayashi (HK) region, is the most celebrated one. Unfortunately, the calculation of the HK region is prohibitively complex, due to the appearance of some auxiliary random variables, whose optimal choice is an open problem. As in other multi-user communication systems, these achievable regions are based on ensembles of i.i.d. (memoryless) codewords, in the sense that the symbols within each codeword are drawn independently. In this paper, we show that for the GIC, it is worthwhile to employ random coding ensembles of codewords with memory. Specifically, we take known achievable regions for the GIC, and generalize/improve them by allowing dependency between the code symbols. For example, we improve the state-of-the-art HK region by drawing the codewords (of each codeword and for each user) from a first-order autoregressive moving average (ARMA) Gaussian process. In this way, we suggest several new achievable rate regions, which are easily calculable, and which are strictly better than state-of-the-art known achievable regions.

cs.IT↗

arXiv2018

Trade-offs Between Weak-Noise Estimation Performance and Outage Exponents in Nonlinear Modulation

We focus on the problem of modulating a parameter onto a power-limited signal transmitted over a discrete-time Gaussian channel and estimating this parameter at the receiver. Considering the well-known threshold effect in non-linear modulation systems, our approach is the following: instead of deriving upper and lower bounds on the total estimation error, which weigh both weak-noise errors and anomalous errors beyond the threshold, we separate the two kinds of errors. In particular, we derive upper and lower bounds on the best achievable trade-off between the exponential decay rate of the weak-noise expected error cost and the exponential decay rate of the probability of the anomalous error event, also referred to as the outage event. This outage event is left to be defined as part of the communication system design problem. Our achievability scheme, which is based on lattice codes, meets the lower bound at the high signal-to-noise (SNR) limit and for a certain range of trade-offs between the weak--noise error cost and the outage exponent.

cs.IT↗