Searcharxiv⌕ Search

arXiv subjects

Abdellatif Zaidi

Publications and source records attributed to Abdellatif Zaidi.

At least 37 records · Page 2Linked to original sources
arXiv2020

Some Results on the Vector Gaussian Hypothesis Testing Problem

This paper studies the problem of discriminating two multivariate Gaussian distributions in a distributed manner. Specifically, it characterizes in a special case the optimal typeII error exponent as a function of the available communication rate. As a side-result, the paper also presents the optimal type-II error exponent of a slight generalization of the hypothesis testing against conditional independence problem where the marginal distributions under the two hypotheses can be different.

cs.IT↗
arXiv2020

Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding

In this paper, we develop an unsupervised generative clustering framework that combines the Variational Information Bottleneck and the Gaussian Mixture Model. Specifically, in our approach, we use the Variational Information Bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the Evidence Lower Bound (ELBO) and provide a variational inference type algorithm that allows computing it. In the algorithm, the coders' mappings are parametrized using neural networks, and the bound is approximated by Monte Carlo sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.

cs.LG↗
arXiv2020

Vector Gaussian CEO Problem Under Logarithmic Loss and Applications

We study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, $K \geq 2$ agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO also has its own Gaussian noisy observation of the source and wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. The result can be seen as the counterpart to the vector Gaussian setting of that by Courtade-Weissman which provides the rate-distortion region of the model in the discrete memoryless setting. For the proof of this result, we obtain an outer bound by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, application of our results allows us to find the complete solutions of two related problems: a quadratic vector Gaussian CEO problem with determinant constraint and the vector Gaussian distributed Information Bottleneck problem. Finally, we develop Blahut-Arimoto type algorithms that allow to compute numerically the regions provided in this paper, for both discrete and Gaussian models. We illustrate the efficiency of our algorithms through some numerical examples.

cs.IT↗
arXiv2020

On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views

This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner-Ahlswede-Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs ("features") distributions under power limitations maximizing the "relevance" for the Gaussian information bottleneck, under "complexity" constraints.

cs.IT↗
arXiv2019

Distributed Hypothesis Testing: Cooperation and Concurrent Detection

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero arbitrarily slow. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system. For the setting with positive communication rates from the sensor to the detectors, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector~2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents. For most cases it shows a tradeoff between the two exponents.

cs.IT↗
arXiv2019

Distributed Variational Representation Learning

The problem of distributed representation learning is one in which multiple sources of information $X_1,\ldots,X_K$ are processed separately so as to learn as much information as possible about some ground truth $Y$. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, $K$ encoders, $K \geq 2$, compress their observations $X_1,\ldots,X_K$ separately in a manner such that, collectively, the produced representations preserve as much information as possible about $Y$. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations $X_1,\ldots,X_K$ being conditionally independent given $Y$). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

stat.ML↗
arXiv2019

Vector Gaussian CEO Problem Under Logarithmic Loss

In this paper, we study the vector Gaussian Chief Executive Officer (CEO) problem under logarithmic loss distortion measure. Specifically, $K \geq 2$ agents observe independently corrupted Gaussian noisy versions of a remote vector Gaussian source, and communicate independently with a decoder or CEO over rate-constrained noise-free links. The CEO wants to reconstruct the remote source to within some prescribed distortion level where the incurred distortion is measured under the logarithmic loss penalty criterion. We find an explicit characterization of the rate-distortion region of this model. For the proof of this result, we obtain an outer bound on the region of the vector Gaussian CEO problem by means of a technique that relies on the de Bruijn identity and the properties of Fisher information. The approach is similar to Ekrem-Ulukus outer bounding technique for the vector Gaussian CEO problem under quadratic distortion measure, for which it was there found generally non-tight; but it is shown here to yield a complete characterization of the region for the case of logarithmic loss measure. Also, we show that Gaussian test channels with time-sharing exhaust the Berger-Tung inner bound, which is optimal. Furthermore, we also show that the established result under logarithmic loss provides an outer bound for a quadratic vector Gaussian CEO problem with determinant constraint, for which we characterize the optimal rate-distortion region.

cs.IT↗
arXiv2019

On the Capacity of Cloud Radio Access Networks with Oblivious Relaying

We study the transmission over a network in which users send information to a remote destination through relay nodes that are connected to the destination via finite-capacity error-free links, i.e., a cloud radio access network. The relays are constrained to operate without knowledge of the users' codebooks, i.e., they perform oblivious processing. The destination, or central processor, however, is informed about the users' codebooks. We establish a single-letter characterization of the capacity region of this model for a class of discrete memoryless channels in which the outputs at the relay nodes are independent given the users' inputs. We show that both relaying à-la Cover-El Gamal, i.e., compress-and-forward with joint decompression and decoding, and "noisy network coding", are optimal. The proof of the converse part establishes, and utilizes, connections with the Chief Executive Officer (CEO) source coding problem under logarithmic loss distortion measure. Extensions to general discrete memoryless channels are also investigated. In this case, we establish inner and outer bounds on the capacity region. For memoryless Gaussian channels within the studied class of channels, we characterize the capacity region when the users are constrained to time-share among Gaussian codebooks. Furthermore, we also discuss the suboptimality of separate decompression-decoding and the role of time-sharing.

cs.IT↗
arXiv2018

Distributed Hypothesis Testing with Collaborative Detection

A detection system with a single sensor and two detectors is considered, where each of the terminals observes a memoryless source sequence, the sensor sends a message to both detectors and the first detector sends a message to the second detector. Communication of these messages is assumed to be error-free but rate-limited. The joint probability mass function (pmf) of the source sequences observed at the three terminals depends on an $\mathsf{M}$-ary hypothesis $(\mathsf{M} \geq 2)$, and the goal of the communication is that each detector can guess the underlying hypothesis. Detector $k$, $k=1,2$, aims to maximize the error exponent \textit{under hypothesis} $i_k$, $i_k \in \{1,\ldots,\mathsf{M}\}$, while ensuring a small probability of error under all other hypotheses. We study this problem in the case in which the detectors aim to maximize their error exponents under the \textit{same} hypothesis (i.e., $i_1=i_2$) and in the case in which they aim to maximize their error exponents under \textit{distinct} hypotheses (i.e., $i_1 \neq i_2$). For the setting in which $i_1=i_2$, we present an achievable exponents region for the case of positive communication rates, and show that it is optimal for a specific case of testing against independence. We also characterize the optimal exponents region in the case of zero communication rates. For the setting in which $i_1 \neq i_2$, we characterize the optimal exponents region in the case of zero communication rates.

cs.IT↗
arXiv2018

Distributed Hypothesis Testing with Concurrent Detections

A detection system with a single sensor and $\mathsf{K}$ detectors is considered, where each of the terminals observes a memoryless source sequence and the sensor sends a common message to all the detectors. The communication of this message is assumed error-free but rate-limited. The joint probability mass function (pmf) of the source sequences observed at the terminals depends on an $\mathsf{M}$-ary hypothesis $(\mathsf{M} \geq \mathsf{K})$, and the goal of the communication is that each detector can guess the underlying hypothesis. Each detector $k$ aims to maximize the error exponent under hypothesis $k$, while ensuring a small probability of error under all other hypotheses. This paper presents an achievable exponents region for the case of positive communication rate, and characterizes the optimal exponents region for the case of zero communication rate. All results extend also to a composite hypothesis testing scenario.

cs.IT↗
arXiv2017

Distributed Information Bottleneck Method for Discrete and Gaussian Sources

We study the problem of distributed information bottleneck, in which multiple encoders separately compress their observations in a manner such that, collectively, the compressed signals preserve as much information as possible about another signal. The model generalizes Tishby's centralized information bottleneck method to the setting of multiple distributed encoders. We establish single-letter characterizations of the information-rate region of this problem for both i) a class of discrete memoryless sources and ii) memoryless vector Gaussian sources. Furthermore, assuming a sum constraint on rate or complexity, for both models we develop Blahut-Arimoto type iterative algorithms that allow to compute optimal information-rate trade-offs, by iterating over a set of self-consistent equations.

cs.IT↗
arXiv2017

A Generalization of Blahut-Arimoto Algorithm to Compute Rate-Distortion Regions of Multiterminal Source Coding Under Logarithmic Loss

In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.

cs.IT↗
arXiv2017

In-network Compression for Multiterminal Cascade MIMO Systems

We study the problem of receive beamforming in uplink cascade multiple-input multiple-output (MIMO) systems as an instance of that of cascade multiterminal source coding for lossy function computation. Using this connection, we develop two coding schemes for the second and show that their application leads to beamforming schemes for the first. In the first coding scheme, each terminal in the cascade sends a description of the source that it observes; the decoder reconstructs all sources, lossily, and then computes an estimate of the desired function. This scheme improves upon standard routing in that every terminal only compresses the innovation of its source w.r.t. the descriptions that are sent by the previous terminals in the cascade. In the second scheme, the desired function is computed gradually in the cascade network, and each terminal sends a finer description of it. In the context of uplink cascade MIMO systems, the application of these two schemes leads to centralized receive-beamforming and distributed receive-beamforming, respectively. Numerical results illustrate the performance of the proposed methods and show that they outperform standard routing.

cs.IT↗
arXiv2017

On the Capacity of Cloud Radio Access Networks with Oblivious Relaying

We study the transmission over a network in which users send information to a remote destination through relay nodes that are connected to the destination via finite-capacity error-free links, i.e., a cloud radio access network. The relays are constrained to operate without knowledge of the users' codebooks, i.e., they perform oblivious processing - The destination, or central processor, however, is informed about the users' codebooks. We establish a single-letter characterization of the capacity region of this model for a class of discrete memoryless channels in which the outputs at the relay nodes are independent given the users' inputs. We show that both relaying à-la Cover-El Gamal, i.e., compress-and-forward with joint decompression and decoding, and "noisy network coding", are optimal. The proof of the converse part establishes, and utilizes, connections with the Chief Executive Officer (CEO) source coding problem under logarithmic loss distortion measure. Extensions to general discrete memoryless channels are also investigated. In this case, we establish inner and outer bounds on the capacity region. For memoryless Gaussian channels within the studied class of channels, we characterize the capacity under Gaussian channel inputs.

cs.IT↗
arXiv2017

Secure Lossy Source Coding for Some Classes of Helper and Gray-Wyner Models

In this work, we investigate two source coding models, a \emph{Helper} problem and a \emph{Gray-Wyner} problem, under equivocation constraints. Specifically, in the Helper problem, an encoder communicates with a legitimate receiver through noise-free rate-limited public and private links; and an external passive eavesdropper intercepts every information that is sent on the public link. We study two classes of this model: i) when a pair of arbitrarily correlated discrete memoryless sources is to be encoded such that one component has to be recovered lossily at the legitimate receiver while the equivocation about both components at the eavesdropper must be maintained no smaller than some prescribed level; and ii) when the legitimate receiver reproduces both components, one of which, that is recovered losslessly, has to be concealed from the eavesdropper to some equivocation level. For both classes problems, we establish single-letter characterizations of optimal rate-distortion-equivocation tradeoffs in the discrete memoryless case. Next, we extend our results to the case of two legitimate receivers, i.e., Gray-Wyner network with equivocation constraints. Here, two legitimate receivers are connected to the encoder each through a dedicated error-free private link as well as a common error-free public link; and an external passive eavesdropper overhears on the public link. We study two classes of this model that are extensions of the aforementioned instances of Helper problems to the case of two receivers. For each of the two classes, we establish a single-letter characterization of the optimal rate-distortion-equivocation region. Throughout the paper, the analysis sheds light on the role of the private links, and we illustrate the results by computing them for some binary examples. Also, we make some meaningful connections, e.g., with problems of secret-sharing and encryption.

cs.IT↗
arXiv2017

Rate-Distortion Region of a Gray-Wyner Model with Side Information

In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray-Wyner model with side information at the decoders. Specifically, in this model an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S^n_1,S^n_2)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S^n_2$ losslessly; and Receiver $1$ also reproduces the source component $S^n_1$ lossily, to within some prescribed fidelity level $D_1$. Also, Receiver $1$ and Receiver $2$ are equipped respectively with memoryless side information sequences $Y^n_1$ and $Y^n_2$. Important in this setup, the side information sequences are arbitrarily correlated among them, and with the source pair $(S^n_1,S^n_2)$; and are not assumed to exhibit any particular ordering. Furthermore, by specializing the main result to two Heegard-Berger models with successive refinement and scalable coding, we shed light on the roles of the common and private descriptions that the encoder should produce and what they should carry optimally. We develop intuitions by analyzing the developed single-letter optimal rate-distortion regions of these models, and discuss some insightful binary examples.

cs.IT↗
arXiv2016

On Achievability for Downlink Cloud Radio Access Networks with Base Station Cooperation

This work investigates the downlink of a cloud radio access network (C-RAN) in which a central processor communicates with two mobile users through two base stations (BSs). The BSs act as relay nodes and cooperate with each other through error-free rate-limited links. We develop and analyze two coding schemes for this scenario. The first coding scheme is based on Liu-Kang scheme for C-RANs without BS cooperation; and extends it to scenarios allowing conferencing between the BSs. Among few other features, our new coding scheme enables arbitrary correlation among the auxiliary codewords that are recovered by the BSs. It also introduces common codewords to be described to both BSs. For the analysis of this coding scheme, we extend the multivariate covering lemma to non-Cartesian product sets, thereby correcting an erroneous application of this lemma in Liu-Kang's related work. We highlight key aspects of this scheme by studying three important instances of it. The second coding scheme extends the so-called compression scheme that was originally developed for memoryless Gaussian C-RANs without BS cooperation to general discrete memoryless C-RANs with BS cooperation. We show that this scheme subsumes the original compression scheme when applied to memoryless Gaussian C-RAN models. In the analysis of this scheme, we also highlight important connections with the so-called distributed decode--forward scheme, and refine the approximate capacity of a general $N$-BS $L$-user C-RAN model in the memoryless Gaussian case.

cs.IT↗