arXiv subjects

Ibrahim Issa

Publications and source records attributed to Ibrahim Issa.

17 recordsLinked to original sources

arXiv2026

Sort, Partition, Randomize: Optimal Binary Hypothesis Testing under Local Differential Privacy

We study optimal design of $\varepsilon$-locally differentially private mechanisms for binary hypothesis testing. Each observation is drawn from one of two known distributions $P_0,P_1$ on a finite alphabet of size $k$, privatized by a mechanism $Q$, and then used to infer which distribution generated the data. We measure testing utility using an $f$-divergence, including total variation, KL, and hockey-stick divergences, between the two induced output distributions. Previous work established structural properties of optimal mechanisms, but only yielded exponential-time algorithms. We prove a sharp structure: for every $\varepsilon$ and every $f$-divergence objective, after sorting the alphabet by likelihood ratio, there exists an optimal mechanism that partitions the sorted alphabet into contiguous blocks and applies randomized response to the block label. We call this class Sort-Partition-Randomize (SPR). This characterization yields an exact dynamic program that computes an optimal mechanism in $O(k^3)$ time, and more generally in $O(\ell k^2)$ time with an $\ell$-output budget. Our results make it possible to efficiently compute and characterize the exact optimum across the full privacy range, beyond asymptotic privacy regimes.

cs.IT↗

arXiv2025

The Generalized Chernoff-Stein Lemma, Applications and Examples

In this manuscript we define the notion of "$\delta$-typicality" for both entropy and relative entropy, as well as a notion of $\epsilon$-goodness and provide an extension to Stein's lemma for continuous quantities as well as correlated setups. We apply the derived results on the Gaussian hypothesis testing problem where the observations are possibly correlated.

cs.IT↗

arXiv2024

Testing the Isotropic Cauchy Hypothesis

Isotropic $\alpha$-stable distributions are central in the theory of heavy-tailed distributions and play a role similar to that of the Gaussian density among finite second-moment laws. Given a sequence of $n$ observations, we are interested in characterizing the performance of Likelihood Ratio Tests where two hypotheses are plausible for the observed quantities: either isotropic Cauchy or isotropic Gaussian. Under various setups, we show that the probability of error of such detectors is not always exponentially decaying with $n$ with the leading term in the exponent shown to be logarithmic instead and we determine the constants in that leading term. Perhaps surprisingly, the optimal Bayesian probabilities of error are found to exhibit different asymptotic behaviors.

cs.IT↗

arXiv2024

Sibson $\alpha$-Mutual Information and Its Variational Representations

Information measures can be constructed from R\'enyi divergences much like mutual information from Kullback-Leibler divergence. One such information measure is known as Sibson $\alpha$-mutual information and has received renewed attention recently in several contexts: concentration of measure under dependence, statistical learning, hypothesis testing, and estimation theory. In this paper, we survey and extend the state of the art. In particular, we introduce variational representations for Sibson $\alpha$-mutual information and employ them in each described context to derive novel results. Namely, we produce generalized Transportation-Cost inequalities and Fano-type inequalities. We also present an overview of known applications, spanning from learning theory and Bayesian risk to universal prediction.

cs.IT↗

arXiv2023

Total Variation Meets Differential Privacy

The framework of approximate differential privacy is considered, and augmented by leveraging the notion of ``the total variation of a (privacy-preserving) mechanism'' (denoted by $\eta$-TV). With this refinement, an exact composition result is derived, and shown to be significantly tighter than the optimal bounds for differential privacy (which do not consider the total variation). Furthermore, it is shown that $(\varepsilon,\delta)$-DP with $\eta$-TV is closed under subsampling. The induced total variation of commonly used mechanisms are computed. Moreover, the notion of total variation of a mechanism is studied in the local privacy setting and privacy-utility tradeoffs are investigated. In particular, total variation distance and KL divergence are considered as utility functions and studied through the lens of contraction coefficients. Finally, the results are compared and connected to the locally differentially private setting.

cs.IT↗

arXiv2023

Generalization Error Bounds for Noisy, Iterative Algorithms via Maximal Leakage

We adopt an information-theoretic framework to analyze the generalization behavior of the class of iterative, noisy learning algorithms. This class is particularly suitable for study under information-theoretic metrics as the algorithms are inherently randomized, and it includes commonly used algorithms such as Stochastic Gradient Langevin Dynamics (SGLD). Herein, we use the maximal leakage (equivalently, the Sibson mutual information of order infinity) metric, as it is simple to analyze, and it implies both bounds on the probability of having a large generalization error and on its expected value. We show that, if the update function (e.g., gradient) is bounded in $L_2$-norm and the additive noise is isotropic Gaussian noise, then one can obtain an upper-bound on maximal leakage in semi-closed form. Furthermore, we demonstrate how the assumptions on the update function affect the optimal (in the sense of minimizing the induced maximal leakage) choice of the noise. Finally, we compute explicit tight upper bounds on the induced maximal leakage for other scenarios of interest.

cs.LG↗

arXiv2022

Age Distribution in Arbitrary Preemptive Memoryless Networks

We study the probability distribution of age of information (AoI) in arbitrary networks with memoryless service times. A source node generates packets following a Poisson process, and then the packets are forwarded across the network in such a way that newer updates preempt older ones. This model is equivalent to gossip networks that was recently studied by Yates, and for which he obtained a recursive formula allowing the computation for the average AoI. In this paper, we obtain a very simple characterization of the stationary distribution of AoI at every node in the network. This allows for the computation of the average of an arbitrary function of the age. In particular, we can compute age-violation probabilities. Furthermore, we show how it is possible to use insights from our simple characterization in order to substantially reduce the computation time of average AoIs in some structured networks. Finally, we describe how it is possible to use our characterization in order to obtain faster and more accurate Monte Carlo simulations estimating the average AoI, or the average of an arbitrary function of the age.

cs.IT↗

arXiv2020

Strong Asymptotic Composition Theorems for Mutual Information Measures

We characterize the growth of the Sibson and Arimoto mutual informations and $\alpha$-maximal leakage, of any order that is at least unity, between a random variable and a growing set of noisy, conditionally independent and identically-distributed observations of the random variable. Each of these measures increases exponentially fast to a limit that is order- and measure-dependent, with an exponent that is order- and measure-independent.

cs.IT↗

arXiv2020

Robust Generalization via $\alpha$-Mutual Information

The aim of this work is to provide bounds connecting two probability measures of the same event using R\'enyi $\alpha$-Divergences and Sibson's $\alpha$-Mutual Information, a generalization of respectively the Kullback-Leibler Divergence and Shannon's Mutual Information. A particular case of interest can be found when the two probability measures considered are a joint distribution and the corresponding product of marginals (representing the statistically independent scenario). In this case, a bound using Sibson's $\alpha-$Mutual Information is retrieved, extending a result involving Maximal Leakage to general alphabets. These results have broad applications, from bounding the generalization error of learning algorithms to the more general framework of adaptive data analysis, provided that the divergences and/or information measures used are amenable to such an analysis ({\it i.e.,} are robust to post-processing and compose adaptively). The generalization error bounds are derived with respect to high-probability events but a corresponding bound on expected generalization error is also retrieved.

cs.IT↗

arXiv2019

Generalization Error Bounds Via R\'enyi-, $f$-Divergences and Maximal Leakage

In this work, the probability of an event under some joint distribution is bounded by measuring it with the product of the marginals instead (which is typically easier to analyze) together with a measure of the dependence between the two random variables. These results find applications in adaptive data analysis, where multiple dependencies are introduced and in learning theory, where they can be employed to bound the generalization error of a learning algorithm. Bounds are given in terms of Sibson's Mutual Information, $\alpha-$Divergences, Hellinger Divergences, and $f-$Divergences. A case of particular interest is the Maximal Leakage (or Sibson's Mutual Information of order infinity), since this measure is robust to post-processing and composes adaptively. The corresponding bound can be seen as a generalization of classical bounds, such as Hoeffding's and McDiarmid's inequalities, to the case of dependent random variables.

cs.IT↗

arXiv2019

Strengthened Information-theoretic Bounds on the Generalization Error

The following problem is considered: given a joint distribution $P_{XY}$ and an event $E$, bound $P_{XY}(E)$ in terms of $P_XP_Y(E)$ (where $P_XP_Y$ is the product of the marginals of $P_{XY}$) and a measure of dependence of $X$ and $Y$. Such bounds have direct applications in the analysis of the generalization error of learning algorithms, where $E$ represents a large error event and the measure of dependence controls the degree of overfitting. Herein, bounds are demonstrated using several information-theoretic metrics, in particular: mutual information, lautum information, maximal leakage, and $J_\infty$. The mutual information bound can outperform comparable bounds in the literature by an arbitrarily large factor.

cs.IT↗

arXiv2019

A New Approach to Adaptive Data Analysis and Learning via Maximal Leakage

There is an increasing concern that most current published research findings are false. The main cause seems to lie in the fundamental disconnection between theory and practice in data analysis. While the former typically relies on statistical independence, the latter is an inherently adaptive process: new hypotheses are formulated based on the outcomes of previous analyses. A recent line of work tries to mitigate these issues by enforcing constraints, such as differential privacy, that compose adaptively while degrading gracefully and thus provide statistical guarantees even in adaptive contexts. Our contribution consists in the introduction of a new approach, based on the concept of Maximal Leakage, an information-theoretic measure of leakage of information. The main result allows us to compare the probability of an event happening when adaptivity is considered with respect to the non-adaptive scenario. The bound we derive represents a generalization of the bounds used in non-adaptive scenarios (e.g., McDiarmid's inequality for $c$-sensitive functions, false discovery error control via significance level, etc.), and allows us to replicate or even improve, in certain regimes, the results obtained using Max-Information or Differential Privacy. In contrast with the line of work started by Dwork et al., our results do not rely on Differential Privacy but are, in principle, applicable to every algorithm that has a bounded leakage, including the differentially private algorithms and the ones with a short description length.

stat.ML↗

arXiv2018

An Operational Approach to Information Leakage

Given two random variables $X$ and $Y$, an operational approach is undertaken to quantify the ``leakage'' of information from $X$ to $Y$. The resulting measure $\mathcal{L}(X \!\! \to \!\! Y)$ is called \emph{maximal leakage}, and is defined as the multiplicative increase, upon observing $Y$, of the probability of correctly guessing a randomized function of $X$, maximized over all such randomized functions. A closed-form expression for $\mathcal{L}(X \!\! \to \!\! Y)$ is given for discrete $X$ and $Y$, and it is subsequently generalized to handle a large class of random variables. The resulting properties are shown to be consistent with an axiomatic view of a leakage measure, and the definition is shown to be robust to variations in the setup. Moreover, a variant of the Shannon cipher system is studied, in which performance of an encryption scheme is measured using maximal leakage. A single-letter characterization of the optimal limit of (normalized) maximal leakage is derived and asymptotically-optimal encryption schemes are demonstrated. Furthermore, the sample complexity of estimating maximal leakage from data is characterized up to subpolynomial factors. Finally, the \emph{guessing} framework used to define maximal leakage is used to give operational interpretations of commonly used leakage measures, such as Shannon capacity, maximal correlation, and local differential privacy.

cs.IT↗

arXiv2017

Measuring Quantum Entropy

The entropy of a quantum system is a measure of its randomness, and has applications in measuring quantum entanglement. We study the problem of measuring the von Neumann entropy, $S(\rho)$, and R\'enyi entropy, $S_\alpha(\rho)$ of an unknown mixed quantum state $\rho$ in $d$ dimensions, given access to independent copies of $\rho$. We provide an algorithm with copy complexity $O(d^{2/\alpha})$ for estimating $S_\alpha(\rho)$ for $\alpha<1$, and copy complexity $O(d^{2})$ for estimating $S(\rho)$, and $S_\alpha(\rho)$ for non-integral $\alpha>1$. These bounds are at least quadratic in $d$, which is the order dependence on the number of copies required for learning the entire state $\rho$. For integral $\alpha>1$, on the other hand, we provide an algorithm for estimating $S_\alpha(\rho)$ with a sub-quadratic copy complexity of $O(d^{2-2/\alpha})$. We characterize the copy complexity for integral $\alpha>1$ up to constant factors by providing matching lower bounds. For other values of $\alpha$, and the von Neumann entropy, we show lower bounds on the algorithm that achieves the upper bound. This shows that we either need new algorithms for better upper bounds, or better lower bounds to tighten the results. For non-integral $\alpha$, and the von Neumann entropy, we consider the well known Empirical Young Diagram (EYD) algorithm, which is the analogue of empirical plug-in estimator in classical distribution estimation. As a corollary, we strengthen a lower bound on the copy complexity of the EYD algorithm for learning the maximally mixed state by showing that the lower bound holds with exponential probability (which was previously known to hold with a constant probability). For integral $\alpha>1$, we provide new concentration results of certain polynomials that arise in Kerov algebra of Young diagrams.

quant-ph↗

arXiv2015

Measuring Secrecy by the Probability of a Successful Guess

The secrecy of a communication system in which both the legitimate receiver and an eavesdropper are allowed some distortion is investigated. The secrecy metric considered is the exponent of the probability that the eavesdropper estimates the source sequence successfully within an acceptable distortion level. The problem is first studied when the transmitter and the legitimate receiver do not share any key and the transmitter is not subject to a rate constraint, which corresponds to a stylized model of a side channel and reveals connections to source coding with side information. The setting is then generalized to include a shared secret key between the transmitter and the legitimate receiver and a rate constraint on the transmitter, which corresponds to the Shannon cipher system. A single-letter characterization of the highest achievable exponent is provided, and asymptotically-optimal strategies for both the primary user and the eavesdropper are demonstrated.

cs.IT↗

arXiv2013

Two-Hop Interference Channels: Impact of Linear Schemes

We consider the two-hop interference channel (IC), which consists of two source-destination pairs communicating with each other via two relays. We analyze the degrees of freedom (DoF) of this network when the relays are restricted to perform linear schemes, and the channel gains are constant (i.e., slow fading). We show that, somewhat surprisingly, by using vector-linear strategies at the relays, it is possible to achieve 4/3 sum-DoF when the channel gains are real. The key achievability idea is to alternate relaying coefficients across time, to create different end-to-end interference structures (or topologies) at different times. Although each of these topologies has only 1 sum-DoF, we manage to achieve 4/3 by coding across them. Furthermore, we develop a novel outer bound that matches our achievability, hence characterizing the sum-DoF of two-hop interference channels with linear schemes. As for the case of complex channel gains, we characterize the sum-DoF with linear schemes to be 5/3. We also generalize the results to the multi-antenna setting, characterizing the sum-DoF with linear schemes to be 2M-1/3 (for complex channel gains), where M is the number of antennas at each node.

cs.IT↗

arXiv2013

Two-Hop Interference Channels: Impact of Linear Time-Varying Schemes

We consider the two-hop interference channel (IC) with constant real channel coefficients, which consists of two source-destination pairs, separated by two relays. We analyze the achievable degrees of freedom (DoF) of such network when relays are restricted to perform scalar amplify-forward (AF) operations, with possibly time-varying coefficients. We show that, somewhat surprisingly, by providing the flexibility of choosing time-varying AF coefficients at the relays, it is possible to achieve 4/3 sum-DoF. We also develop a novel outer bound that matches our achievability, hence characterizing the sum-DoF of two-hop interference channels with time-varying AF relaying strategies.

cs.IT↗