arXiv subjects

Fady Alajaji

Publications and source records attributed to Fady Alajaji.

At least 19 recordsLinked to original sources

arXiv2026

Rejection Sampling is Optimal for Relative Entropy Coding

In relative entropy coding, a sender aims to design a stochastic code such that, on input $X \sim P_X$, the receiver can generate a sample $Y \sim P_{Y \mid X}$. It is a standard result that (1) this requires at least $I(X; Y)$ bits, (2) the lower bound is achievable within a logarithmic gap, and (3) this gap cannot be reduced in general. The necessity of the gap suggests that the mutual information is not the correct information measure to quantify the rate of relative entropy coding. A potential alternative emerged in the work of Flamich et al. (2025), who proved a tighter lower bound of $I_F(X \to Y)$, a quantity we call the functional information. In this paper, we construct the ring toss code, an encoding method for rejection sampling which uses at most $I_F(X \to Y) + \log e$ bits, hence obtaining a tight one-shot characterization of relative entropy coding. For the trivial channel $Y = X$, our result recovers the noiseless source coding theorem within a small constant. For a general channel, it implies that the classical mutual information lower bound is achievable within $\log(I(X; Y) + 1) + 2.45$ bits in general and within $1.45$ bits for singular channels, which are both the tightest bounds of their kind to date. Moreover, our one-shot result also recovers Sriramu and Wagner's asymptotic results on the second-order redundancy of relative entropy codes.

cs.IT↗

arXiv2026

Variable Rate Lossy Source-Channel Coding over Channels with Feedback

A variable-rate lossy joint source-channel coding scheme for burst-noise communication channels with noiseless feedback is introduced. The scheme comprises a multi-stage channel optimized vector quantization system that dynamically allocates bits via a greedy algorithm among the variable-rate residual quantizers at each stage, based on the channel output sequence received at the encoder through the feedback link, thereby generalizing a prior fixed-rate scheme. Simulations over an $M$-th order Markov noise Polya contagion channel demonstrate that the proposed variable-rate scheme consistently outperforms the fixed-rate scheme, regardless of the channel bit error rate and noise correlation, achieving signal-to-noise ratio gains of up to about 4.5 dB.

cs.IT↗

arXiv2026

SLAM as a Stochastic Control Problem with Partial Information: Optimal Solutions and Rigorous Approximations

Simultaneous localization and mapping (SLAM) is a foundational state estimation problem in robotics in which a robot accurately constructs a map of its environment while also localizing itself within this construction. We study the active SLAM problem through the lens of optimal stochastic control, thereby recasting it as a decision-making problem under partial information. After reviewing several commonly studied models, we present a general stochastic control formulation of active SLAM together with a rigorous treatment of motion, sensing, and map representation. We introduce a new exploration stage cost that encodes the geometry of the state when evaluating information-gathering actions. This formulation, constructed as a nonstandard partially observable Markov decision process (POMDP), is then analyzed to derive rigorously justified approximate solutions that are near-optimal. To enable this analysis, the associated regularity conditions are studied under general assumptions that apply to a wide range of robotics applications. For a particular case, we conduct an extensive numerical study in which standard learning algorithms are used to learn near-optimal policies.

cs.RO↗

arXiv2026

Communication Complexity of Exact Sampling under Rényi Information

We study the problem of exact sampling under an exponential communication cost, specifically Campbell's average codeword length $L(t)$ of order $t$, and Rényi's entropy. We provide a lower bound on the Campbell cost of exact sampling that grows approximately as $D_{1/α}(P||Q)$, the Rényi divergence of order $1/α$, with $α= \frac{1}{1+t}$. Using the Poisson functional representation of Li and El Gamal, we prove an upper bound on $L(t)$ whose leading Rényi divergence term has order within $ε$ of that of the lower bound. Our results reduce to the bounds of Harsha et al. as $α\to 1$. We also provide numerical examples comparing the bounds in the cases of normal and Laplacian distributions, demonstrating that the upper and lower bounds are typically within 5-10 bits of each other. Our results characterize exactly the optimal asymptotic Campbell cost $L(t)$ per sample as the number of independent and identically distributed (i.i.d.) samples grows to infinity. We show that under the exponential cost, any causal sampler performs strictly worse asymptotically than noncausal samplers. This contrasts with the case of expected message length, where both causal and noncausal samplers have the same optimal asymptotic cost.

cs.IT↗

arXiv2026

Pólya Thresholds Graphs

We introduce the Pólya threshold graph model and derive its stochastic and algebraic properties. This random threshold graph is generated sequentially via a two-color Pólya urn process. Starting from an empty graph, each time step involves a draw from the urn that produces an indicator variable, determining whether a newly added node is universal (connected to all existing nodes and itself) or isolated (connected to no existing nodes). This construction yields a random threshold graph with an adjacency matrix that admits an explicit representation in terms of the draw sequence. Using the structure of the Pólya draw process, we derive the exact degree distribution for any arbitrary node, including its mean and variance. Furthermore, we evaluate a distance-based decay centrality score and provide an explicit expression for its expectation. On the algebraic side, we explicitly characterize the Laplacian matrix of the random threshold graph, obtaining a closed-form description of its spectrum and corresponding eigenbasis. Finally, as an application of these structural results, we analyze discrete-time consensus dynamics on Pólya threshold graphs.

cs.IT↗

arXiv2025

Sliding Window Codes: Near-Optimality and Q-Learning for Zero-Delay Coding

We study the problem of zero-delay coding for the transmission of a Markov source over a noisy channel with feedback and present a reinforcement learning solution which is guaranteed to achieve near-optimality. To this end, we formulate the problem as a Markov decision process (MDP) where the state is a probability-measure valued predictor/belief and the actions are quantizer maps. This MDP formulation has been used to show the optimality of certain classes of encoder policies in prior work, but their computation is prohibitively complex due to the uncountable nature of the constructed state space and the lack of minorization or strong ergodicity results. These challenges invite rigorous reinforcement learning methods, which entail several open questions: can we approximate this MDP with a finite-state one with some performance guarantee? Can we ensure convergence of a reinforcement learning algorithm for this approximate MDP? What regularity assumptions are required for the above to hold? We address these questions as follows: we present an approximation of the belief MDP using a sliding finite window of channel outputs and quantizers. Under an appropriate notion of predictor stability, we show that policies based on this finite window are near-optimal, in the sense that the lowest distortion achievable by such a policy approaches the true lowest distortion as the window length increases. We give sufficient conditions for predictor stability to hold. Finally, we propose a Q-learning algorithm which provably converges to a near-optimal policy and provide a detailed comparison of~the sliding finite window scheme with another approximation scheme which quantizes the belief MDP in a nearest neighbor fashion.

math.OC↗

arXiv2025

Bounds on the Excess Minimum Risk via Generalized Information Divergence Measures

Given finite-dimensional random vectors $Y$, $X$, and $Z$ that form a Markov chain in that order (i.e., $Y \to X \to Z$), we derive upper bounds on the excess minimum risk using generalized information divergence measures. Here, $Y$ is a target vector to be estimated from an observed feature vector $X$ or its stochastically degraded version $Z$. The excess minimum risk is defined as the difference between the minimum expected loss in estimating $Y$ from $X$ and from $Z$. We present a family of bounds that generalize the mutual information based bound of Györfi et al. (2023), using the Rényi and $α$-Jensen-Shannon divergences, as well as Sibson's mutual information. Our bounds are similar to those developed by Modak et al. (2021) and Aminian et al. (2024) for the generalization error of learning algorithms. However, unlike these works, our bounds do not require the sub-Gaussian parameter to be constant and therefore apply to a broader class of joint distributions over $Y$, $X$, and $Z$. We also provide numerical examples under both constant and non-constant sub-Gaussianity assumptions, illustrating that our generalized divergence based bounds can be tighter than the one based on mutual information for certain regimes of the parameter $α$.

cs.IT↗

arXiv2024

A Unifying Generator Loss Function for Generative Adversarial Networks

A unifying $α$-parametrized generator loss function is introduced for a dual-objective generative adversarial network (GAN), which uses a canonical (or classical) discriminator loss function such as the one in the original GAN (VanillaGAN) system. The generator loss function is based on a symmetric class probability estimation type function, $\mathcal{L}_α$, and the resulting GAN system is termed $\mathcal{L}_α$-GAN. Under an optimal discriminator, it is shown that the generator's optimization problem consists of minimizing a Jensen-$f_α$-divergence, a natural generalization of the Jensen-Shannon divergence, where $f_α$ is a convex function expressed in terms of the loss function $\mathcal{L}_α$. It is also demonstrated that this $\mathcal{L}_α$-GAN problem recovers as special cases a number of GAN problems in the literature, including VanillaGAN, Least Squares GAN (LSGAN), Least $k$th order GAN (L$k$GAN) and the recently introduced $(α_D,α_G)$-GAN with $α_D=1$. Finally, experimental results are conducted on three datasets, MNIST, CIFAR-10, and Stacked MNIST to illustrate the performance of various examples of the $\mathcal{L}_α$-GAN system.

cs.LG↗

arXiv2024

Optimal Binary Signaling for a Two Sensor Gaussian MAC Network

We consider a two sensor distributed detection system transmitting a binary non-uniform source over a Gaussian multiple access channel (MAC). We model the network via binary sensors whose outputs are generated by binary symmetric channels of different noise levels. We prove an optimal one dimensional constellation design under individual sensor power constraints which minimizes the error probability of detecting the source. Three distinct cases arise for this optimization based on the parameters in the problem setup. In the most notable case (Case III), the optimal signaling design is to not necessarily use all of the power allocated to the more noisy sensor (with less correlation to the source). We compare the error performance of the optimal one dimensional constellation to orthogonal signaling. The results show that the optimal one dimensional constellation achieves lower error probability than using orthogonal channels.

cs.IT↗

arXiv2024

Generating Preferential Attachment Graphs via a Pólya Urn with Expanding Colors

We introduce a novel preferential attachment model using the draw variables of a modified Pólya urn with an expanding number of colors, notably capable of modeling influential opinions (in terms of vertices of high degree) as the graph evolves. Similar to the Barabási-Albert model, the generated graph grows in size by one vertex at each time instance; in contrast however, each vertex of the graph is uniquely characterized by a color, which is represented by a ball color in the Pólya urn. More specifically at each time step, we draw a ball from the urn and return it to the urn along with a number (potentially time-varying and non-integer) of reinforcing balls of the same color; we also add another ball of a new color to the urn. We then construct an edge between the new vertex (corresponding to the new color) and the existing vertex whose color ball is drawn. Using color-coded vertices in conjunction with the time-varying reinforcing parameter allows for vertices added (born) later in the process to potentially attain a high degree in a way that is not captured in the Barabási-Albert model. We study the degree count of the vertices by analyzing the draw vectors of the underlying stochastic process. In particular, we establish the probability distribution of the random variable counting the number of draws of a given color which determines the degree of the vertex corresponding to that color in the graph. We further provide simulation results presenting a comparison between our model and the Barabási-Albert network.

math.PR↗

arXiv2023

Classification Utility, Fairness, and Compactness via Tunable Information Bottleneck and Rényi Measures

Designing machine learning algorithms that are accurate yet fair, not discriminating based on any sensitive attribute, is of paramount importance for society to accept AI for critical applications. In this article, we propose a novel fair representation learning method termed the Rényi Fair Information Bottleneck Method (RFIB) which incorporates constraints for utility, fairness, and compactness (compression) of representation, and apply it to image and tabular data classification. A key attribute of our approach is that we consider - in contrast to most prior work - both demographic parity and equalized odds as fairness constraints, allowing for a more nuanced satisfaction of both criteria. Leveraging a variational approach, we show that our objectives yield a loss function involving classical Information Bottleneck (IB) measures and establish an upper bound in terms of two Rényi measures of order $α$ on the mutual information IB term measuring compactness between the input and its encoded embedding. We study the influence of the $α$ parameter as well as two other tunable IB parameters on achieving utility/fairness trade-off goals, and show that the $α$ parameter gives an additional degree of freedom that can be used to control the compactness of the representation. Experimenting on three different image datasets (EyePACS, CelebA, and FairFace) and two tabular datasets (Adult and COMPAS), using both binary and categorical sensitive attributes, we show that on various utility, fairness, and compound utility/fairness metrics RFIB outperforms current state-of-the-art approaches.

cs.LG↗

arXiv2023

Optimized Constellation Design for Two User Binary Sensor Networks Using NOMA

Data Fusion of wireless sensors is a common technique employed in many communication systems. This work focuses on incorporating the principles of non-orthogonal-multiple-access (NOMA) to optimize error performance directly in the choice of constellation design. More specifically, the problem of two sensor data fusion of a binary uniform source sent over a Gaussian multiple access channel via symmetric binary constellations is investigated. A so-called planar upper bound on the error probability is analytically derived. A constellation design is then obtained by establishing in closed form its rotation parameter that minimizes the upper bound. Simulation results show that the resulting constellations achieve a near identical performance as experimentally determined optimal constellations.

cs.IT↗

arXiv2023

On Decoder Ties for the Binary Symmetric Channel with Arbitrarily Distributed Input

The error probability of block codes sent under a non-uniform input distribution over the memoryless binary symmetric channel (BSC) and decoded via the maximum a posteriori (MAP) decoding rule is investigated. It is proved that the ratio of the probability of MAP decoder ties to the probability of error when no MAP decoding ties occur grows at most linearly in blocklength, thus showing that decoder ties do not affect the code's error exponent. This result generalizes a similar recent result shown for the case of block codes transmitted over the BSC under a uniform input distribution.

cs.IT↗

arXiv2023

Evaluating Trade-offs in Computer Vision Between Attribute Privacy, Fairness and Utility

This paper investigates to what degree and magnitude tradeoffs exist between utility, fairness and attribute privacy in computer vision. Regarding privacy, we look at this important problem specifically in the context of attribute inference attacks, a less addressed form of privacy. To create a variety of models with different preferences, we use adversarial methods to intervene on attributes relating to fairness and privacy. We see that that certain tradeoffs exist between fairness and utility, privacy and utility, and between privacy and fairness. The results also show that those tradeoffs and interactions are more complex and nonlinear between the three goals than intuition would suggest.

cs.CV↗

arXiv2022

Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory

Two Rényi-type generalizations of the Shannon cross-entropy, the Rényi cross-entropy and the Natural Rényi cross-entropy, were recently used as loss functions for the improved design of deep learning generative adversarial networks. In this work, we build upon our results in [1] by deriving the Rényi and Natural Rényi differential cross-entropy measures in closed form for a wide class of common continuous distributions belonging to the exponential family and tabulating the results for ease of reference. We also summarise the Rényi-type cross-entropy rates between stationary Gaussian processes and between finite-alphabet time-invariant Markov sources.

cs.IT↗

arXiv2022

On the Rényi Cross-Entropy

The Rényi cross-entropy measure between two distributions, a generalization of the Shannon cross-entropy, was recently used as a loss function for the improved design of deep learning generative adversarial networks. In this work, we examine the properties of this measure and derive closed-form expressions for it when one of the distributions is fixed and when both distributions belong to the exponential family. We also analytically determine a formula for the cross-entropy rate for stationary Gaussian processes and for finite-alphabet Markov sources.

cs.IT↗

arXiv2022

Renyi Fair Information Bottleneck for Image Classification

We develop a novel method for ensuring fairness in machine learning which we term as the Renyi Fair Information Bottleneck (RFIB). We consider two different fairness constraints - demographic parity and equalized odds - for learning fair representations and derive a loss function via a variational approach that uses Renyi's divergence with its tunable parameter $α$ and that takes into account the triple constraints of utility, fairness, and compactness of representation. We then evaluate the performance of our method for image classification using the EyePACS medical imaging dataset, showing it outperforms competing state of the art techniques with performance measured using a variety of compound utility/fairness metrics, including accuracy gap and Rawls' minimal accuracy.

cs.LG↗

arXiv2021

A Finite Memory Interacting Pólya Contagion Network and its Approximating Dynamical Systems

We introduce a new model for contagion spread using a network of interacting finite memory two-color Pólya urns, which we refer to as the finite memory interacting Pólya contagion network. The urns interact in the sense that the probability of drawing a red ball (which represents an infection state) for a given urn, not only depends on the ratio of red balls in that urn but also on the ratio of red balls in the other urns in the network, hence accounting for the effect of spatial contagion. The resulting network-wide contagion process is a discrete-time finite-memory ($M$th order) Markov process, whose transition probability matrix is determined. The stochastic properties of the network contagion Markov process are analytically examined, and for homogeneous system parameters, we characterize the limiting state of infection in each urn. For the non-homogeneous case, given the complexity of the stochastic process, and in the same spirit as the well-studied SIS models, we use a mean-field type approximation to obtain a discrete-time dynamical system for the finite memory interacting Pólya contagion network. Interestingly, for $M=1$, we obtain a linear dynamical system which exactly represents the corresponding Markov process. For $M>1$, we use mean-field approximation to obtain a nonlinear dynamical system. Furthermore, noting that the latter dynamical system admits a linear variant (realized by retaining its leading linear terms), we study the asymptotic behavior of the linear systems for both memory modes and characterize their equilibrium. Finally, we present simulation studies to assess the quality of the approximation purveyed by the linear and non-linear dynamical systems.

math.DS↗