arXiv subjects

Deniz Gündüz

Publications and source records attributed to Deniz Gündüz.

At least 55 records · Page 3Linked to original sources

arXiv2025

Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding

LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights.

cs.LG↗

arXiv2025

Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality

While test-time scaling with verification has shown promise in improving the performance of large language models (LLMs), the role of the verifier and its imperfections remain underexplored. The effect of verification manifests through interactions of three quantities: (i) the generator's coverage, (ii) the verifier's region of convergence (ROC), and (iii) the sampling algorithm's sub-optimality. Though recent studies capture subsets of these factors, a unified framework quantifying the geometry of their interplay is missing. We frame verifiable test-time scaling as a transport problem. This characterizes the interaction of coverage, ROC, and sub-optimality, and uncovers that the sub-optimality--coverage curve exhibits three regimes. A transport regime -- where sub-optimality increases with coverage, a policy improvement regime -- where sub-optimality may decrease with coverage, depending on the verifier's ROC, and a saturation regime -- where sub-optimality plateaus, unaffected by coverage. We further propose and analyze two classes of sampling algorithms -- sequential and batched, and examine how their computational complexities shape these trade-offs. Empirical results with Qwen, Llama, and Gemma models corroborate our theoretical findings.

cs.AI↗

arXiv2025

Goal-Oriented Joint Source-Channel Coding: Distortion-Classification-Power Trade-off

Joint source-channel coding is a compelling paradigm when low-latency and low-complexity communication is required. This work proposes a theoretical framework that integrates classification and anomaly detection within the conventional signal reconstruction objective. Assuming a Gaussian scalar source and constraining the encoder to piecewise linear mappings, we derive tractable design rules and explicitly characterize the trade-offs between distortion, classification error, and transmission power.

eess.SP↗

arXiv2025

Actions Speak Louder Than Words: Rate-Reward Trade-off in Markov Decision Processes

The impact of communication on decision-making systems has been extensively studied under the assumption of dedicated communication channels. We instead consider communicating through actions, where the message is embedded into the actions of an agent which interacts with the environment in a Markov decision process (MDP) framework. We conceptualize the MDP environment as a finite-state channel (FSC), where the actions of the agent serve as the channel input, while the states of the MDP observed by another agent (i.e., receiver) serve as the channel output. Here, we treat the environment as a communication channel over which the agent communicates through its actions, while at the same time, trying to maximize its reward. We first characterize the optimal information theoretic trade-off between the average reward and the rate of reliable communication in the infinite-horizon regime. Then, we propose a novel framework to design a joint control/coding policy, termed \textit{Act2Comm}, which seamlessly embeds messages into actions. From a communication perspective, \textit{Act2Comm} functions as a learning-based channel coding scheme for non-differentiable FSCs under input-output constraints. From a control standpoint, \textit{Act2Comm} learns an MDP policy that incorporates communication capabilities, though at the cost of some control performance. Overall, \textit{Act2Comm} effectively balances the dual objectives of control and communication in this environment. Experimental results validate \textit{Act2Comm}'s capability to enable reliable communication while maintaining a certain level of control performance.

cs.IT↗

arXiv2025

LotteryCodec: Searching the Implicit Representation in a Random Network for Low-Complexity Image Compression

We introduce and validate the lottery codec hypothesis, which states that untrained subnetworks within randomly initialized networks can serve as synthesis networks for overfitted image compression, achieving rate-distortion (RD) performance comparable to trained networks. This hypothesis leads to a new paradigm for image compression by encoding image statistics into the network substructure. Building on this hypothesis, we propose LotteryCodec, which overfits a binary mask to an individual image, leveraging an over-parameterized and randomly initialized network shared by the encoder and the decoder. To address over-parameterization challenges and streamline subnetwork search, we develop a rewind modulation mechanism that improves the RD performance. LotteryCodec outperforms VTM and sets a new state-of-the-art in single-image compression. LotteryCodec also enables adaptive decoding complexity through adjustable mask ratios, offering flexible compression solutions for diverse device constraints and application requirements.

eess.IV↗

arXiv2025

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed {\em SharedRep-RLHF}. At its core, SharedRep-RLHF learns and leverages {\em shared traits} in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep-RLHF. Experiments across diverse natural language tasks showcase the effectiveness of SharedRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.

cs.LG↗

arXiv2025

Realizing Fully-Connected Layers Over the Air via Reconfigurable Intelligent Surfaces

By leveraging the waveform superposition property of the multiple access channel, over-the-air computation (AirComp) enables the execution of digital computations through analog means in the wireless domain, leading to faster processing and reduced latency. In this paper, we propose a novel approach to implement a neural network (NN) consisting of digital fully connected (FC) layers using physically reconfigurable hardware. Specifically, we investigate reconfigurable intelligent surfaces (RISs)-assisted multiple-input multiple-output (MIMO) systems to emulate the functionality of a NN for over-the-air inference. In this setup, both the RIS and the transceiver are jointly configured to manipulate the ambient wireless propagation environment, effectively reproducing the adjustable weights of a digital FC layer. We refer to this new computational paradigm as \textit{AirFC}. We formulate an imitation error minimization problem between the effective channel created by RIS and a target FC layer by jointly optimizing over-the-air parameters. To solve this non-convex optimization problem, an extremely low-complexity alternating optimization algorithm is proposed, where semi-closed-form/closed-form solutions for all optimization variables are derived. Simulation results show that the RIS-assisted MIMO-based AirFC can achieve competitive classification accuracy. Furthermore, it is also shown that a multi-RIS configuration significantly outperforms a single-RIS setup, particularly in line-of-sight (LoS)-dominated channels.

cs.IT↗

arXiv2025

Communication via Sensing

We present an alternative take on the recently popularized concept of `\textit{joint sensing and communications}', which focuses on using communication resources also for sensing. Here, we propose the opposite, where we utilize the receiver's sensing capabilities for communication. Our goal is to characterize the fundamental limits of communication over such a channel, which we call `\textit{communication via sensing}'. We assume that changes in the sensed attributes, such as location and speed, are limited due to practical constraints, which are captured by assuming a finite-state channel (FSC) with an input cost constraint. We first formulate an upper bound on the $N$-letter capacity as a cost-constrained optimization problem over the input sequence distribution, and then convert it to an equivalent problem over the state sequence distribution. Moreover, by breaking a walk on the underlying Markov chain into a weighted sum of traversed graph cycles in the long walk limit, we obtain a compact single-letter formulation of the capacity upper bound. Finally, for a specific case of a two-state FSC with noisy sensing characterized by a binary symmetric channel (BSC), we obtain a closed-form expression for the capacity upper bound. Comparison with an existing numerical lower bound shows that our proposed upper bound is very tight for all crossover probabilities.

cs.IT↗

arXiv2025

Semantics-Guided Diffusion for Deep Joint Source-Channel Coding in Wireless Image Transmission

Joint source-channel coding (JSCC) offers a promising avenue for enhancing transmission efficiency by jointly incorporating source and channel statistics into the system design. A key advancement in this area is the deep joint source and channel coding (DeepJSCC) technique that designs a direct mapping of input signals to channel symbols parameterized by a neural network, which can be trained for arbitrary channel models and semantic quality metrics. This paper advances the DeepJSCC framework toward a semantics-aligned, high-fidelity transmission approach, called semantics-guided diffusion DeepJSCC (SGD-JSCC). Existing schemes that integrate diffusion models (DMs) with JSCC face challenges in transforming random generation into accurate reconstruction and adapting to varying channel conditions. SGD-JSCC incorporates two key innovations: (1) utilizing some inherent information that contributes to the semantics of an image, such as text description or edge map, to guide the diffusion denoising process; and (2) enabling seamless adaptability to varying channel conditions with the help of a semantics-guided DM for channel denoising. The DM is guided by diverse semantic information and integrates seamlessly with DeepJSCC. In a slow fading channel, SGD-JSCC dynamically adapts to the instantaneous signal-to-noise ratio (SNR) directly estimated from the channel output, thereby eliminating the need for additional pilot transmissions for channel estimation. In a fast fading channel, we introduce a training-free denoising strategy, allowing SGD-JSCC to effectively adjust to fluctuations in channel gains. Numerical results demonstrate that, guided by semantic information and leveraging the powerful DM, our method outperforms existing DeepJSCC schemes, delivering satisfactory reconstruction performance even at extremely poor channel conditions.

cs.IT↗

arXiv2025

Remote Channel Synthesis

We consider the problem of synthesizing a memoryless channel between an unobserved source and a remote terminal. An encoder has access to a partial or noisy version $Z^n = (Z_1, \ldots, Z_n)$ of a remote source sequence $X^n = (X_1, \ldots, X_n),$ with $(X_i,Z_i)$ independent and identically distributed with joint distribution $q_{X,Z}.$ The encoder communicates through a noiseless link to a decoder which aims to produce an output $Y^n$ coordinated with the remote source; that is, the total variation distance between the joint distribution of $X^n$ and $Y^n$ and some i.i.d. target distribution $q_{X,Y}^{\otimes n}$ is required to vanish as $n$ goes to infinity. The two terminals may have access to a source of rate-limited common randomness. We present a single-letter characterization of the optimal compression and common randomness rates. We also show that when the common randomness rate is small, then in most cases, coordinating $Z^n$ and $Y^n$ using a standard channel synthesis scheme is strictly sub-optimal. In other words, schemes for which the joint distribution of $Z^n$ and $Y^n$ approaches a product distribution asymptotically are strictly sub-optimal.

cs.IT↗

arXiv2025

Rate-Distortion-Perception Trade-off with Strong Realism Constraints: Role of Side Information and Common Randomness

In image compression, with recent advances in generative modeling, existence of a trade-off between the rate and perceptual quality has been brought to light, where the perceptual quality is measured by the closeness of the output and source distributions. We consider the compression of a memoryless source sequence $X^n=(X_1, \ldots, X_n)$ in the presence of memoryless side information $Z^n=(Z_1, \ldots, Z_n),$ originally studied by Wyner and Ziv, but elucidate the impact of a strong perfect realism constraint, which requires the joint distribution of output symbols $Y^n=(Y_1,...,Y_n)$ to match the distribution of the source sequence. We consider two cases: when $Z^n$ is available only at the decoder, or at both the encoder and decoder, and characterize the information theoretic limits under various scenarios. Previous works show the superiority of randomized codes under strong perceptual quality constraints. When $Z^n$ is available at both terminals, we characterize its dual role, as a source of common randomness, and as a second look on the source for the receiver. We also study different notions of strong perfect realism which we call marginal realism, joint realism and near-perfect realism. We derive explicit solutions when $X$ and $Z$ are jointly Gaussian under the squared error distortion measure. In traditional lossy compression, having $Z$ only at the decoder imposes no rate penalty in the Gaussian scenario. We show that, when strong perfect realism constraints are imposed this holds only when sufficient common randomness is available.

cs.IT↗

arXiv2025

Token-Domain Multiple Access: Exploiting Semantic Orthogonality for Collision Mitigation

Token communications is an emerging generative semantic communication concept that reduces transmission rates by using context and transformer-based token processing, with tokens serving as universal semantic units. In this paper, we propose a semantic multiple access scheme in the token domain, referred to as ToDMA, where a large number of devices share a tokenizer and a modulation codebook for source and channel coding, respectively. Specifically, the source signal is tokenized into sequences, with each token modulated into a codeword. Codewords from multiple devices are transmitted simultaneously, resulting in overlap at the receiver. The receiver detects the transmitted tokens, assigns them to their respective sources, and mitigates token collisions by leveraging context and semantic orthogonality across the devices' messages. Simulations demonstrate that the proposed ToDMA framework outperforms context-unaware orthogonal and non-orthogonal communication methods in image transmission tasks, achieving lower latency and better image quality.

cs.IT↗

arXiv2025

Zero-Shot Semantic Communication with Multimodal Foundation Models

Most existing semantic communication (SemCom) systems use deep joint source-channel coding (DeepJSCC) to encode task-specific semantics in a goal-oriented manner. However, their reliance on predefined tasks and datasets significantly limits their flexibility and generalizability in practical deployments. Multi-modal foundation models provide a promising solution by generating universal semantic tokens. Inspired by this, we introduce SemCLIP, a zero-shot SemCom framework leveraging the contrastive language-image pre-training (CLIP) model. By transmitting CLIP-generated image tokens instead of raw images, SemCLIP enables efficient SemCom under low bandwidth and challenging channel conditions, facilitating diverse downstream tasks and zero-shot applications. Specifically, we propose a DeepJSCC scheme for efficient CLIP token encoding. To mitigate potential degradation caused by compression and channel noise, a multi-modal transmission-aware prompt learning mechanism is designed at the receiver, which adapts prompts based on transmission quality, enhancing system robustness and channel adaptability. Simulation results demonstrate that SemCLIP outperforms the baselines, achieving a $41\%$ improvement in zero-shot performance at low signal-to-noise ratios. Meanwhile, SemCLIP reduces bandwidth usage by more than $50$-fold compared to alternative image transmission methods, demonstrating the potential of foundation models towards a generalized, task-agnostic SemCom solution.

eess.SP↗

arXiv2025

State Characterisation of Self-Directed Channel Memristive Devices

Knowing how to reliably use memristors as information storage devices is crucial not only to their role as emerging memories, but also for their application in neural network acceleration and as components of novel neuromorphic systems. In order to better understand the dynamics of information storage on memristors, it is essential to be able to characterise and measure their state. To this end, in this paper we propose a general, physics-inspired modelling approach for characterising the state of self-directed channel (SDC) memristors. Additionally, to enable the identification of the proposed state from device data, we introduce a noise-aware approach to the minimum-variance estimation of the state from voltage and current pairs.

cs.ET↗

arXiv2025

Multi-Terminal Remote Generation and Estimation Over a Broadcast Channel With Correlated Priors

We study the multi-terminal remote estimation problem under a rate constraint, in which the goal of the encoder is to help each decoder estimate a function over a certain distribution -- while the distribution is known only to the encoder, the function to be estimated is known only to the decoders, and can also be different for each decoder. The decoders can observe correlated samples from prior distributions, instantiated through shared randomness with the encoder. To achieve this, we employ remote generation, where the encoder helps decoders generate samples from the underlying distribution by using the samples from the prior through importance sampling. While methods such as minimal random coding can be used to efficiently transmit samples to each decoder individually using their importance scores, it is unknown if the correlation among the samples from the priors can reduce the communication cost using the availability of a broadcast link. We propose a hierarchical importance sampling strategy that facilitates, in the case of non-zero Gács-Körner common information among the priors of the decoders, a common sampling step leveraging the availability of a broadcast channel. This is followed by a refinement step for the individual decoders. We present upper bounds on the bias and the estimation error for unicast transmission, which is of independent interest. We then introduce a method that splits into two phases, dedicated to broadcast and unicast transmission, respectively, and show the reduction in communication cost.

cs.IT↗

arXiv2025

Speculative Sampling via Exponential Races

Speculative decoding accelerates large language model inference using a smaller draft model. In this paper, we establish a surprising connection between speculative decoding and channel simulation, which aims at simulating a noisy channel using as few bits as possible. This connection allows us to provide an information-theoretic analysis of the speed up that can be achieved by speculative decoding. Leveraging this link, we derive an explicit relation between generation speed-up and the number of tokens $k$ generated by the draft model for large $k$, which serves as an upper bound for all $k$. We also propose a novel speculative decoding method via exponential race ERSD that matches state-of-the-art performance.

cs.CL↗

arXiv2025

A Fully Asynchronous Unsourced Random Access Scheme

We investigate fully asynchronous unsourced random access (URA), and propose a high-performing scheme that employs on-off division multiple access (ODMA). In this scheme, active users distribute their data over the transmit block based on a sparse transmission pattern without any limitations on the starting time. At the receiver side, we adopt a double sliding-window decoding approach, utilizing a smaller inner decoding window of two block lengths within a larger outer window to enhance the interference cancellation process. Within the inner window, the receiver iteratively applies preamble-free joint starting time and pattern detection, single-user decoding, and successive interference cancellation operations. A notable feature of the proposed scheme is its elimination of the need for a preamble for starting time detection; this is achieved using ODMA transmission patterns. Numerical results demonstrate that the proposed asynchronous URA scheme outperforms existing alternatives.

eess.SP↗

arXiv2025

Deep Joint Source Channel Coding for Privacy-Aware End-to-End Image Transmission

Deep neural network (DNN)-based joint source and channel coding is proposed for privacy-aware end-to-end image transmission against multiple eavesdroppers. Both scenarios of colluding and non-colluding eavesdroppers are considered. Unlike prior works that assume perfectly known and independent identically distributed (i.i.d.) source and channel statistics, the proposed scheme operates under unknown and non-i.i.d. conditions, making it more applicable to real-world scenarios. The goal is to transmit images with minimum distortion, while simultaneously preventing eavesdroppers from inferring certain private attributes of images. Simultaneously generalizing the ideas of privacy funnel and wiretap coding, a multi-objective optimization framework is expressed that characterizes the tradeoff between image reconstruction quality and information leakage to eavesdroppers, taking into account the structural similarity index (SSIM) for improving the perceptual quality of image reconstruction. Extensive experiments on the CIFAR-10 and CelebA, along with ablation studies, demonstrate significant performance improvements in terms of SSIM, adversarial accuracy, and the mutual information leakage compared to benchmarks. Experiments show that the proposed scheme restrains the adversarially-trained eavesdroppers from intercepting privatized data for both cases of eavesdropping a common secret, as well as the case in which eavesdroppers are interested in different secrets. Furthermore, useful insights on the privacy-utility trade-off are also provided.

cs.IT↗