arXiv subjects

Mikael Skoglund

Publications and source records attributed to Mikael Skoglund.

At least 91 records · Page 5Linked to original sources

arXiv2024

Extended Functional Representation Lemma: A Tool For Privacy, Semantic Representation, Caching, and Compression Design

This paper provides an overview of a problem in information-theoretic privacy mechanism design, addressing two scenarios in which private data is either observable or hidden. In each scenario, different privacy measures are used, including bounded mutual information and two types of per-letter privacy constraints. Considering the first scenario, an agent observes useful data that is correlated with private data, and wants to disclose the useful information to a user. Due to the privacy concerns, direct disclosure is prohibited. Hence, a privacy mechanism is designed to generate disclosed data which maximizes the revealed information about the useful data while satisfying a privacy constraint. In the second scenario, the agent has additionally access to the private data. We discuss how the Functional Representation Lemma, the Strong Functional Representation Lemma, and their extended versions are useful for designing low-complexity privacy mechanisms that achieve optimal privacy-utility trade-offs under certain constraints. Furthermore, another privacy design problem is presented where part of the private attribute is more private than the remaining part. Finally, we provide applications including semantic communications, caching and delivery, and compression designs, where the approach can be applied.

cs.IT↗

arXiv2024

Improving Achievability of Cache-Aided Private Variable-Length Coding with Zero Leakage

A statistical cache-aided compression problem with a privacy constraint is studied, where a server has access to a database of $N$ files, $(Y_1,...,Y_N)$, each of size $F$ bits and is linked through a shared channel to $K$ users, where each has access to a local cache memory of size $MF$ bits. During the placement phase, the server fills the users' caches without prior knowledge of their demands, while the delivery phase takes place after the users send their demands to the server. We assume that each file in database $Y_i$ is arbitrarily correlated with a private attribute $X$, and an adversary is assumed to have access to the shared channel. The users and the server have access to a shared key $W$. The goal is to design the cache contents and the delivered message $\cal C$ such that the average length of $\mathcal{C}$ is minimized, while satisfying: i. The response $\cal C$ does not reveal any information about $X$, i.e., $I(X;\mathcal{C})=0$; ii. User $i$ can decode its demand, $Y_{d_i}$, by using the shared key $W$, $\cal C$, and its local cache $Z_i$. In a previous work, we have proposed a variable-length coding scheme that combines privacy-aware compression with coded caching techniques. In this paper, we propose a new achievability scheme using minimum entropy coupling concept and a greedy entropy-based algorithm. We show that the proposed scheme improves the previous results. Moreover, considering two special cases we improve the obtained bounds using the common information concept.

cs.IT↗

arXiv2024

On Information Theoretic Fairness: Compressed Representations With Perfect Demographic Parity

In this article, we study the fundamental limits in the design of fair and/or private representations achieving perfect demographic parity and/or perfect privacy through the lens of information theory. More precisely, given some useful data $X$ that we wish to employ to solve a task $T$, we consider the design of a representation $Y$ that has no information of some sensitive attribute or secret $S$, that is, such that $I(Y;S) = 0$. We consider two scenarios. First, we consider a design desiderata where we want to maximize the information $I(Y;T)$ that the representation contains about the task, while constraining the level of compression (or encoding rate), that is, ensuring that $I(Y;X) \leq r$. Second, inspired by the Conditional Fairness Bottleneck problem, we consider a design desiderata where we want to maximize the information $I(Y;T|S)$ that the representation contains about the task which is not shared by the sensitive attribute or secret, while constraining the amount of irrelevant information, that is, ensuring that $I(Y;X|T,S) \leq r$. In both cases, we employ extended versions of the Functional Representation Lemma and the Strong Functional Representation Lemma and study the tightness of the obtained bounds. Every result here can also be interpreted as a coding with perfect privacy problem by considering the sensitive attribute as a secret.

cs.IT↗

arXiv2024

An Information-Theoretic Approach to Generalization Theory

We investigate the in-distribution generalization of machine learning algorithms. We depart from traditional complexity-based approaches by analyzing information-theoretic bounds that quantify the dependence between a learning algorithm and the training data. We consider two categories of generalization guarantees: 1) Guarantees in expectation: These bounds measure performance in the average case. Here, the dependence between the algorithm and the data is often captured by information measures. While these measures offer an intuitive interpretation, they overlook the geometry of the algorithm's hypothesis class. Here, we introduce bounds using the Wasserstein distance to incorporate geometry, and a structured, systematic method to derive bounds capturing the dependence between the algorithm and an individual datum, and between the algorithm and subsets of the training data. 2) PAC-Bayesian guarantees: These bounds measure the performance level with high probability. Here, the dependence between the algorithm and the data is often measured by the relative entropy. We establish connections between the Seeger--Langford and Catoni's bounds, revealing that the former is optimized by the Gibbs posterior. We introduce novel, tighter bounds for various types of loss functions. To achieve this, we introduce a new technique to optimize parameters in probabilistic statements. To study the limitations of these approaches, we present a counter-example where most of the information-theoretic bounds fail while traditional approaches do not. Finally, we explore the relationship between privacy and generalization. We show that algorithms with a bounded maximal leakage generalize. For discrete data, we derive new bounds for differentially private algorithms that guarantee generalization even with a constant privacy parameter, which is in contrast to previous bounds in the literature.

stat.ML↗

arXiv2024

Towards Quantum Federated Learning

Quantum Federated Learning (QFL) is an emerging interdisciplinary field that merges the principles of Quantum Computing (QC) and Federated Learning (FL), with the goal of leveraging quantum technologies to enhance privacy, security, and efficiency in the learning process. Currently, there is no comprehensive survey for this interdisciplinary field. This review offers a thorough, holistic examination of QFL. We aim to provide a comprehensive understanding of the principles, techniques, and emerging applications of QFL. We discuss the current state of research in this rapidly evolving field, identify challenges and opportunities associated with integrating these technologies, and outline future directions and open research questions. We propose a unique taxonomy of QFL techniques, categorized according to their characteristics and the quantum techniques employed. As the field of QFL continues to progress, we can anticipate further breakthroughs and applications across various industries, driving innovation and addressing challenges related to data privacy, security, and resource optimization. This review serves as a first-of-its-kind comprehensive guide for researchers and practitioners interested in understanding and advancing the field of QFL.

cs.LG↗

arXiv2024

Simultaneous Information and Energy Transmission with Short Packets and Finite Constellations

This paper characterizes the trade-offs between information and energy transmission over an additive white Gaussian noise channel in the finite block-length regime with finite channel input symbols. These trade-offs are characterized in the form of inequalities involving the information transmission rate, energy transmission rate, decoding error probability (DEP) and energy outage probability (EOP) for a given finite block-length code. The first set of results identify the set of necessary conditions that a given code must satisfy for simultaneous information and energy transmission. Following this, a novel method for constructing a family of codes that can satisfy a target information rate, energy rate, DEP and EOP is proposed. Finally, the achievability results identify the set of tuples of information rate, energy rate, DEP and EOP that can be simultaneously achieved by the constructed family of codes.

cs.IT↗

arXiv2024

Supplementary File: Cooperative Gradient Coding for Semi-Decentralized Federated Learning

Stragglers' effects are known to degrade FL performance. In this paper, we investigate federated learning (FL) over wireless networks in the presence of communication stragglers, where the power-constrained clients collaboratively train a global model by iteratively optimizing a local objective function with their local datasets and transmitting local model updates to the central parameter server (PS) through fading channels. To tackle communication stragglers without dataset sharing or prior information about the network at PS, we propose cooperative gradient coding (CoGC) for semi-decentralized FL to enable the exact global model recovery at PS. Furthermore, we conduct a thorough theoretical analysis of the proposed approach. Namely, an outage analysis of the proposed approach is provided, followed by a convergence analysis based on the failure probability of the global model recovery at PS. Nevertheless, simulation results reveal the superiority of the proposed approach in the presence of stragglers under imbalanced data distribution.

eess.SP↗

arXiv2024

Distribution-Preserving Integrated Sensing and Communication with Secure Reconstruction

Distribution-preserving integrated sensing and communication with secure reconstruction is investigated in this paper. In addition to the distortion constraint, we impose another constraint on the distance between the reconstructed sequence distribution and the original state distribution to force the system to preserve the statistical property of the channel states. An inner bound of the distribution-preserving capacity-distortion region is provided with some capacity region results under special cases. A numerical example demonstrates the tradeoff between the communication rate, reconstruction distortion and distribution preservation. Furthermore, we consider the case that the reconstructed sequence should be kept secret from an eavesdropper who also observes the channel output. An inner bound of the tradeoff region and a capacity-achieving special case are presented.

cs.IT↗

arXiv2024

Subspace-Informed Matrix Completion

In this work, we consider the matrix completion problem, where the objective is to reconstruct a low-rank matrix from a few observed entries. A commonly employed approach involves nuclear norm minimization. For this method to succeed, the number of observed entries needs to scale at least proportional to both the rank of the ground-truth matrix and the coherence parameter. While the only prior information is oftentimes the low-rank nature of the ground-truth matrix, in various real-world scenarios, additional knowledge about the ground-truth low-rank matrix is available. For instance, in collaborative filtering, Netflix problem, and dynamic channel estimation in wireless communications, we have partial or full knowledge about the signal subspace in advance. Specifically, we are aware of some subspaces that form multiple angles with the column and row spaces of the ground-truth matrix. Leveraging this valuable information has the potential to significantly reduce the required number of observations. To this end, we introduce a multi-weight nuclear norm optimization problem that concurrently promotes the low-rank property as well the information about the available subspaces. The proposed weights are tailored to penalize each angle corresponding to each basis of the prior subspace independently. We further propose an optimal weight selection strategy by minimizing the coherence parameter of the ground-truth matrix, which is equivalent to minimizing the required number of observations. Simulation results validate the advantages of incorporating multiple weights in the completion procedure. Specifically, our proposed multi-weight optimization problem demonstrates a substantial reduction in the required number of observations compared to the state-of-the-art methods.

eess.SP↗

arXiv2024

Optimal Transmitter Design and Pilot Spacing in MIMO Non-Stationary Aging Channels

This work considers an uplink wireless communication system where multiple users with multiple antennas transmit data frames over dynamic channels. Previous studies have shown that multiple transmit and receive antennas can substantially enhance the sum-capacity of all users when the channel is known at the transmitter and in the case of uncorrelated transmit and receive antennas. However, spatial correlations stemming from close proximity of transmit antennas and channel variation between pilot and data time slots, known as channel aging, can substantially degrade the transmission rate if they are not properly into account. In this work, we provide an analytical framework to concurrently exploit both of these features. Specifically, we first propose a beamforming framework to capture spatial correlations. Then, based on random matrix theory tools, we introduce a deterministic expression that approximates the average sum-capacity of all users. Subsequently, we obtain the optimal values of pilot spacing and beamforming vectors upon maximizing this expression. Simulation results show the impacts of path loss, velocity of mobile users and Rician factor on the resulting sum-capacity and underscore the efficacy of our methodology compared to prior works.

eess.SP↗

arXiv2024

Improved Downlink Channel Estimation in Time-Varying FDD Massive MIMO Systems

In this work, we address the challenge of accurately obtaining channel state information at the transmitter (CSIT) for frequency division duplexing (FDD) multiple input multiple output systems. Although CSIT is vital for maximizing spatial multiplexing gains, traditional CSIT estimation methods often suffer from impracticality due to the substantial training and feedback overhead they require. To address this challenge, we leverage two sources of prior information simultaneously: the presence of limited local scatterers at the base station (BS) and the time-varying characteristics of the channel. The former results in a redundant angular sparsity of users' channels exceeding the spatial dimension (i.e., the number of BS antennas), while the latter provides a prior non-uniform distribution in the angular domain. We propose a weighted optimization framework that simultaneously reflects both of these features. The optimal weights are then obtained by minimizing the expected recovery error of the optimization problem. This establishes an analytical closed-form relationship between the optimal weights and the angular domain characteristics. Numerical experiments verify the effectiveness of our proposed approach in reducing the recovery error and consequently resulting in decreased training and feedback overhead.

eess.SP↗

arXiv2024

Gradient Coding in Decentralized Learning for Evading Stragglers

In this paper, we consider a decentralized learning problem in the presence of stragglers. Although gradient coding techniques have been developed for distributed learning to evade stragglers, where the devices send encoded gradients with redundant training data, it is difficult to apply those techniques directly to decentralized learning scenarios. To deal with this problem, we propose a new gossip-based decentralized learning method with gradient coding (GOCO). In the proposed method, to avoid the negative impact of stragglers, the parameter vectors are updated locally using encoded gradients based on the framework of stochastic gradient coding and then averaged in a gossip-based manner. We analyze the convergence performance of GOCO for strongly convex loss functions. And we also provide simulation results to demonstrate the superiority of the proposed method in terms of learning performance compared with the baseline methods.

cs.LG↗

arXiv2024

More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validity

In this paper, we present new high-probability PAC-Bayes bounds for different types of losses. Firstly, for losses with a bounded range, we recover a strengthened version of Catoni's bound that holds uniformly for all parameter values. This leads to new fast-rate and mixed-rate bounds that are interpretable and tighter than previous bounds in the literature. In particular, the fast-rate bound is equivalent to the Seeger--Langford bound. Secondly, for losses with more general tail behaviors, we introduce two new parameter-free bounds: a PAC-Bayes Chernoff analogue when the loss' cumulative generating function is bounded, and a bound when the loss' second moment is bounded. These two bounds are obtained using a new technique based on a discretization of the space of possible events for the ``in probability'' parameter optimization problem. This technique is both simpler and more general than previous approaches optimizing over a grid on the parameters' space. Finally, using a simple technique that is applicable to any existing bound, we extend all previous results to anytime-valid bounds.

stat.ML↗

arXiv2024

Multi-Task Private Semantic Communication

We study a multi-task private semantic communication problem, in which an encoder has access to an information source arbitrarily correlated with some latent private data. A user has $L$ tasks with priorities. The encoder designs a message to be revealed which is called the semantic of the information source. Due to the privacy constraints the semantic can not be disclosed directly and the encoder adds noise to produce disclosed data. The goal is to design the disclosed data that maximizes the weighted sum of the utilities achieved by the user while satisfying a privacy constraint on the private data. In this work, we first consider a single-task scenario and design the added noise utilizing various methods including the extended versions of the Functional Representation Lemma, Strong Functional Representation Lemma, and separation technique. We then study the multi-task scenario and derive a simple design of the source semantics. We show that in the multi-task scenario the main problem can be divided into multiple parallel single-task problems.

cs.IT↗

arXiv2024

A note on generalization bounds for losses with finite moments

This paper studies the truncation method from Alquier [1] to derive high-probability PAC-Bayes bounds for unbounded losses with heavy tails. Assuming that the $p$-th moment is bounded, the resulting bounds interpolate between a slow rate $1 / \sqrt{n}$ when $p=2$, and a fast rate $1 / n$ when $p \to \infty$ and the loss is essentially bounded. Moreover, the paper derives a high-probability PAC-Bayes bound for losses with a bounded variance. This bound has an exponentially better dependence on the confidence parameter and the dependency measure than previous bounds in the literature. Finally, the paper extends all results to guarantees in expectation and single-draw PAC-Bayes. In order to so, it obtains analogues of the PAC-Bayes fast rate bound for bounded losses from [2] in these settings.

stat.ML↗

arXiv2024

Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers

This paper considers the problem of distributed learning (DL) in the presence of stragglers. For this problem, DL methods based on gradient coding have been widely investigated, which redundantly distribute the training data to the workers to guarantee convergence when some workers are stragglers. However, these methods require the workers to transmit real-valued vectors during the process of learning, which induces very high communication burden. To overcome this drawback, we propose a novel DL method based on 1-bit gradient coding (1-bit GCDL), where 1-bit data encoded from the locally computed gradients are transmitted by the workers to reduce the communication overhead. We theoretically provide the convergence guarantees of the proposed method for both the convex loss functions and nonconvex loss functions. It is shown empirically that 1-bit GC-DL outperforms the baseline methods, which attains better learning performance under the same communication overhead.

cs.LG↗

arXiv2024

Chained Information-Theoretic bounds and Tight Regret Rate for Linear Bandit Problems

This paper studies the Bayesian regret of a variant of the Thompson-Sampling algorithm for bandit problems. It builds upon the information-theoretic framework of [Russo and Van Roy, 2015] and, more specifically, on the rate-distortion analysis from [Dong and Van Roy, 2020], where they proved a bound with regret rate of $O(d\sqrt{T \log(T)})$ for the $d$-dimensional linear bandit setting. We focus on bandit problems with a metric action space and, using a chaining argument, we establish new bounds that depend on the metric entropy of the action space for a variant of Thompson-Sampling. Under suitable continuity assumption of the rewards, our bound offers a tight rate of $O(d\sqrt{T})$ for $d$-dimensional linear bandit problems.

stat.ML↗

arXiv2024

Quantifying Privacy via Information Density

We examine the relationship between privacy metrics that utilize information density to measure information leakage between a private and a disclosed random variable. Firstly, we prove that bounding the information density from above or below in turn implies a lower or upper bound on the information density, respectively. Using this result, we establish new relationships between local information privacy, asymmetric local information privacy, pointwise maximal leakage and local differential privacy. We further provide applications of these relations to privacy mechanism design. Furthermore, we provide statements showing the equivalence between a lower bound on information density and risk-averse adversaries. More specifically, we prove an equivalence between a guessing framework and a cost-function framework that result in the desired lower bound on the information density.

cs.IT↗