arXiv subjects

Shuo Shao

Publications and source records attributed to Shuo Shao.

At least 55 records · Page 3Linked to original sources

arXiv2024

Joint Data and Semantics Lossy Compression: Nonasymptotic Converse Bounds and Second-Order Asymptotics

This paper studies the joint data and semantics lossy compression problem, i.e., an extension of the hidden lossy source coding problem that entails recovering both the hidden and observable sources. We aim to study the nonasymptotic and second-order properties of this problem, especially the converse aspect. Specifically, we begin by deriving general nonasymptotic converse bounds valid for general sources and distortion measures, utilizing properties of distortion-tilted information. Subsequently, a second-order converse bound is derived under the standard block coding setting through asymptotic analysis of the nonasymptotic bounds. This bound is tight since it coincides with a known second-order achievability bound. We then examine the case of erased fair coin flips (EFCF), providing its specific nonasymptotic achievability and converse bounds. Numerical results under the EFCF case demonstrate that our second-order asymptotic approximation effectively approximates the optimum rate at given blocklengths.

cs.IT↗

arXiv2024

Joint Coding-Modulation for Digital Semantic Communications via Variational Autoencoder

Semantic communications have emerged as a new paradigm for improving communication efficiency by transmitting the semantic information of a source message that is most relevant to a desired task at the receiver. Most existing approaches typically utilize neural networks (NNs) to design end-to-end semantic communication systems, where NN-based semantic encoders output continuously distributed signals to be sent directly to the channel in an analog fashion. In this work, we propose a joint coding-modulation (JCM) framework for digital semantic communications by using variational autoencoder (VAE). Our approach learns the transition probability from source data to discrete constellation symbols, thereby avoiding the non-differentiability problem of digital modulation. Meanwhile, by jointly designing the coding and modulation process together, we can match the obtained modulation strategy with the operating channel condition. We also derive a matching loss function with information-theoretic meaning for end-to-end training. Experiments on image semantic communication validate the superiority of our proposed JCM framework over the state-of-the-art quantization-based digital semantic coding-modulation methods across a wide range of channel conditions, transmission rates, and modulation orders. Furthermore, its performance gap to analog semantic communication reduces as the modulation order increases while enjoying the hardware implementation convenience.

cs.IT↗

arXiv2023

Watermarking in Secure Federated Learning: A Verification Framework Based on Client-Side Backdooring

Federated learning (FL) allows multiple participants to collaboratively build deep learning (DL) models without directly sharing data. Consequently, the issue of copyright protection in FL becomes important since unreliable participants may gain access to the jointly trained model. Application of homomorphic encryption (HE) in secure FL framework prevents the central server from accessing plaintext models. Thus, it is no longer feasible to embed the watermark at the central server using existing watermarking schemes. In this paper, we propose a novel client-side FL watermarking scheme to tackle the copyright protection issue in secure FL with HE. To our best knowledge, it is the first scheme to embed the watermark to models under the Secure FL environment. We design a black-box watermarking scheme based on client-side backdooring to embed a pre-designed trigger set into an FL model by a gradient-enhanced embedding method. Additionally, we propose a trigger set construction mechanism to ensure the watermark cannot be forged. Experimental results demonstrate that our proposed scheme delivers outstanding protection performance and robustness against various watermark removal attacks and ambiguity attack.

cs.CR↗

arXiv2023

Excess Distortion Exponent Analysis for Semantic-Aware MIMO Communication Systems

In this paper, the analysis of excess distortion exponent for joint source-channel coding (JSCC) in semantic-aware communication systems is presented. By introducing an unobservable semantic source, we extend the classical results by Csiszar to semantic-aware communication systems. Both upper and lower bounds of the exponent for the discrete memoryless source-channel pair are established. Moreover, an extended achievable bound of the excess distortion exponent for MIMO systems is derived. Further analysis explores how the block fading and numbers of antennas influence the exponent of semanticaware MIMO systems. Our results offer some theoretical bounds of error decay performance and can be used to guide future semantic communications with joint source-channel coding scheme.

cs.IT↗

arXiv2022

Learning Based Joint Coding-Modulation for Digital Semantic Communication Systems

In learning-based semantic communications, neural networks have replaced different building blocks in traditional communication systems. However, the digital modulation still remains a challenge for neural networks. The intrinsic mechanism of neural network based digital modulation is mapping continuous output of the neural network encoder into discrete constellation symbols, which is a non-differentiable function that cannot be trained with existing gradient descend algorithms. To overcome this challenge, in this paper we develop a joint coding-modulation scheme for digital semantic communications with BPSK modulation. In our method, the neural network outputs the likelihood of each constellation point, instead of having a concrete mapping. A random code rather than a deterministic code is hence used, which preserves more information for the symbols with a close likelihood on each constellation point. The joint coding-modulation design can match the modulation process with channel states, and hence improve the performance of digital semantic communications. Experiment results show that our method outperforms existing digital modulation methods in semantic communications over a wide range of SNR, and outperforms neural network based analog modulation method in low SNR regime.

cs.IT↗

arXiv2022

Energy Efficiency of Massive Random Access in MIMO Quasi-Static Rayleigh Fading Channels with Finite Blocklength

This paper considers the massive random access problem in MIMO quasi-static Rayleigh fading channels. Specifically, we derive achievability and converse bounds on the minimum energy-per-bit required for each active user to transmit $J$ bits with blocklength $n$ and power $P$ under a per-user probability of error (PUPE) constraint, in the cases with and without \emph{a priori} channel state information at the receiver (CSIR and no-CSI). In the case of no-CSI, we consider both the settings with and without knowing the number $K_a$ of active users. The achievability bounds rely on the design of the ``good region''. Numerical evaluation shows the gap between achievability and converse bounds is less than $2.5$ dB in the CSIR case and less than $4$ dB in the no-CSI case in most considered regimes. When the distribution of $K_a$ is known, the performance gap between the cases with and without knowing the value of $K_a$ is small. For example, in the setup with blocklength $n=1000$, payload $J=100$, error requirement $ε=0.001$, and $L=128$ receive antennas, compared to the case with known $K_a$, the extra required energy-per-bit when $K_a$ is unknown and distributed as $K_a\sim\text{Binom}(K,0.4)$ is less than $0.3$ dB on the converse side and $1.1$ dB on the achievability side. The spectral efficiency grows approximately linearly with $L$ in the CSIR case, whereas the growth rate decreases with no-CSI. Moreover, we study the performance of a pilot-assisted scheme, which is suboptimal especially when $K_a$ is large. Building on non-asymptotic results, when all users are active and $J=Θ(1)$, we obtain scaling laws as follows: when $L=Θ\left(n^2\right)$ and $P=Θ\left(\frac{1}{n^2}\right)$, one can reliably serve $K=\mathcal{O}(n^2)$ users with no-CSI; under mild conditions with CSIR, the PUPE requirement is satisfied if and only if $\frac{nL\ln KP}{K}=Ω\left(1\right)$.

cs.IT↗

arXiv2022

Deep Learning-Enabled Semantic Communication Systems with Task-Unaware Transmitter and Dynamic Data

Existing deep learning-enabled semantic communication systems often rely on shared background knowledge between the transmitter and receiver that includes empirical data and their associated semantic information. In practice, the semantic information is defined by the pragmatic task of the receiver and cannot be known to the transmitter. The actual observable data at the transmitter can also have non-identical distribution with the empirical data in the shared background knowledge library. To address these practical issues, this paper proposes a new neural network-based semantic communication system for image transmission, where the task is unaware at the transmitter and the data environment is dynamic. The system consists of two main parts, namely the semantic coding (SC) network and the data adaptation (DA) network. The SC network learns how to extract and transmit the semantic information using a receiver-leading training process. By using the domain adaptation technique from transfer learning, the DA network learns how to convert the data observed into a similar form of the empirical data that the SC network can process without retraining. Numerical experiments show that the proposed method can be adaptive to observable datasets while keeping high performance in terms of both data recovery and task execution.

cs.IT↗

arXiv2022

An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation

A new source model, which consists of an intrinsic state part and an extrinsic observation part, is proposed and its information-theoretic characterization, namely its rate-distortion function, is defined and analyzed. Such a source model is motivated by the recent surge of interest in the semantic aspect of information: the intrinsic state corresponds to the semantic feature of the source, which in general is not observable but can only be inferred from the extrinsic observation. There are two distortion measures, one between the intrinsic state and its reproduction, and the other between the extrinsic observation and its reproduction. Under a given code rate, the tradeoff between these two distortion measures is characterized by the rate-distortion function, which is solved via the indirect rate-distortion theory and is termed as the semantic rate-distortion function of the source. As an application of the general model and its analysis, the case of Gaussian extrinsic observation is studied, assuming a linear relationship between the intrinsic state and the extrinsic observation, under a quadratic distortion structure. The semantic rate-distortion function is shown to be the solution of a convex programming programming problem with respect to an error covariance matrix, and a reverse water-filling type of solution is provided when the model further satisfies a diagonalizability condition.

cs.IT↗

arXiv2021

On the Fundamental Limits of Coded Caching Systems with Restricted Demand Types

Caching is a technique to reduce the communication load in peak hours by prefetching contents during off-peak hours. An information-theoretic framework for coded caching was introduced by Maddah-Ali and Niesen in a recent work, where it was shown that significant improvement can be obtained compared to uncoded caching. Considerable efforts have been devoted to identify the precise information-theoretic fundamental limits of the coded caching systems, however the difficulty of this task has also become clear. One of the reasons for this difficulty is that the original coded caching setting allows all possible multiple demand types during delivery, which in fact introduces tension in the coding strategy. In this paper, we seek to develop a better understanding of the fundamental limits of coded caching by investigating systems with certain demand type restrictions. We first consider the canonical three-user three-file system, and show that, contrary to popular beliefs, the worst demand type is not the one in which all three files are requested. Motivated by these findings, we focus on coded caching systems where every file must be requested by at least one user. A novel coding scheme is proposed, which can provide new operating points that are not covered by any previously known schemes.

cs.IT↗

arXiv2021

Cascading Bandit under Differential Privacy

This paper studies \emph{differential privacy (DP)} and \emph{local differential privacy (LDP)} in cascading bandits. Under DP, we propose an algorithm which guarantees $ε$-indistinguishability and a regret of $\mathcal{O}((\frac{\log T}ε)^{1+ξ})$ for an arbitrarily small $ξ$. This is a significant improvement from the previous work of $\mathcal{O}(\frac{\log^3 T}ε)$ regret. Under ($ε$,$δ$)-LDP, we relax the $K^2$ dependence through the tradeoff between privacy budget $ε$ and error probability $δ$, and obtain a regret of $\mathcal{O}(\frac{K\log (1/δ) \log T}{ε^2})$, where $K$ is the size of the arm subset. This result holds for both Gaussian mechanism and Laplace mechanism by analyses on the composition. Our results extend to combinatorial semi-bandit. We show respective lower bounds for DP and LDP cascading bandits. Extensive experiments corroborate our theoretic findings.

cs.LG↗

arXiv2021

Conservative Contextual Combinatorial Cascading Bandit

Conservative mechanism is a desirable property in decision-making problems which balance the tradeoff between the exploration and exploitation. We propose the novel \emph{conservative contextual combinatorial cascading bandit ($C^4$-bandit)}, a cascading online learning game which incorporates the conservative mechanism. At each time step, the learning agent is given some contexts and has to recommend a list of items but not worse than the base strategy and then observes the reward by some stopping rules. We design the $C^4$-UCB algorithm to solve the problem and prove its n-step upper regret bound for two situations: known baseline reward and unknown baseline reward. The regret in both situations can be decomposed into two terms: (a) the upper bound for the general contextual combinatorial cascading bandit; and (b) a constant term for the regret from the conservative mechanism. We also improve the bound of the conservative contextual combinatorial bandit as a by-product. Experiments on synthetic data demonstrate its advantages and validate our theoretical analysis.

cs.LG↗

arXiv2020

Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip

Although Shannon theory states that it is asymptotically optimal to separate the source and channel coding as two independent processes, in many practical communication scenarios this decomposition is limited by the finite bit-length and computational power for decoding. Recently, neural joint source-channel coding (NECST) is proposed to sidestep this problem. While it leverages the advancements of amortized inference and deep learning to improve the encoding and decoding process, it still cannot always achieve compelling results in terms of compression and error correction performance due to the limited robustness of its learned coding networks. In this paper, motivated by the inherent connections between neural joint source-channel coding and discrete representation learning, we propose a novel regularization method called Infomax Adversarial-Bit-Flip (IABF) to improve the stability and robustness of the neural joint source-channel coding scheme. More specifically, on the encoder side, we propose to explicitly maximize the mutual information between the codeword and data; while on the decoder side, the amortized reconstruction is regularized within an adversarial framework. Extensive experiments conducted on various real-world datasets evidence that our IABF can achieve state-of-the-art performances on both compression and error correction benchmarks and outperform the baselines by a significant margin.

cs.LG↗

arXiv2020

Symmetric uncoded caching schemes with low subpacketization levels

Caching is a commonly used technique in content-delivery networks which aims to deliver information from hosting servers to users in the most efficient way. In 2014, Maddah-Ali and Niessen formulated caching into a formal information theoretic problem and it has gained a lot of attention since then. It is known that the caching schemes proposed by Ali-Niesen and Yu et. al. are optimal, that is, they require the least number of transmissions from the server to satisfy all users' demands. However for these schemes to work, each file needs to be partitioned into $F^*$ subfiles ($F^*$ is called the subpacketization level of files) with $F^*$ growing exponentially in the number $K$ of users. As a result, it is problematic to apply these schemes in practical situations, where $K$ tends to be very large. There rise the following questions: (1) are there optimal schemes in which each file is partitioned into $F$ subfiles, where $F$ is not exponential, say polynomial for example, in $K$? (2) if the answer to this question is no, is there a near-optimal scheme, a scheme which is as asymptotically good as the one in \cite{ali1,yu}, with $F$ polynomial in $K$? Both these questions are open. Our main contribution in this paper is to provide answers to above questions. Firstly, we prove that under some mild restriction on user's cache rate, there are no optimal schemes with $F$ smaller than $F^*$. Moreover, we give necessary and sufficient conditions for the existence of optimal schemes in this case. Secondly, we provide an affirmative answer to the second question raised above by an explicit construction and a detailed performance analysis.

cs.IT↗

arXiv2019

Improving Unsupervised Domain Adaptation with Variational Information Bottleneck

Domain adaptation aims to leverage the supervision signal of source domain to obtain an accurate model for target domain, where the labels are not available. To leverage and adapt the label information from source domain, most existing methods employ a feature extracting function and match the marginal distributions of source and target domains in a shared feature space. In this paper, from the perspective of information theory, we show that representation matching is actually an insufficient constraint on the feature space for obtaining a model with good generalization performance in target domain. We then propose variational bottleneck domain adaptation (VBDA), a new domain adaptation method which improves feature transferability by explicitly enforcing the feature extractor to ignore the task-irrelevant factors and focus on the information that is essential to the task of interest for both source and target domains. Extensive experimental results demonstrate that VBDA significantly outperforms state-of-the-art methods across three domain adaptation benchmark datasets.

cs.LG↗

arXiv2019

Secure Coded Caching with Colluding Users

In a secure coded caching system, a central server balances the traffic flow between peak and off-peak periods by distributing some public data to the users' caches in advance. Meanwhile, these data are securely protected against the possible colluding users, who might share their cache. We model the system as a flow network and study its capacity region via a network information-theoretic approach. Due to the difficulty of characterizing the capacity region straightforwardly, our approach is two folded from the perspective of network information theory. On one hand, we identify the inner bound of capacity region by proposing a coded caching scheme to achieve a low load secure data transmission. On the other hand, we also establish outer outer bounds on the capacity region, which show that our proposed scheme is order optimal in general under specific circumstance.

cs.IT↗

arXiv2019

On Geometric Structure of Activation Spaces in Neural Networks

In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex hull of massive points in high dimensional space. Based on this new algorithm, four common geometric properties shared by the activation spaces are concluded, which gives a rather clear description of the activation spaces. We then propose an alternative classification method grounding on the geometric structure description, which works better than neural networks alone. Surprisingly, this data classification method can be an indicator of overfitting in neural networks. We believe our work reveals several critical intrinsic properties of modern neural networks and further gives a new metric for evaluating them.

cs.LG↗

arXiv2018

New Results on Multilevel Diversity Coding with Secure Regeneration

The problem of multilevel diversity coding with secure regeneration is revisited. Under the assumption that the eavesdropper can access the repair data for all compromised storage nodes, Shao el al. provided a precise characterization of the minimum-bandwidth-regeneration (MBR) point of the achievable normalized storage-capacity repair-bandwidth tradeoff region. In this paper, it is shown that the MBR point of the achievable normalized storage-capacity repair-bandwidth tradeoff region remains the same even if we assume that the eavesdropper can access the repair data for some compromised storage nodes (type 2 compromised nodes) but only the data contents of the remaining compromised nodes (type 1 compromised nodes), as long as the number of type 1 compromised nodes is no greater than that of type 2 compromised nodes.

cs.IT↗

arXiv2017

Multilevel Diversity Coding with Secure Regeneration: Separate Coding Achieves the MBR Point

The problem of multilevel diversity coding with secure regeneration (MDC-SR) is considered, which includes the problems of multilevel diversity coding with regeneration (MDC-R) and secure regenerating code (SRC) as special cases. Two outer bounds are established, showing that separate coding of different messages using the respective SRCs can achieve the minimum-bandwidth-regeneration (MBR) point of the achievable normalized storage-capacity repair-bandwidth tradeoff regions for the general MDC-SR problem. The core of the new converse results is an exchange lemma, which can be established using Han's subset inequality.

cs.IT↗