Searcharxiv⌕ Search

arXiv subjects

Marios Kountouris

Publications and source records attributed to Marios Kountouris.

At least 73 records · Page 4Linked to original sources
arXiv2024

Version Innovation Age and Age of Incorrect Version for Monitoring Markovian Sources

In this paper, we propose two new performance metrics, coined the Version Innovation Age (VIA) and the Age of Incorrect Version (AoIV) for real-time monitoring of a two-state Markov process over an unreliable channel. We analyze their performance under the change-aware, semantics-aware, and randomized stationary sampling and transmission policies. We derive closed-form expressions for the distribution and the average of VIA, AoIV, and AoII for these policies. We then formulate and solve an optimization problem to minimize the average VIA, subject to constraints on the time-averaged sampling cost and time-averaged reconstruction error. Finally, we compare the performance of various sampling and transmission policies and identify the conditions under which each policy outperforms the others in optimizing the proposed metrics.

cs.IT↗
arXiv2024

Copula-based Estimation of Continuous Sources for a Class of Constrained Rate-Distortion-Functions

We present a new method to estimate the rate-distortion-perception function in the perfect realism regime (PR-RDPF), for multivariate continuous sources subject to a single-letter average distortion constraint. The proposed approach is not only able to solve the specific problem but also two related problems: the entropic optimal transport (EOT) and the output-constrained rate-distortion function (OC-RDF), of which the PR-RDPF represents a special case. Using copula distributions, we show that the OC-RDF can be cast as an I-projection problem on a convex set, based on which we develop a parametric solution of the optimal projection proving that its parameters can be estimated, up to an arbitrary precision, via the solution of a convex program. Subsequently, we propose an iterative scheme via gradient methods to estimate the convex program. Lastly, we characterize a Shannon lower bound (SLB) for the PR-RDPF under a mean squared error (MSE) distortion constraint. We support our theoretical findings with numerical examples by assessing the estimation performance of our iterative scheme using the PR-RDPF with the obtained SLB for various sources.

cs.IT↗
arXiv2023

On the Computation of the Gaussian Rate-Distortion-Perception Function

In this paper, we study the computation of the rate-distortion-perception function (RDPF) for a multivariate Gaussian source under mean squared error (MSE) distortion and, respectively, Kullback-Leibler divergence, geometric Jensen-Shannon divergence, squared Hellinger distance, and squared Wasserstein-2 distance perception metrics. To this end, we first characterize the analytical bounds of the scalar Gaussian RDPF for the aforementioned divergence functions, also providing the RDPF-achieving forward "test-channel" realization. Focusing on the multivariate case, we establish that, for tensorizable distortion and perception metrics, the optimal solution resides on the vector space spanned by the eigenvector of the source covariance matrix. Consequently, the multivariate optimization problem can be expressed as a function of the scalar Gaussian RDPFs of the source marginals, constrained by global distortion and perception levels. Leveraging this characterization, we design an alternating minimization scheme based on the block nonlinear Gauss-Seidel method, which optimally solves the problem while identifying the Gaussian RDPF-achieving realization. Furthermore, the associated algorithmic embodiment is provided, as well as the convergence and the rate of convergence characterization. Lastly, for the "perfect realism" regime, the analytical solution for the multivariate Gaussian RDPF is obtained. We corroborate our results with numerical simulations and draw connections to existing results.

cs.IT↗
arXiv2023

State-aware Real-time Tracking and Remote Reconstruction of a Markov Source

The problem of real-time remote tracking and reconstruction of a two-state Markov process is considered here. A transmitter sends samples from an observed information source to a remote monitor over an unreliable wireless channel. The receiver, in turn, performs an action according to the state of the reconstructed source. We propose a state-aware randomized stationary sampling and transmission policy which accounts for the importance of different states of the information source, and their impact on the goal of the communication process. We then analyze the performance of the proposed policy, and compare it with existing goal-oriented joint sampling and transmission policies, with respect to a set of performance metrics. Specifically, we study the real-time reconstruction error, the cost of actuation error, the consecutive error, and a new metric, coined importance-aware consecutive error. In addition, we formulate and solve a constrained optimization problem that aims to obtain the optimal sampling probabilities that minimize the average cost of actuation error. Our results show that in the scenario of constrained sampling generation, the optimal state-aware randomized stationary policy outperforms all other sampling policies for fast evolving sources, and, under certain conditions, for slowly varying sources. Otherwise, a semantics-aware policy performs better only when the source is slowly varying.

cs.IT↗
arXiv2023

Towards Disentangling Information Paths with Coded ResNeXt

The conventional, widely used treatment of deep learning models as black boxes provides limited or no insights into the mechanisms that guide neural network decisions. Significant research effort has been dedicated to building interpretable models to address this issue. Most efforts either focus on the high-level features associated with the last layers, or attempt to interpret the output of a single layer. In this paper, we take a novel approach to enhance the transparency of the function of the whole network. We propose a neural network architecture for classification, in which the information that is relevant to each class flows through specific paths. These paths are designed in advance before training leveraging coding theory and without depending on the semantic similarities between classes. A key property is that each path can be used as an autonomous single-purpose model. This enables us to obtain, without any additional training and for any class, a lightweight binary classifier that has at least $60\%$ fewer parameters than the original network. Furthermore, our coding theory based approach allows the neural network to make early predictions at intermediate layers during inference, without requiring its full evaluation. Remarkably, the proposed architecture provides all the aforementioned properties while improving the overall accuracy. We demonstrate these properties on a slightly modified ResNeXt model tested on CIFAR-10/100 and ImageNet-1k.

cs.CV↗
arXiv2023

Blind Asynchronous Goal-Oriented Detection for Massive Connectivity

Resource allocation and multiple access schemes are instrumental for the success of communication networks, which facilitate seamless wireless connectivity among a growing population of uncoordinated and non-synchronized users. In this paper, we present a novel random access scheme that addresses one of the most severe barriers of current strategies to achieve massive connectivity and ultra-reliable and low latency communications for 6G. The proposed scheme utilizes wireless channels' angular continuous group-sparsity feature to provide low latency, high reliability, and massive access features in the face of limited time-bandwidth resources, asynchronous transmissions, and preamble errors. Specifically, a reconstruction-free goal-oriented optimization problem is proposed, which preserves the angular information of active devices and is then complemented by a clustering algorithm to assign active users to specific groups. This allows us to identify active stationary devices according to their line of sight angles. Additionally, for mobile devices, an alternating minimization algorithm is proposed to recover their preamble, data, and channel gains simultaneously, enabling the identification of active mobile users. Simulation results show that the proposed algorithm provides excellent performance and supports a massive number of devices. Moreover, the performance of the proposed scheme is independent of the total number of devices, distinguishing it from other random access schemes. The proposed method provides a unified solution to meet the requirements of machine-type communications and ultra-reliable and low-latency communications, making it an important contribution to the emerging 6G networks.

eess.SP↗
arXiv2023

Indirect Rate Distortion Functions with $f$-Separable Distortion Criterion

We consider a remote source coding problem subject to a {distortion function}. Contrary to the use of the classical separable distortion criterion, herein we consider the more general, $f$-separable distortion measure and study its implications on the characterization of the minimum achievable rates (also called $f$-separable indirect rate distortion function (iRDF)) under both excess and average distortion constraints. First, we provide a single-letter characterization of the optimal rates subject to an excess distortion using properties of the $f$-separable distortion. Our main result is a single-letter characterization of the $f$-separable iRDF subject to an average distortion constraint. As a consequence of the previous results, we also show a series of equalities that hold using either indirect or classical RDF under $f$-separable excess or average distortions. We corroborate our results with two application examples in which new closed-form solutions are derived, and based on these, we also recover known special cases.

cs.IT↗
arXiv2023

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

In this paper, we study the computation of the rate-distortion-perception function (RDPF) for discrete memoryless sources subject to a single-letter average distortion constraint and a perception constraint that belongs to the family of f-divergences. For that, we leverage the fact that RDPF, assuming mild regularity conditions on the perception constraint, forms a convex programming problem. We first develop parametric characterizations of the optimal solution and utilize them in an alternating minimization approach for which we prove convergence guarantees. The resulting structure of the iterations of the alternating minimization approach renders the implementation of a generalized Blahut-Arimoto (BA) type of algorithm infeasible. To overcome this difficulty, we propose a relaxed formulation of the structure of the iterations in the alternating minimization approach, which allows for the implementation of an approximate iterative scheme. This approximation is shown, via the derivation of necessary and sufficient conditions, to guarantee convergence to a globally optimal solution. We also provide sufficient conditions on the distortion and the perception constraints which guarantee that our algorithm converges exponentially fast. We corroborate our theoretical results with numerical simulations, and we draw connections with existing results.

cs.IT↗
arXiv2023

Robust PAC$^m$: Training Ensemble Models Under Misspecification and Outliers

Standard Bayesian learning is known to have suboptimal generalization capabilities under misspecification and in the presence of outliers. PAC-Bayes theory demonstrates that the free energy criterion minimized by Bayesian learning is a bound on the generalization error for Gibbs predictors (i.e., for single models drawn at random from the posterior) under the assumption of sampling distributions uncontaminated by outliers. This viewpoint provides a justification for the limitations of Bayesian learning when the model is misspecified, requiring ensembling, and when data is affected by outliers. In recent work, PAC-Bayes bounds -- referred to as PAC$^m$ -- were derived to introduce free energy metrics that account for the performance of ensemble predictors, obtaining enhanced performance under misspecification. This work presents a novel robust free energy criterion that combines the generalized logarithm score function with PAC$^m$ ensemble bounds. The proposed free energy training criterion produces predictive distributions that are able to concurrently counteract the detrimental effects of misspecification -- with respect to both likelihood and prior distribution -- and outliers.

cs.LG↗
arXiv2023

Real-time Remote Reconstruction of a Markov Source and Actuation over Wireless

In this work, we study the problem of real-time tracking and reconstruction of an information source with the purpose of actuation. A device monitors an $N$-state Markov process and transmits status updates to a receiver over a wireless erasure channel. We consider a set of joint sampling and transmission policies, including a semantics-aware one, and we study their performance with respect to relevant metrics. Specifically, we investigate the real-time reconstruction error and its variance, the consecutive error, the cost of memory error, and the cost of actuation error. Furthermore, we propose a randomized stationary sampling and transmission policy and derive closed-form expressions for all aforementioned metrics. We then formulate an optimization problem for minimizing the real-time reconstruction error subject to a sampling cost constraint. Our results show that in the scenario of constrained sampling generation, the optimal randomized stationary policy outperforms all other sampling policies when the source is rapidly evolving. Otherwise, the semantics-aware policy performs the best.

cs.IT↗
arXiv2023

Goal-oriented Policies for Cost of Actuation Error Minimization in Wireless Autonomous Systems

We consider the minimization of the cost of actuation error under resource constraints for real-time tracking in wireless autonomous systems. A transmitter monitors the state of a discrete random process and sends updates to the receiver over an unreliable wireless channel. The receiver takes actions according to the estimated state of the source. For each discrepancy between the real state of the source and the estimated one, we consider a different cost of actuation error. This models the case where some states, and consequently the corresponding actions to be taken, are more important than others. We provide two algorithms: one reaching an optimal solution but of high complexity, and one providing a suboptimal solution but with low complexity. The performance of the two algorithms are quite close as shown by the simulations.

cs.NI↗
arXiv2023

When is Importance Weighting Correction Needed for Covariate Shift Adaptation?

This paper investigates when the importance weighting (IW) correction is needed to address covariate shift, a common situation in supervised learning where the input distributions of training and test data differ. Classic results show that the IW correction is needed when the model is parametric and misspecified. In contrast, recent results indicate that the IW correction may not be necessary when the model is nonparametric and well-specified. We examine the missing case in the literature where the model is nonparametric and misspecified, and show that the IW correction is needed for obtaining the best approximation of the true unknown function for the test distribution. We do this by analyzing IW-corrected kernel ridge regression, covering a variety of settings, including parametric and nonparametric models, well-specified and misspecified settings, and arbitrary weighting functions.

stat.ML↗
arXiv2023

Real-time Reconstruction of Markov Sources and Remote Actuation over Wireless Channels

In this work, we study the real-time tracking and reconstruction of an information source with the purpose of actuation. A device monitors the state of the information source and transmits status updates to a receiver over a wireless erasure channel. We consider two models for the source, namely an $N$-state Markov chain and an $N$-state Birth-Death Markov process. We investigate several joint sampling and transmission policies, including a semantics-aware one, and we study their performance with respect to a set of metrics. Specifically, we investigate the real-time reconstruction error and its variance, the cost of actuation error, the consecutive error, and the cost of memory error. These metrics capture different characteristics of the system performance, such as the impact of erroneous actions and the timing of errors. In addition, we propose a randomized stationary sampling and transmission policy and we derive closed-form expressions for the aforementioned metrics. We then formulate two optimization problems. The first optimization problem aims to minimize the time-averaged reconstruction error subject to time-averaged sampling cost constraint. Then, we compare the optimal randomized stationary policy with uniform, change-aware, and semantics-aware sampling policies. Our results show that in the scenario of constrained sampling generation, the optimal randomized stationary policy outperforms all other sampling policies when the source is rapidly evolving. Otherwise, the semantics-aware policy performs the best. The objective of the second optimization problem is to obtain an optimal sampling policy that minimizes the average consecutive error with a constraint on the time-averaged sampling cost. Based on this, we propose a \emph{wait-then-generate} sampling policy which is simple to implement.

cs.IT↗
arXiv2023

Personalized Decentralized Federated Learning with Knowledge Distillation

Personalization in federated learning (FL) functions as a coordinator for clients with high variance in data or behavior. Ensuring the convergence of these clients' models relies on how closely users collaborate with those with similar patterns or preferences. However, it is generally challenging to quantify similarity under limited knowledge about other users' models given to users in a decentralized network. To cope with this issue, we propose a personalized and fully decentralized FL algorithm, leveraging knowledge distillation techniques to empower each device so as to discern statistical distances between local models. Each client device can enhance its performance without sharing local data by estimating the similarity between two intermediate outputs from feeding local samples as in knowledge distillation. Our empirical studies demonstrate that the proposed algorithm improves the test accuracy of clients in fewer iterations under highly non-independent and identically distributed (non-i.i.d.) data distributions and is beneficial to agents with small datasets, even without the need for a central server.

cs.LG↗
arXiv2023

Communication-Efficient Distributionally Robust Decentralized Learning

Decentralized learning algorithms empower interconnected devices to share data and computational resources to collaboratively train a machine learning model without the aid of a central coordinator. In the case of heterogeneous data distributions at the network nodes, collaboration can yield predictors with unsatisfactory performance for a subset of the devices. For this reason, in this work, we consider the formulation of a distributionally robust decentralized learning task and we propose a decentralized single loop gradient descent/ascent algorithm (AD-GDA) to directly solve the underlying minimax optimization problem. We render our algorithm communication-efficient by employing a compressed consensus scheme and we provide convergence guarantees for smooth convex and non-convex loss functions. Finally, we corroborate the theoretical findings with empirical results that highlight AD-GDA's ability to provide unbiased predictors and to greatly improve communication efficiency compared to existing distributionally robust algorithms.

cs.LG↗
arXiv2023

Multi-User Distributed Computing Via Compressed Sensing

The multi-user linearly-separable distributed computing problem is considered here, in which $N$ servers help to compute the real-valued functions requested by $K$ users, where each function can be written as a linear combination of up to $L$ (generally non-linear) subfunctions. Each server computes a fraction $γ$ of the subfunctions, then communicates a function of its computed outputs to some of the users, and then each user collects its received data to recover its desired function. Our goal is to bound the ratio between the computation workload done by all servers over the number of datasets. To this end, we here reformulate the real-valued distributed computing problem into a matrix factorization problem and then into a basic sparse recovery problem, where sparsity implies computational savings. Building on this, we first give a simple probabilistic scheme for subfunction assignment, which allows us to upper bound the optimal normalized computation cost as $γ\leq \frac{K}{N}$ that a generally intractable $\ell_0$-minimization would give. To bypass the intractability of such optimal scheme, we show that if these optimal schemes enjoy $γ\leq - r\frac{K}{N}W^{-1}_{-1}(- \frac{2K}{e N r} )$ (where $W_{-1}(\cdot)$ is the Lambert function and $r$ calibrates the communication between servers and users), then they can actually be derived using a tractable Basis Pursuit $\ell_1$-minimization. This newly-revealed connection between distributed computation and compressed sensing opens up the possibility of designing practical distributed computing algorithms by employing tools and methods from compressed sensing.

cs.IT↗
arXiv2022

Goal-Oriented Communications for the IoT and Application to Data Compression

Internet of Things (IoT) devices will play an important role in emerging applications, since their sensing, actuation, processing, and wireless communication capabilities stimulate data collection, transmission and decision processes of smart applications. However, new challenges arise from the widespread popularity of IoT devices, including the need for processing more complicated data structures and high dimensional data/signals. The unprecedented volume, heterogeneity, and velocity of IoT data calls for a communication paradigm shift from a search for accuracy or fidelity to semantics extraction and goal accomplishment. In this paper, we provide a partial but insightful overview of recent research efforts in this newly formed area of goal-oriented (GO) and semantic communications, focusing on the problem of GO data compression for IoT applications.

eess.SP↗
arXiv2022

Robust Bayesian Learning for Reliable Wireless AI: Framework and Applications

This work takes a critical look at the application of conventional machine learning methods to wireless communication problems through the lens of reliability and robustness. Deep learning techniques adopt a frequentist framework, and are known to provide poorly calibrated decisions that do not reproduce the true uncertainty caused by limitations in the size of the training data. Bayesian learning, while in principle capable of addressing this shortcoming, is in practice impaired by model misspecification and by the presence of outliers. Both problems are pervasive in wireless communication settings, in which the capacity of machine learning models is subject to resource constraints and training data is affected by noise and interference. In this context, we explore the application of the framework of robust Bayesian learning. After a tutorial-style introduction to robust Bayesian learning, we showcase the merits of robust Bayesian learning on several important wireless communication problems in terms of accuracy, calibration, and robustness to outliers and misspecification.

cs.LG↗