arXiv subjects

Germán Bassi

Publications and source records attributed to Germán Bassi.

11 recordsLinked to original sources

arXiv2022

Tighter expected generalization error bounds via Wasserstein distance

This work presents several expected generalization error bounds based on the Wasserstein distance. More specifically, it introduces full-dataset, single-letter, and random-subset bounds, and their analogues in the randomized subsample setting from Steinke and Zakynthinou [1]. Moreover, when the loss function is bounded and the geometry of the space is ignored by the choice of the metric in the Wasserstein distance, these bounds recover from below (and thus, are tighter than) current bounds based on the relative entropy. In particular, they generate new, non-vacuous bounds based on the relative entropy. Therefore, these results can be seen as a bridge between works that account for the geometry of the hypothesis space and those based on the relative entropy, which is agnostic to such geometry. Furthermore, it is shown how to produce various new bounds based on different information measures (e.g., the lautum information or several $f$-divergences) based on these bounds and how to derive similar bounds with respect to the backward channel using the presented proof techniques.

stat.ML↗

arXiv2021

Upper Bounds on the Generalization Error of Private Algorithms for Discrete Data

In this work, we study the generalization capability of algorithms from an information-theoretic perspective. It has been shown that the expected generalization error of an algorithm is bounded from above by a function of the relative entropy between the conditional probability distribution of the algorithm's output hypothesis, given the dataset with which it was trained, and its marginal probability distribution. We build upon this fact and introduce a mathematical formulation to obtain upper bounds on this relative entropy. Assuming that the data is discrete, we then develop a strategy using this formulation, based on the method of types and typicality, to find explicit upper bounds on the generalization error of stable algorithms, i.e., algorithms that produce similar output hypotheses given similar input datasets. In particular, we show the bounds obtained with this strategy for the case of $ε$-DP and $μ$-GDP algorithms.

cs.IT↗

arXiv2021

Testing for directed information graphs

In this paper, we study a hypothesis test to determine the underlying directed graph structure of nodes in a network, where the nodes represent random processes and the direction of the links indicate a causal relationship between said processes. Specifically, a k-th order Markov structure is considered for them, and the chosen metric to determine a connection between nodes is the directed information. The hypothesis test is based on the empirically calculated transition probabilities which are used to estimate the directed information. For a single edge, it is proven that the detection probability can be chosen arbitrarily close to one, while the false alarm probability remains negligible. When the test is performed on the whole graph, we derive bounds for the false alarm and detection probabilities, which show that the test is asymptotically optimal by properly setting the threshold test and using a large number of samples. Furthermore, we study how the convergence of the measures relies on the existence of links in the true graph.

cs.IT↗

arXiv2021

On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm

In this work, we unify several expected generalization error bounds based on random subsets using the framework developed by Hellström and Durisi [1]. First, we recover the bounds based on the individual sample mutual information from Bu et al. [2] and on a random subset of the dataset from Negrea et al. [3]. Then, we introduce their new, analogous bounds in the randomized subsample setting from Steinke and Zakynthinou [4], and we identify some limitations of the framework. Finally, we extend the bounds from Haghifam et al. [5] for Langevin dynamics to stochastic gradient Langevin dynamics and we refine them for loss functions with potentially large gradient norms.

cs.IT↗

arXiv2020

Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling

The estimation of mutual information (MI) or conditional mutual information (CMI) from a set of samples is a long-standing problem. A recent line of work in this area has leveraged the approximation power of artificial neural networks and has shown improvements over conventional methods. One important challenge in this new approach is the need to obtain, given the original dataset, a different set where the samples are distributed according to a specific product density function. This is particularly challenging when estimating CMI. In this paper, we introduce a new technique, based on k nearest neighbors (k-NN), to perform the resampling and derive high-confidence concentration bounds for the sample average. Then the technique is employed to train a neural network classifier and the CMI is estimated accordingly. We propose three estimators using this technique and prove their consistency, make a comparison between them and similar approaches in the literature, and experimentally show improvements in estimating the CMI in terms of accuracy and variance of the estimators.

cs.IT↗

arXiv2020

Causality Graph of Vehicular Traffic Flow

In an intelligent transportation system, the effects and relations of traffic flow at different points in a network are valuable features which can be exploited for control system design and traffic forecasting. In this paper, we define the notion of causality based on the directed information, a well-established data-driven measure, to represent the effective connectivity among nodes of a vehicular traffic network. This notion indicates whether the traffic flow at any given point affects another point's flow in the future and, more importantly, reveals the extent of this effect. In contrast with conventional methods to express connections in a network, it is not limited to linear models and normality conditions. In this work, directed information is used to determine the underlying graph structure of a network, denoted directed information graph, which expresses the causal relations among nodes in the network. We devise an algorithm to estimate the extent of the effects in each link and build the graph. The performance of the algorithm is then analyzed with synthetic data and real aggregated data of vehicular traffic.

eess.SY↗

arXiv2020

Conditional Mutual Information Neural Estimator

Several recent works in communication systems have proposed to leverage the power of neural networks in the design of encoders and decoders. In this approach, these blocks can be tailored to maximize the transmission rate based on aggregated samples from the channel. Motivated by the fact that, in many communication schemes, the achievable transmission rate is determined by a conditional mutual information term, this paper focuses on neural-based estimators for this information-theoretic quantity. Our results are based on variational bounds for the KL-divergence and, in contrast to some previous works, we provide a mathematically rigorous lower bound. However, additional challenges with respect to the unconditional mutual information emerge due to the presence of a conditional density function which we address here.

cs.IT↗

arXiv2019

Non-Asymptotic Behavior of the Maximum Likelihood Estimate of a Discrete Distribution

In this paper, we study the maximum likelihood estimate of the probability mass function (pmf) of $n$ independent and identically distributed (i.i.d.) random variables, in the non-asymptotic regime. We are interested in characterizing the Neyman--Pearson criterion, i.e., the log-likelihood ratio for testing a true hypothesis within a larger hypothesis. Wilks' theorem states that this ratio behaves like a $χ^2$ random variable in the asymptotic case; however, less is known about the precise behavior of the ratio when the number of samples is finite. In this work, we find an explicit bound for the difference between the cumulative distribution function (cdf) of the log-likelihood ratio and the cdf of a $χ^2$ random variable. Furthermore, we show that this difference vanishes with a rate of order $1/\sqrt{n}$ in accordance with Wilks' theorem.

math.ST↗

arXiv2018

Secret Key Generation over Noisy Channels with Correlated Sources

This paper investigates the problem of secret key generation over a wiretap channel when the terminals observe correlated sources. These sources are independent of the main channel and the users overhear them before the transmission takes place. A novel achievable scheme is proposed, and its optimality is shown under certain less-noisy conditions. This result improves upon the existing literature where the more stringent condition of degradedness is required. Furthermore, numerical evaluation of the proposed scheme and previously reported results for a binary model are presented; a comparison of the numerical bounds provides insights on the benefit of the novel scheme.

cs.IT↗

arXiv2018

The Wiretap Channel with Generalized Feedback: Secure Communication and Key Generation

It is a well-known fact that feedback does not increase the capacity of point-to-point memoryless channels, however, its effect in secure communications is not fully understood yet. In this work, an achievable scheme for the wiretap channel with generalized feedback is presented. This scheme, which uses the feedback signal to generate a shared secret key between the legitimate users, encrypts the message to be sent at the bit level. New capacity results for a class of channels are provided, as well as some new insights into the secret key agreement problem. Moreover, this scheme recovers previously reported rate regions from the literature, and thus it can be seen as a generalization that unifies several results in the field.

cs.IT↗

arXiv2015

Capacity Bounds for a Class of Interference Relay Channels

The capacity of a class of Interference Relay Channels (IRC) -the Injective Semideterministic IRC where the relay can only observe one of the sources- is investigated. We first derive a novel outer bound and two inner bounds which are based on a careful use of each of the available cooperative strategies together with the adequate interference decoding technique. The outer bound extends Telatar and Tse's work while the inner bounds contain several known results in the literature as special cases. Our main result is the characterization of the capacity region of the Gaussian class of IRCs studied within a fixed number of bits per dimension -constant gap. The proof relies on the use of the different cooperative strategies in specific SNR regimes due to the complexity of the schemes. As a matter of fact, this issue reveals the complex nature of the Gaussian IRC where the combination of a single coding scheme for the Gaussian relay and interference channel may not lead to a good coding scheme for this problem, even when the focus is only on capacity to within a constant gap over all possible fading statistics.

cs.IT↗