arXiv subjects

Samir M. Perlaza

Publications and source records attributed to Samir M. Perlaza.

At least 19 recordsLinked to original sources

arXiv2026

A Unified Framework for Training-Dataset Aggregation and Exact Machine Unlearning via Gibbs Probability Measures

In this paper, three operations on Gibbs probability measures are studied: renormalization, normalized log-linear combination, and nesting. The resulting measures are shown to be Gibbs probability measures with explicit expressions for their objective functions, reference measures, and regularization factors. These measures are further characterized as unique solutions to optimization problems involving linear combinations of the original objective functions, subject to either relative entropy regularization or relative entropy constraints. When the reference measures are probability measures, geometric interpretations of these three operations are established involving Pythagorean identities and exponential geodesics.A central result is the equivalence between nesting and normalized log-linear combinations of Gibbs probability measures. These results are then applied to supervised machine learning with Gibbs algorithms, which are represented by Gibbs probability measures. In this context, either through nesting or log-linear combinations, Gibbs algorithms trained on local datasets are used to form new Gibbs algorithms trained upon the aggregation of all local training datasets. Interestingly, the parameters of such operations, determine the weights of each training data point on the aggregated dataset. %Operations on local Gibbs probability measures are shown to control the contributions of training data. This observation is leveraged for achieving: (a) exact machine unlearning in centralized, decentralized, and federated learning systems; and (b) centralized-equivalent performance, in the sense of producing algorithms trained on the aggregation of all local datasets in decentralized and federated learning .

stat.ML↗

arXiv2026

Corrections to ``The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithms''

In this paper, a counterexample to Lemma 5 of the paper referenced in the title is provided. Consequently, the formulation of the method of data-driven gaps presented in the referenced paper relies on an additional assumption, namely, that the conditional distribution of the training dataset given the learned model has identical data-point-specific marginals. To overcome this limitation, a generalized formulation of the method of data-driven gaps that does not rely on Lemma 5 and does not require the training samples to be independent or identically distributed is presented. Through this formulation, the applicability of the method of data-driven gaps is extended to a broader class of data distributions and randomized learning algorithms.

cs.IT↗

arXiv2026

Performance Guarantees of SIC-Based Decoding for VLSF Codes in Hardcore-Regulated MAC

Spatial regulation suffices to guarantee reliable, low-latency communication using variable-length stop-feedback (VLSF) codes over multiple access channels. The delay performance of VLSF codes under successive interference cancellation is characterized for transmitters within a coverage region. For deployments modeled by hardcore point processes, the signal-to-interference-plus-noise ratio (SINR) admits a positive lower bound for all transmitters within the coverage region. Under a predefined total error probability constraint, these transmitters are decodable within an upper-bounded delay. For each transmitter, the decoding delay is well approximated by an inverse Gaussian distribution parameterized by the codebook size, target error probability, and the SINR lower bound. Numerical results confirm that hardcore regulation provides uniform performance guarantees across links.

cs.IT↗

arXiv2026

Machine Unlearning for Gibbs Supervised Learning Algorithms

In this paper, a method for achieving exact unlearning for Gibbs supervised learning algorithms is proposed using a variational formulation inspired by empirical risk minimization subject to relative entropy regularization (ERM-RER). Such a method consists of maximizing the expected empirical risk over the dataset to be unlearned subject to a regularization by relative entropy with respect to the original algorithm. The optimization variable is a probability measure on the models; and the solution is another Gibbs probability measure that represents a new Gibbs supervised learning algorithm. The method guarantees exact unlearning in the sense that the new Gibbs algorithm coincides in distribution with the algorithm that would have been obtained by retraining from scratch on the dataset to be retained. As a byproduct, a framework for reweighting data points in ERM-RER by strategically choosing both the reference measure and the regularization factor is obtained. In this framework, exact unlearning is the special case in which zero-weight is assigned to the contribution of the data points to be unlearned. More generally, depending on the choice of certain parameters, data points can be up-weighted or down-weighted in ERM-RER problems for particular purposes, e.g., controlling the generalization error of Gibbs algorithms. This paves the way for new constructive or adversarial views on classical reweighting data points in ERM-RER.

stat.ML↗

arXiv2026

Equivalence Between Nested Gibbs Measures and Log-Linear Combinations of Gibbs Measures

In this paper, three operations on Gibbs probability measures are studied. The first operation, often referred to as renormalization, takes one Gibbs probability measure and generates a new Gibbs measure by normalizing a power of its density. This normalization has a twofold effect: it changes the regularization factor and concentrates the support within a subset of the original support. Interestingly, these effects can be independently controlled by different parameters. The second operation consists of a normalized log-linear combination of the densities of Gibbs probability measures. The third operation takes two Gibbs probability measures and changes the reference measure of the latter with the former. Hence, the former is said to be "nested" within the latter, yielding a new Gibbs probability measure. The resulting measures from the second and third operations are also Gibbs probability measures and are shown, respectively, to solve optimization problems involving the expectations of linear combinations of the objective functions of the given measures, subject to a relative entropy regularization. These optimization problems differ exclusively in the coefficients of the linear combinations. This leads to the conclusion that there exists a set of parameters for which nesting one Gibbs probability measure into another has the same effect as log-linearly combining them. These operations have relevant applications in statistical learning. As an example, a one-shot federated learning system in which clients send their locally trained Gibbs algorithms to the server for log-linear combination is shown to achieve the same performance as a Gibbs algorithm trained upon the aggregation of all local training datasets.

cs.IT↗

arXiv2026

The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithms

In this paper, the method of gaps, a technique for deriving closed-form expressions in terms of information measures for the generalization error of supervised learning algorithms, is introduced. This method relies on the notion of gaps, which characterize the variation of the expected empirical risk (when either the model or dataset is kept fixed) with respect to changes in the probability measure on the varying parameter. This distinction results in two classes of gaps: algorithm-driven gaps (fixed dataset) and data-driven gaps (fixed model). The method relies on two central observations: (i) the generalization error is the expectation of an algorithm-driven gap or a data-driven gap. In the first case, the expectation is with respect to a measure on the datasets; in the second case, it is with respect to a measure on the models. (ii) Both algorithm-driven gaps and data-driven gaps exhibit closed-form expressions in terms of relative entropies. In particular, algorithm-driven gaps involve a Gibbs probability measure on the set of models, which represents a supervised Gibbs algorithm. Alternatively, data-driven gaps involve a worst-case data-generating (WCDG) probability measure on the set of data points, which is also a Gibbs probability measure. Interestingly, such Gibbs measures, which are exogenous to the analysis of generalization, place the supervised Gibbs algorithm and the WCDG probability measure as natural references for the analysis of supervised learning algorithms. New exact expressions and all existing exact expressions for the generalization error of supervised learning algorithms can be obtained with the proposed method. Such new expressions are intended as structural and conceptual characterizations, not computational shortcuts. Finally, these expressions unveil strong connections among generalization, hypothesis testing, information measures, and Pythagorean identities.

cs.LG↗

arXiv2026

Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms

In this paper, it is shown, for the first time, that centralized performance is achievable in decentralized learning without sharing the local datasets. Specifically, when clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) learning framework and a forward-backward communication between clients is established, it suffices to share the locally obtained Gibbs measures to achieve the same performance as that of a centralized ERM-RER with access to all the datasets. The core idea is that the Gibbs measure produced by client~$k$ is used, as reference measure, by client~$k+1$. This effectively establishes a principled way to encode prior information through a reference measure. In particular, achieving centralized performance in the decentralized setting requires a specific scaling of the regularization factors with the local sample sizes. Overall, this result opens the door to novel decentralized learning paradigms that shift the collaboration strategy from sharing data to sharing the local inductive bias via the reference measures over the set of models.

stat.ML↗

arXiv2026

Optimization of Sparse VLSF Codes for Short-Packet Transmission via Saddlepoint Methods

In this work, we present an optimization framework for sparse variable-length stop-feedback (VLSF) codes based on a saddlepoint approximation, which jointly optimizes the decoding configuration parameters. Thanks to the analytical tractability of the saddlepoint approximation, the framework enables efficient gradient-based optimization of such parameters for common memoryless channels, including the additive white Gaussian noise, binary symmetric, and binary erasure channels. We further propose a refined decoding rule that extends the conventional fixed-threshold rule and leads to a tighter achievability bound. Numerical results demonstrate that our framework provides near-optimal decoding configurations at low computational cost. Moreover, the results from our refined rule demonstrate that the fixed-threshold decoding rule is restrictive and that achievability bounds can be further tightened.

cs.IT↗

arXiv2026

VLSF Decoding with Reliability Guarantees over Correlated Noncoherent Fading Channels

This paper studies reliability-guaranteed decoding for variable-length stop-feedback (VLSF) codes over correlated noncoherent fading channels. The decoding rule is based on the evolution of the information density associated with a given channel input-output realization. Due to channel memory, exact evaluation of this information density is intractable. To enable constructive decoding, computable finite-blocklength lower and upper bounds on the information density that hold uniformly over time along each input-output sequence are derived. The lower bound enables a stopping-time analysis for VLSF decoding and has an operational meaning, while the upper bound provides a reference for the relaxation gap, which is explicitly characterized. As a concrete application, the Gauss-Markov fading channel with Gaussian signaling is considered to numerically investigate the stopping-time distribution and the impact of fading correlation on decoding performance.

cs.IT↗

arXiv2026

Equivalence of optimal transport problems to regularization on the family of f-divergences

This work establishes that an optimal transport~(OT) problem regularized by a given $f$-divergence admits the same solution as another OT problem regularized by a different $g$-divergence, under an appropriate transformation of the cost function. This structural equivalence between OT problems regularized by distinct divergences, in the sense of sharing the same unique minimizer, is demonstrated within the framework of Polish spaces with bounded cost functions.

math.ST↗

arXiv2026

Empirical Risk Minimization with $f$-Divergence Regularization

In this paper, the solution to the empirical risk minimization problem with $f$-divergence regularization (ERM-$f$DR) is presented and conditions under which the solution also serves as the solution to the minimization of the expected empirical risk subject to an $f$-divergence constraint are established. The proposed approach extends applicability to a broader class of $f$-divergences than previously reported and yields theoretical results that recover previously known results. Additionally, the difference between the expected empirical risk of the ERM-$f$DR solution and that of its reference measure is characterized, providing insights into previously studied cases of $f$-divergences. A central contribution is the introduction of the normalization function, a mathematical object that is critical in both the dual formulation and practical computation of the ERM-$f$DR solution. This work presents an implicit characterization of the normalization function as a nonlinear ordinary differential equation (ODE), establishes its key properties, and subsequently leverages them to construct a numerical algorithm for approximating the normalization factor under mild assumptions. Further analysis demonstrates structural equivalences between ERM-$f$DR problems with different $f$-divergences via transformations of the empirical risk. Finally, the proposed algorithm is used to compute the training and test risks of ERM-$f$DR solutions under different $f$-divergence regularizers. This numerical example highlights the practical implications of choosing different functions $f$ in ERM-$f$DR problems.

stat.ML↗

arXiv2025

A Dual Optimization View to Empirical Risk Minimization with f-Divergence Regularization

The dual formulation of empirical risk minimization with f-divergence regularization (ERM-fDR) is introduced. The solution of the dual optimization problem to the ERM-fDR is connected to the notion of normalization function introduced as an implicit function. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem to provide a nonlinear ODE expression to the normalization function. Furthermore, the nonlinear ODE expression and its properties provide a computationally efficient method to calculate the normalization function of the ERM-fDR solution under a mild condition.

stat.ML↗

arXiv2025

Proofs for Folklore Theorems on the Radon-Nikodym Derivative

In this technical report, rigorous statements and formal proofs are presented for both foundational and advanced folklore theorems on the Radon-Nikodym derivative. The cases of conditional and marginal probability measures are carefully considered, which leads to an identity involving the sum of mutual and lautum information suggesting a new interpretation for such a sum.

cs.IT↗

arXiv2025

Decentralized Stealth Attacks on Cyber-Physical Systems

Decentralized stealth attack constructions that minimize the mutual information between the state variables and the measurements are proposed. The attack constructions are formulated as random Gaussian attacks targeting Cyber-physical systems that aims at minimizing the mutual information between the state variables and measurements while constraining the Kullback-Leibler divergence between the distribution of the measurements under attacks and the distribution of the measurements without attacks. The proposed information metrics adopted measure the disruption and attack detection both globally and locally. The decentralized attack constructions are formulated in a framework of normal games. The global and local information metrics yield games with global and local objectives in disruption and attack detection. We have proven the games are potential games and the convexity of the potential functions followed by the uniqueness and the achievability of the Nash Equilibrium, accordingly. We proposed a best response dynamics to achieve the Nash Equilibrium of the games. We numerically evaluate the performance of the proposed decentralized stealth random attacks on IEEE test systems and show it is feasible to exploit game theoretic techniques in decentralized attack constructions.

eess.SY↗

arXiv2025

Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimization

The effect of relative entropy asymmetry is analyzed in the context of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER). Two regularizations are considered: $(a)$ the relative entropy of the measure to be optimized with respect to a reference measure (Type-I ERM-RER); and $(b)$ the relative entropy of the reference measure with respect to the measure to be optimized (Type-II ERM-RER). The main result is the characterization of the solution to the Type-II ERM-RER problem and its key properties. By comparing the well-understood Type-I ERM-RER with Type-II ERM-RER, the effects of entropy asymmetry are highlighted. The analysis shows that in both cases, regularization by relative entropy forces the solution's support to collapse into the support of the reference measure, introducing a strong inductive bias that negates the evidence provided by the training data. Finally, it is shown that Type-II regularization is equivalent to Type-I regularization with an appropriate transformation of the empirical risk function.

stat.ML↗

arXiv2025

Variations on the Expectation due to Changes in the Probability Measure

In this paper, closed-form expressions are presented for the variation of the expectation of a given function due to changes in the probability measure used for the expectation. They unveil interesting connections with Gibbs probability measures, mutual information, and lautum information.

cs.IT↗

arXiv2025

Generalization Error of $f$-Divergence Stabilized Algorithms via Duality

The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is extended to constrained optimization problems, establishing conditions for equivalence between the solution and constraints. A dual formulation of ERM-$f$DR is introduced, providing a computationally efficient method to derive the normalization function of the ERM-$f$DR solution. This dual approach leverages the Legendre-Fenchel transform and the implicit function theorem, enabling explicit characterizations of the generalization error for general algorithms under mild conditions, and another for ERM-$f$DR solutions.

stat.ML↗

arXiv2024

Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

The solution to empirical risk minimization with $f$-divergence regularization (ERM-$f$DR) is presented under mild conditions on $f$. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function $f$ are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of $f$-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of $f$-divergences when used in the ERM-$f$DR problem: $i\bigl)$ $f$-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and $ii\bigl)$ any $f$-divergence regularization is equivalent to a different $f$-divergence regularization with an appropriate transformation of the empirical risk function.

stat.ML↗