arXiv subjects

Thuan Nguyen

Publications and source records attributed to Thuan Nguyen.

At least 19 recordsLinked to original sources

arXiv2026

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.

cs.LG↗

arXiv2025

Universal Rate-Distortion-Classification Representations for Lossy Compression

In lossy compression, Wang et al. [1] recently introduced the rate-distortion-perception-classification function, which supports multi-task learning by jointly optimizing perceptual quality, classification accuracy, and reconstruction fidelity. Building on the concept of a universal encoder introduced in [2], we investigate the universal representations that enable a broad range of distortion-classification tradeoffs through a single shared encoder coupled with multiple task-specific decoders. We establish, through both theoretical analysis and numerical experiments, that for Gaussian source under mean squared error (MSE) distortion, the entire distortion-classification tradeoff region can be achieved using a single universal encoder. For general sources, we characterize the achievable region and identify conditions under which encoder reuse results in negligible distortion penalty. The experimental result on the MNIST dataset further supports our theoretical findings. We show that universal encoders can obtain distortion performance comparable to task-specific encoders. These results demonstrate the practicality and effectiveness of the proposed universal framework in multi-task compression scenarios.

cs.IT↗

arXiv2025

On Symbol Error Probability-based Beamforming in MIMO Gaussian Wiretap Channels

This paper investigates beamforming schemes designed to minimize the symbol error probability (SEP) for an authorized user while guaranteeing that the likelihood of an eavesdropper correctly recovering symbols remains below a predefined threshold. Unlike previous works that focus on maximizing secrecy capacity, our work is centered around finding an optimal beamforming vector for binary antipodal signal detection in multiple-input multiple-output (MIMO) Gaussian wiretap channels. Finding the optimal beamforming vector in this setting is challenging. Computationally efficient algorithms such as convex techniques cannot be applied to find the optimal solution. To that end, our proposed algorithm relies on Karush-Kuhn-Tucker (KKT) conditions and a generalized eigen-decomposition method to find the exact solution. In addition, we also develop an approximate, practical algorithm to find a good beamforming matrix when using M-ary detection schemes. Numerical results are presented to assess the performance of the proposed methods across various scenarios.

eess.SP↗

arXiv2024

A Random-Effects Approach to Generalized Linear Mixed Model Analysis of Incomplete Longitudinal Data

We propose a random-effects approach to missing values for generalized linear mixed model (GLMM) analysis. The method converts a GLMM with missing covariates to another GLMM without missing covariates. The standard GLMM analysis tools for longitudinal data then apply. The method applies, in particular, to the cases of linear mixed models and logistic regression. Performance of the method is evaluated empirically, and compared with alternative approaches, including the popular MICE procedure of multiple imputation. Theoretical justification of the method is given, and explained, for the patterns observed in the simulation studies. Two real-data examples from healthcare studies are discussed.

stat.ME↗

arXiv2023

Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse

For a widely-studied data model and general loss and sample-hardening functions we prove that the losses of Supervised Contrastive Learning (SCL), Hard-SCL (HSCL), and Unsupervised Contrastive Learning (UCL) are minimized by representations that exhibit Neural-Collapse (NC), i.e., the class means form an Equiangular Tight Frame (ETF) and data from the same class are mapped to the same representation. We also prove that for any representation mapping, the HSCL and Hard-UCL (HUCL) losses are lower bounded by the corresponding SCL and UCL losses. In contrast to existing literature, our theoretical results for SCL do not require class-conditional independence of augmented views and work for a general loss function class that includes the widely used InfoNCE loss function. Moreover, our proofs are simpler, compact, and transparent. Similar to existing literature, our theoretical claims also hold for the practical scenario where batching is used for optimization. We empirically demonstrate, for the first time, that Adam optimization (with batching) of HSCL and HUCL losses with random initialization and suitable hardness levels can indeed converge to the NC-geometry if we incorporate unit-ball or unit-sphere feature normalization. Without incorporating hard-negatives or feature normalization, however, the representations learned via Adam suffer from Dimensional-Collapse (DC) and fail to attain the NC-geometry. These results exemplify the role of hard-negative sampling in contrastive representation learning and we conclude with several open theoretical problems for future work. The code can be found at https://github.com/rjiang03/HCL/tree/main

cs.LG↗

arXiv2023

Barycentric-alignment and reconstruction loss minimization for domain generalization

This paper advances the theory and practice of Domain Generalization (DG) in machine learning. We consider the typical DG setting where the hypothesis is composed of a representation mapping followed by a labeling function. Within this setting, the majority of popular DG methods aim to jointly learn the representation and the labeling functions by minimizing a well-known upper bound for the classification risk in the unseen domain. In practice, however, methods based on this theoretical upper bound ignore a term that cannot be directly optimized due to its dual dependence on both the representation mapping and the unknown optimal labeling function in the unseen domain. To bridge this gap between theory and practice, we introduce a new upper bound that is free of terms having such dual dependence, resulting in a fully optimizable risk upper bound for the unseen domain. Our derivation leverages classical and recent transport inequalities that link optimal transport metrics with information-theoretic measures. Compared to previous bounds, our bound introduces two new terms: (i) the Wasserstein-2 barycenter term that aligns distributions between domains, and (ii) the reconstruction loss term that assesses the quality of representation in reconstructing the original data. Based on this new upper bound, we propose a novel DG algorithm named Wasserstein Barycenter Auto-Encoder (WBAE) that simultaneously minimizes the classification loss, the barycenter loss, and the reconstruction loss. Numerical results demonstrate that the proposed method outperforms current state-of-the-art DG algorithms on several datasets.

cs.LG↗

arXiv2023

A principled approach to model validation in domain generalization

Domain generalization aims to learn a model with good generalization ability, that is, the learned model should not only perform well on several seen domains but also on unseen domains with different data distributions. State-of-the-art domain generalization methods typically train a representation function followed by a classifier jointly to minimize both the classification risk and the domain discrepancy. However, when it comes to model selection, most of these methods rely on traditional validation routines that select models solely based on the lowest classification risk on the validation set. In this paper, we theoretically demonstrate a trade-off between minimizing classification risk and mitigating domain discrepancy, i.e., it is impossible to achieve the minimum of these two objectives simultaneously. Motivated by this theoretical result, we propose a novel model selection method suggesting that the validation process should account for both the classification risk and the domain discrepancy. We validate the effectiveness of the proposed method by numerical results on several domain generalization datasets.

cs.LG↗

arXiv2022

Bounded Guaranteed Algorithms for Concave Impurity Minimization Via Maximum Likelihood

Partitioning algorithms play a key role in many scientific and engineering disciplines. A partitioning algorithm divides a set into a number of disjoint subsets or partitions. Often, the quality of the resulted partitions is measured by the amount of impurity in each partition, the smaller impurity the higher quality of the partitions. In general, for a given impurity measure specified by a function of the partitions, finding the minimum impurity partitions is an NP-hard problem. Let $M$ be the number of $N$-dimensional elements in a set and $K$ be the number of desired partitions, then an exhaustive search over all the possible partitions to find a minimum partition has the complexity of $O(K^M)$ which quickly becomes impractical for many applications with modest values of $K$ and $M$. Thus, many approximate algorithms with polynomial time complexity have been proposed, but few provide bounded guarantee. In this paper, an upper bound and a lower bound for a class of impurity functions are constructed. Based on these bounds, we propose a low-complexity partitioning algorithm with bounded guarantee based on the maximum likelihood principle. The theoretical analyses on the bounded guarantee of the algorithms are given for two well-known impurity functions Gini index and entropy. When $K \geq N$, the proposed algorithm achieves state-of-the-art results in terms of lowest approximations and polynomial time complexity $O(NM)$. In addition, a heuristic greedy-merge algorithm having the time complexity of $O((N-K)N^2+NM)$ is proposed for $K<N$. Although the greedy-merge algorithm does not provide a bounded guarantee, its performance is comparable to that of the state-of-the-art methods. Our results also generalize some well-known information-theoretic bounds such as Fano's inequality and Boyd-Chiang's bound.

cs.IT↗

arXiv2022

Trade-off between reconstruction loss and feature alignment for domain generalization

Domain generalization (DG) is a branch of transfer learning that aims to train the learning models on several seen domains and subsequently apply these pre-trained models to other unseen (unknown but related) domains. To deal with challenging settings in DG where both data and label of the unseen domain are not available at training time, the most common approach is to design the classifiers based on the domain-invariant representation features, i.e., the latent representations that are unchanged and transferable between domains. Contrary to popular belief, we show that designing classifiers based on invariant representation features alone is necessary but insufficient in DG. Our analysis indicates the necessity of imposing a constraint on the reconstruction loss induced by representation functions to preserve most of the relevant information about the label in the latent space. More importantly, we point out the trade-off between minimizing the reconstruction loss and achieving domain alignment in DG. Our theoretical results motivate a new DG framework that jointly optimizes the reconstruction loss and the domain discrepancy. Both theoretical and numerical results are provided to justify our approach.

cs.LG↗

arXiv2022

Supervised Contrastive Learning with Hard Negative Samples

Through minimization of an appropriate loss function such as the InfoNCE loss, contrastive learning (CL) learns a useful representation function by pulling positive samples close to each other while pushing negative samples far apart in the embedding space. The positive samples are typically created using "label-preserving" augmentations, i.e., domain-specific transformations of a given datum or anchor. In absence of class information, in unsupervised CL (UCL), the negative samples are typically chosen randomly and independently of the anchor from a preset negative sampling distribution over the entire dataset. This leads to class-collisions in UCL. Supervised CL (SCL), avoids this class collision by conditioning the negative sampling distribution to samples having labels different from that of the anchor. In hard-UCL (H-UCL), which has been shown to be an effective method to further enhance UCL, the negative sampling distribution is conditionally tilted, by means of a hardening function, towards samples that are closer to the anchor. Motivated by this, in this paper we propose hard-SCL (H-SCL) {wherein} the class conditional negative sampling distribution {is tilted} via a hardening function. Our simulation results confirm the utility of H-SCL over SCL with significant performance gains {in downstream classification tasks.} Analytically, we show that {in the} limit of infinite negative samples per anchor and a suitable assumption, the {H-SCL loss} is upper bounded by the {H-UCL loss}, thereby justifying the utility of H-UCL {for controlling} the H-SCL loss in the absence of label information. Through experiments on several datasets, we verify the assumption as well as the claimed inequality between H-UCL and H-SCL losses. We also provide a plausible scenario where H-SCL loss is lower bounded by UCL loss, indicating the limited utility of UCL in controlling the H-SCL loss.

cs.LG↗

arXiv2022

Joint covariate-alignment and concept-alignment: a framework for domain generalization

In this paper, we propose a novel domain generalization (DG) framework based on a new upper bound to the risk on the unseen domain. Particularly, our framework proposes to jointly minimize both the covariate-shift as well as the concept-shift between the seen domains for a better performance on the unseen domain. While the proposed approach can be implemented via an arbitrary combination of covariate-alignment and concept-alignment modules, in this work we use well-established approaches for distributional alignment namely, Maximum Mean Discrepancy (MMD) and covariance Alignment (CORAL), and use an Invariant Risk Minimization (IRM)-based approach for concept alignment. Our numerical results show that the proposed methods perform as well as or better than the state-of-the-art for domain generalization on several data sets.

cs.LG↗

arXiv2022

Conditional entropy minimization principle for learning domain invariant representation features

Invariance-principle-based methods such as Invariant Risk Minimization (IRM), have recently emerged as promising approaches for Domain Generalization (DG). Despite promising theory, such approaches fail in common classification tasks due to the mixing of true invariant features and spurious invariant features. To address this, we propose a framework based on the conditional entropy minimization (CEM) principle to filter-out the spurious invariant features leading to a new algorithm with a better generalization capability. We show that our proposed approach is closely related to the well-known Information Bottleneck (IB) framework and prove that under certain assumptions, entropy minimization can exactly recover the true invariant features. Our approach provides competitive classification accuracy compared to recent theoretically-principled state-of-the-art alternatives across several DG datasets.

cs.LG↗

arXiv2020

Optimal quantizer structure for binary discrete input continuous output channels under an arbitrary quantized-output constraint

Given a channel having binary input X = (x_1, x_2) having the probability distribution p_X = (p_{x_1}, p_{x_2}) that is corrupted by a continuous noise to produce a continuous output y \in Y = R. For a given conditional distribution p(y|x_1) = ϕ_1(y) and p(y|x_2) = ϕ_2(y), one wants to quantize the continuous output y back to the final discrete output Z = (z_1, z_2, ..., z_N) with N \leq 2 such that the mutual information between input and quantized-output I(X; Z) is maximized while the probability of the quantized-output p_Z = (p_{z_1}, p_{z_2}, ..., p_{z_N}) has to satisfy a certain constraint. Consider a new variable r_y=p_{x_1}ϕ_1(y)/ (p_{x_1}ϕ_1(y)+p_{x_2}ϕ_2(y)), we show that the optimal quantizer has a structure of convex cells in the new variable r_y. Based on the convex cells property of the optimal quantizers, a fast algorithm is proposed to find the global optimal quantizer in a polynomial time complexity.

eess.SP↗

arXiv2020

Entropy-Constrained Maximizing Mutual Information Quantization

In this paper, we investigate the quantization of the output of a binary input discrete memoryless channel that maximizing the mutual information between the input and the quantized output under an entropy-constrained of the quantized output. A polynomial time algorithm is introduced that can find the truly global optimal quantizer. These results hold for binary input channels with an arbitrary number of quantized output. Finally, we extend these results to binary input continuous output channels and show a sufficient condition such that a single threshold quantizer is an optimal quantizer. Both theoretical results and numerical results are provided to justify our techniques.

cs.IT↗

arXiv2020

On the Uniqueness of Binary Quantizers for Maximizing Mutual Information

We consider a channel with a binary input X being corrupted by a continuous-valued noise that results in a continuous-valued output Y. An optimal binary quantizer is used to quantize the continuous-valued output Y to the final binary output Z to maximize the mutual information I(X; Z). We show that when the ratio of the channel conditional density r(y) = P(Y=y|X=0)/ P(Y =y|X=1) is a strictly increasing/decreasing function of y, then a quantizer having a single threshold can maximize mutual information. Furthermore, we show that an optimal quantizer (possibly with multiple thresholds) is the one with the thresholding vector whose elements are all the solutions of r(y) = r* for some constant r* > 0. Interestingly, the optimal constant r* is unique. This uniqueness property allows for fast algorithmic implementation such as a bisection algorithm to find the optimal quantizer. Our results also confirm some previous results using alternative elementary proofs. We show some numerical examples of applying our results to channels with additive Gaussian noises.

eess.SP↗

arXiv2020

Single-bit Quantization Capacity of Binary-input Continuous-output Channels

We consider a channel with discrete binary input X that is corrupted by a given continuous noise to produce a continuous-valued output Y. A quantizer is then used to quantize the continuous-valued output Y to the final binary output Z. The goal is to design an optimal quantizer Q* and also find the optimal input distribution p*(X) that maximizes the mutual information I(X; Z) between the binary input and the binary quantized output. A linear time complexity searching procedure is proposed. Based on the properties of the optimal quantizer and the optimal input distribution, we reduced the searching range that results in a faster implementation algorithm. Both theoretical and numerical results are provided to illustrate our method.

eess.SP↗

arXiv2020

On Bounds and Closed Form Expressions for Capacities of Discrete Memoryless Channels with Invertible Positive Matrices

While capacities of discrete memoryless channels are well studied, it is still not possible to obtain a closed-form expression for the capacity of an arbitrary discrete memoryless channel. This paper describes an elementary technique based on Karush Kuhn Tucker (KKT) conditions to obtain (1) a good upper bound of a discrete memoryless channel having an invertible positive channel matrix and (2) a closed-form expression for the capacity if the channel matrix satisfies certain conditions related to its singular value and its Gershgorin disk.

cs.IT↗

arXiv2020

Communication-Channel Optimized Partition

Given an original discrete source X with the distribution p_X that is corrupted by noise to produce the noisy data Y with the given joint distribution p(X, Y). A quantizer/classifier Q : Y -> Z is then used to classify/quantize the data Y to the discrete partitioned output Z with probability distribution p_Z. Next, Z is transmitted over a deterministic channel with a given channel matrix A that produces the final discrete output T. One wants to design the optimal quantizer/classifier Q^* such that the cost function F(X; T) between the input X and the final output T is minimized while the probability of the partitioned output Z satisfies a concave constraint G(p_Z) < C. Our results generalized some famous previous results. First, an iteration linear time complexity algorithm is proposed to find the local optimal quantizer. Second, we show that the optimal partition should produce a hard partition that is equivalent to the cuts by hyper-planes in the probability space of the posterior probability p(X|Y). This result finally provides a polynomial-time algorithm to find the globally optimal quantizer.

cs.IT↗