Searcharxiv⌕ Search

arXiv subjects

Dung Le

Publications and source records attributed to Dung Le.

At least 19 recordsLinked to original sources
arXiv2026

When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts

Sparse mixtures of experts combine multiple prediction models while evaluating only a small subset for each input. Although this architecture reduces computation, how expert separation and routing sparsity jointly govern expert recovery remains an open problem. In this paper, we address this question for location Gaussian mixtures of experts with a fixed number of experts by establishing how expert separation and spatial arrangement determine recovery rates. For dense models, our results reveal that when experts form well-separated clusters, the separation geometry is governed by the size of the largest cluster; without a prescribed cluster structure, this geometry is instead driven by the total number of experts. For sparse models, the separation geometry on retained routing regions is determined by the number of active experts, while the contribution of omitted regions is controlled by their total probability. Consequently, activating fewer experts reduces the separation complexity in these bounds. Furthermore, Top-1 routing yields a particularly striking result: experts can be recovered at a rate independent of the separation quantity. Our technical approach combines polynomial interpolation and Gaussian reproducing-kernel Hilbert space (RKHS) test functions with quantitative restriction inequalities that extend pointwise comparisons to expert functions.

stat.ML↗
arXiv2026

The Statistical Geometry of Latent Recovery in Hidden Markov Models

Hidden Markov models (HMMs) offer an important capability to interpret fitted latent states and transitions. When emission distributions nearly coincide, a small observation-density error does not ensure accurate estimation of the latent quantities needed for interpretation. This motivates us to characterize convergence rates for latent recovery that account for emission separation. Our main contribution lies in deriving deterministic inverse inequalities that convert Hellinger error between fitted and true densities into bounds on latent recovery errors. We develop these bounds for multivariate Gaussian HMMs with a common, known emission covariance matrix and combine them with a Hellinger estimation guarantee for the global maximum likelihood estimator to obtain latent recovery rates. Although the joint observation distribution is a Gaussian mixture with exponentially many path components, they share the same emission centers across time. Leveraging this shared structure, we show that the separation exponents are governed by the maximum number of simultaneously approaching centers rather than the number of path components. Under exact specification, the global bounds on state-sequence probability discrepancies and the 1-Wasserstein loss for latent location distributions exhibit sharp separation exponents that do not increase with the number of selected times. Moreover, sufficiently small observation-density error guarantees unique matching between fitted and true emission centers, which yields error bounds for individual emission centers and transition rows. Under over-specification, individual split-state transition rows are not necessarily identifiable, so we quantify aggregate latent recovery errors and show that state-sequence probability bounds incur larger separation exponents compared to exact specification. We conduct synthetic experiments to validate our theoretical results.

math.ST↗
arXiv2026

Weak Limits of Parameter Estimation in Strongly Identifiable Mixture Models

We study the asymptotic behavior of global maximum likelihood estimation in finite mixtures with an unknown number of components bounded by \(k\). Although convergence rates are well understood under strong identifiability, they do not determine the limiting distribution of mixing-measure loss. Under second-order strong identifiability, we establish two geometric results that connect density asymptotics to uncertainty about the mixing measure. First, we characterize the exact attainable density tangent set, accounting for the component budget shared by nonlocal vanishing components and local splitting directions. Second, we derive a sequential first-order expansion of squared Wasserstein loss, showing that its leading transportation cost is uniquely determined by the relative-density direction. Together with a Gaussian likelihood argmax theorem, these results yield joint weak convergence of the relative-density maximum likelihood estimator and \(\sqrt n W_2^2(\widehat G_n,G_*)\). The density limit maximizes a Gaussian likelihood process over the attainable tangent set, and the induced limit of \(n^{1/4}W_2(\widehat G_n,G_*)\) is generally non-Gaussian. We establish consistency of an \(m\)-out-of-\(n\) parametric bootstrap when \(m\to\infty\) and \(m/n\to0\). Numerical studies assess the approximation and distinguish bootstrap and optimization errors; a flow-cytometry application illustrates uncertainty summaries invariant to component labeling under a working mixture model.

math.ST↗
arXiv2026

Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transport

Parameter estimation in finite mixture models can exhibit highly heterogeneous convergence behavior: locally isolated components may be estimated substantially faster than groups of competing components. Existing analyses based on Wasserstein distances typically characterize only the worst-case rate and therefore do not fully capture this local heterogeneity. In this paper, we introduce a Voronoi-based partial optimal transport (VPOT) framework for obtaining refined local and global convergence guarantees for the maximum likelihood estimator of the mixing measure. The key geometric idea is to localize the comparison of two mixing measures to extended Voronoi neighborhoods and use partial optimal transport to accommodate the unequal masses of their local restrictions. Within each neighborhood, the first-order POT discrepancy is raised to a power determined by the number of locally competing atoms, allowing the resulting loss to adapt to the local degree of singularity. Under suitable regularity and strong identifiability conditions, we establish uniform local and global upper bounds for a maximum likelihood estimator under the VPOT loss. These bounds reveal a configuration-dependent form of parameter estimation: less singular local configurations admit faster convergence, whereas the most singular configuration recovers the classical worst-case behavior characterized by Wasserstein-based analyses. We further establish a minimax lower bound showing that the convergence rate for estimating the mixing measure under the VPOT loss is optimal. Our results hold in arbitrary fixed dimension without requiring mixing proportions to be uniformly bounded away from zero or prior knowledge of the true number of mixture components. Overall, VPOT provides a configuration-adaptive framework for capturing heterogeneous parameter-estimation behavior in finite mixture models.

math.ST↗
arXiv2026

Berry-Esseen bounds for multivariate martingale difference sequences in the Kolmogorov distance

We derive new Gaussian approximations for finite martingale difference sequences in $\mathbb{R}^d$ with respect to the Kolmogorov distance. Under appropriate conditions, our bounds exhibit a dependence of order $n^{-1/4}$ on the length of the sequence and of order $\mathrm{polylog}(d)$ on the dimension. As an application, we derive a high-dimensional Berry-Esseen bound over hyper-rectangles for martingale sequences generated from Markov chains.

math.PR↗
arXiv2026

Partial Differential Equation Barriers to Identifiability in Infinite Mixture Models

We study identifiability of mixing measures in infinite mixture models. We show that, in many common cases, lack of identifiability can be characterized in terms of certain differential structures of the kernel family with respect to its parameters. In our main results, we prove that when the kernel is annihilated by a non-trivial differential or difference-differential operator over the parameter space, there exist infinitely many distinct mixing measures yielding the same mixture density. We give verifiable conditions for such operators to exist, covering many common cases, including the location-scale Gaussian, location-scale Student-t, Gamma, Beta, Dirichlet, negative binomial and non-central Chi-squared families. Furthermore, our conditions apply to any exponential family whose parameter dimension exceeds the dimension of its sufficient statistic and, more generally, to kernels with polynomially-growing score functions. We complement our results with a minimax lower bound on the estimation error for the mixing measure in the Wasserstein distance under non-identifiability. On the flips side, we describe three classes of kernels for which identifiability is preserved and nonparametric statistical inference remains possible.

math.ST↗
arXiv2026

Convergence Rates for Latent Mixing Measures in Infinite Homoscedastic Location-Scale Mixture Models

We study posterior contraction rates for mixing measures in homoscedastic location-scale mixture models with infinitely many components. While posterior convergence at the level of densities is well understood, ensuring convergence of the latent mixing measure is more challenging and has remained an open problem in settings where both location and scale parameters are unknown. We address this by deriving novel lower-bounds that connect the $L^1$ distance between mixture densities to discrepancies, based on the Wasserstein distances and the operator norm, between the underlying mixing measures and scale matrices. Our approach combines the dual formulation of the $W_1$ distance with functional-analytic approximation techniques. This leads to general inequalities, whose strength is determined (i) by the smoothness of the mixture kernel via the rate of decay of its characteristic function, and (ii) by a key lower-bound on the $L^1$ metric involving the operator norm discrepancy between scale parameters. Moreover, a novel PDE inversion condition yields a sharper inequality for important ordinary-smooth cases. We specialize these bounds to popular mixtures based on multivariate Gaussian, Cauchy, and Laplace kernels. As a consequence, we obtain first-of-their-kind contraction rates in the context of Dirichlet process mixtures with an unknown scale parameter shared across components. As a byproduct of our inequalities, we can distinguish the convergence behavior of the location mixing measure from that of the scale parameter across a range of kernel choices, leading to nuanced insights into their respective rates.

math.ST↗
arXiv2026

On the Geometry of Separation in Finite Gaussian Mixtures

We study an open problem of understanding the effects of the minimum component separation on the convergence rates of parameter estimation in finite Gaussian mixtures. We address this by developing a unified geometric framework based on novel Hellinger lower bounds that directly relate discrepancies between mixture densities directly to Wasserstein distances between their underlying mixing measures, with explicit dependence on both the minimum separation and the minimum weight. Our approach combines carefully designed interpolation polynomials with confluent divided difference techniques to construct specialized moment-extraction test functions. When the number of components is known, these bounds uncover a localization phenomenon: the separation complexity is driven strictly by the spatial configuration of mixture components, namely, whether they are concentrated in a single cluster, partitioned into multiple clusters separated by a macroscopic gap, or arranged without any structural constraints. On the other hand, when the number of components becomes unknown and is over-specified, the separation complexity is slightly reduced, while the minimum mixture weight disappears entirely from the convergence rates due to a transition from first-order to second-order Wasserstein geometry. As a consequence, we obtain separation-dependent convergence rates that continuously interpolate between point-wise and uniform estimation regimes, thereby settling the fundamental limits of parameter recovery in finite Gaussian mixtures.

math.ST↗
arXiv2026

The Confidence Trap: Calibration Attacks for Graph Neural Networks

While confidence calibration is essential for trustworthy decision-making in safety-critical applications, the robustness of calibrated GNNs to adversarial structural perturbations remains largely unexplored. However, studying calibration attacks on graphs presents unique technical challenges: (1) the discrete nature of graph structures complicates gradient-based optimization, (2) existing underconfidence objectives fail to drive predictions toward uniform distributions, and (3) GNNs are highly sensitive to edge perturbations, often causing unintended label changes that violate attack constraints. To address these challenges, we propose a \textbf{Unified Graph Calibration Attack (UGCA)} framework designed for \textbf{worst-case (white-box) analysis} of GNN calibration robustness. UGCA introduces a KL-divergence loss to encourage uniform predictive distributions, a reranking mechanism to reduce label flipping, a hybrid loss to recover labels when violations occur, and beam search to explore a broader adversarial search space. We further provide theoretical insights linking model generalization, dataset complexity, and calibration vulnerability, showing that models with higher accuracy or trained on datasets with more classes are more susceptible under this threat model. Extensive experiments demonstrate that UGCA substantially increases Expected Calibration Error while preserving classification accuracy. Our code is publicly available at https://github.com/CaptainCuong/Graph-Calibration-Attack.git.

cs.LG↗
arXiv2026

Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads

Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among PEFT methods, Low-Rank Adaptation (LoRA) stands out for its effectiveness by inserting trainable low-rank matrices into weight updates to enable efficient adaptation. However, when applied to multi-head self-attention, existing LoRA-based methods typically fine-tune each attention head independently, overlooking potential interactions and shared structure among heads. To address this limitation, we propose Hypernetwork-Driven Low-rank Adaptation (HyRA) that employs a hypernetwork to generate joint low-rank matrices for all attention heads within a layer. The shared generator promotes cross-head information sharing, helping low-rank modules avoid the redundant feature learning seen in traditional LoRA methods. Theoretically, our method achieves significantly better sample efficiency compared to standard LoRA. Empirically, we evaluate HyRA on a comprehensive suite of language and vision benchmarks. Our approach consistently outperforms existing parameter-efficient fine-tuning (PEFT) baselines across a wide range of tasks. Notably, in low-data regimes, HyRA achieves substantial improvements over LoRA, underscoring its practical sample efficiency and effectiveness in data-scarce scenarios.

cs.LG↗
arXiv2026

Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity

Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioning as a trainable expert, in order to learn a new task. Despite recent efforts to analyze the convergence behavior of parameter estimation in this model, there are still two unresolved problems in the literature. First, the contaminated MoE model has been studied solely in regression settings, while its theoretical foundation in classification settings remains absent. Second, previous works on MoE models for classification capture pointwise convergence rates for parameter estimation without any guaranty of minimax optimality. In this work, we close these gaps by performing, for the first time, the convergence analysis of a contaminated mixture of multinomial logistic experts with homogeneous and heterogeneous structures, respectively. In each regime, we characterize uniform convergence rates for estimating parameters under challenging settings where ground-truth parameters vary with the sample size. Furthermore, we also establish corresponding minimax lower bounds to ensure that these rates are minimax optimal. Notably, our theories offer an important insight into the design of contaminated MoE, that is, expert heterogeneity yields faster parameter estimation rates and, therefore, is more sample-efficient than expert homogeneity.

math.ST↗
arXiv2025

On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts

The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstream tasks by including a new contamination part, or prompt, functioning as a new, trainable expert. Despite its popularity and relevance, the theoretical properties of the softmax-contaminated MoE have remained unexplored in the literature. In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters in order to gain insights into the statistical properties and potential challenges of fine-tuning with a new prompt. We find that the estimability of these parameters is compromised when the prompt acquires overlapping knowledge with the pre-trained model, in the sense that we make precise by formulating a novel analytic notion of distinguishability. Under distinguishability of the pre-trained and prompt models, we derive minimax optimal estimation rates for all the gating and prompt parameters. By contrast, when the distinguishability condition is violated, these estimation rates become significantly slower due to their dependence on the prompt convergence rate to the pre-trained model. Finally, we empirically corroborate our theoretical findings through several numerical experiments.

stat.ML↗
arXiv2025

Formal Reasoning for Intelligent QA Systems: A Case Study in the Educational Domain

Reasoning is essential for closed-domain QA systems in which procedural correctness and policy compliance are critical. While large language models (LLMs) have shown strong performance on many reasoning tasks, recent work reveals that their reasoning traces are often unfaithful - serving more as plausible justifications than as causally grounded derivations. Efforts to combine LLMs with symbolic engines (e.g., Prover9, Z3) have improved reliability but remain limited to static forms of logic, struggling with dynamic, state-based reasoning such as multi-step progressions and conditional transitions. In this paper, we propose MCFR (Model Checking for Formal Reasoning), a neuro-symbolic framework that integrates LLMs with model checking to support property verification. MCFR translates natural language into formal specifications and verifies them over transition models. To support evaluation, we introduce EduMC-QA, a benchmark dataset grounded in real academic procedures. Our results show that MCFR improves reasoning faithfulness and interpretability, offering a viable path toward verifiable QA in high-stakes closed-domain applications. In addition to evaluating MCFR, we compare its performance with state-of-the-art LLMs such as ChatGPT, DeepSeek, and Claude to contextualize its effectiveness.

cs.AI↗
arXiv2025

Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts

We conduct the convergence analysis of parameter estimation in the contaminated mixture of experts. This model is motivated from the prompt learning problem where ones utilize prompts, which can be formulated as experts, to fine-tune a large-scale pre-trained model for learning downstream tasks. There are two fundamental challenges emerging from the analysis: (i) the proportion in the mixture of the pre-trained model and the prompt may converge to zero during the training, leading to the prompt vanishing issue; (ii) the algebraic interaction among parameters of the pre-trained model and the prompt can occur via some partial differential equations and decelerate the prompt learning. In response, we introduce a distinguishability condition to control the previous parameter interaction. Additionally, we also investigate various types of expert structure to understand their effects on the convergence behavior of parameter estimation. In each scenario, we provide comprehensive convergence rates of parameter estimation along with the corresponding minimax lower bounds. Finally, we run several numerical experiments to empirically justify our theoretical findings.

cs.LG↗
arXiv2024

On Barycenter Computation: Semi-Unbalanced Optimal Transport-based Method on Gaussians

We explore a robust version of the barycenter problem among $n$ centered Gaussian probability measures, termed Semi-Unbalanced Optimal Transport (SUOT)-based Barycenter, wherein the barycenter remains fixed while the others are relaxed using Kullback-Leibler divergence. We develop optimization algorithms on Bures-Wasserstein manifold, named the Exact Geodesic Gradient Descent and Hybrid Gradient Descent algorithms. While the Exact Geodesic Gradient Descent method is based on computing the exact closed form of the first-order derivative of the objective function of the barycenter along a geodesic on the Bures manifold, the Hybrid Gradient Descent method utilizes optimizer components when solving the SUOT problem to replace outlier measures before applying the Riemannian Gradient Descent. We establish the theoretical convergence guarantees for both methods and demonstrate that the Exact Geodesic Gradient Descent algorithm attains a dimension-free convergence rate. Finally, we conduct experiments to compare the normal Wasserstein Barycenter with ours and perform an ablation study.

cs.LG↗
arXiv2024

Fixed points of maps and nontrivial weak solutions to a class of nonlinear strongly coupled elliptic systems

Local indices at isolated fixed points of a differentiable compact nonlinear map $T$ on Banach spaces will be discussed. These results are applied to establish the existence of nontrivial solutions. As an example, the existence of nontrivial weak solutions to a class of nonlinear strongly coupled elliptic systems of two equations will also be studied. A bifurcation phenomenon due to cross-diffusion is reported.

math.AP↗