arXiv subjects

Zhongyi Huang

Publications and source records attributed to Zhongyi Huang.

At least 37 records · Page 2Linked to original sources

arXiv2025

Grid-like Error-Correcting Codes for Matrix Multiplication with Better Correcting Capability

Matrix multiplication over the real field constitutes a foundational operation in the training of deep learning models, serving as a computational cornerstone for both forward and backward propagation processes. However, the presence of silent data corruption (SDC) in large-scale distributed training environments poses a significant threat to model convergence and predictive accuracy, particularly when such errors manifest during matrix multiplication. Due to their transient and non-intrusive nature, these errors often evade detection, allowing them to propagate and accumulate over time, ultimately leading to substantial degradation in model performance. In this paper, we introduce a novel error-correcting coding framework specifically tailored for matrix multiplication operations. Our proposed framework is designed to detect and correct multiple computational errors that may arise during the execution of matrix products. By leveraging a grid-based structural encoding scheme, our approach enhances error localization and correction capabilities across all participating matrices, thereby significantly improving the fault tolerance of the computation. Experimental results demonstrate that our method achieves deterministic correction of up to two erroneous symbols distributed across three matrices with 100\% reliability, while incurring only a 24\% overhead in computational time on GPU architectures. Furthermore, we provide a rigorous theoretical analysis of the error-correction properties inherent to our coding scheme, establishing its correctness and robustness under well-defined fault models.

cs.IT↗

arXiv2025

Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks

The optimization algorithms are crucial in training physics-informed neural networks (PINNs), as unsuitable methods may lead to poor solutions. Compared to the common gradient descent (GD) algorithm, implicit gradient descent (IGD) outperforms it in handling certain multi-scale problems. In this paper, we provide convergence analysis for the IGD in training over-parameterized two-layer PINNs. We first derive the training dynamics of IGD in training two-layer PINNs. Then, over-parameterization allows us to prove that the randomly initialized IGD converges to a globally optimal solution at a linear convergence rate. Moreover, due to the distinct training dynamics of IGD compared to GD, the learning rate can be selected independently of the sample size and the least eigenvalue of the Gram matrix. Additionally, the novel approach used in our convergence analysis imposes a milder requirement on the network width. Finally, empirical results validate our theoretical findings.

cs.LG↗

arXiv2025

The Role of Rank in Mismatched Low-Rank Symmetric Matrix Estimation

We investigate the performance of a Bayesian statistician tasked with recovering a rank-$k$ signal matrix $\bS \bS^{\top} \in \mathbb{R}^{n \times n}$, corrupted by element-wise additive Gaussian noise. This problem lies at the core of numerous applications in machine learning, signal processing, and statistics. We derive an analytic expression for the asymptotic mean-square error (MSE) of the Bayesian estimator under mismatches in the assumed signal rank, signal power, and signal-to-noise ratio (SNR), considering both sphere and Gaussian signals. Additionally, we conduct a rigorous analysis of how rank mismatch influences the asymptotic MSE. Our primary technical tools include the spectrum of Gaussian orthogonal ensembles (GOE) with low-rank perturbations and asymptotic behavior of $k$-dimensional spherical integrals.

cs.IT↗

arXiv2025

Heterogeneous and anisotropic elastic parameter estimation using a novel semi-analytical forward solver

An efficient procedure using a novel semi-analytical forward solver for identifying heterogeneous and anisotropic elastic parameters from only one full-field measurement is proposed and explored. We formulate the inverse problem as an special energy functional minimization with total variation(TV) regularization. The minimization problem is solved by Adam algorithm, which only requires solving one forward problem and no adjoint problem in each iteration. In order to deal with the irregularity of the elastic regions, the anisotropy and heterogeneity of parameters and potential singularities in forward-modeled issues, a novel semi-analytical forward solver named the direct method of lines is proposed, which discretizes angular variable while preserving analytical solutions along remaining coordinates. To validate the efficacy of our procedure, a series of numerical experiments are implemented subsequently, achieving reliable performance in both forward modeling and the six elastic arguments reconstruction scenarios.

math.NA↗

arXiv2025

Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

In the context of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent (GD) converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD for training two-layer neural networks exhibits poor dependence on the sample size and the Gram matrix, leading to a slow training process. In this paper, we show that for training two-layer $\text{ReLU}^3$ Physics-Informed Neural Networks (PINNs), the learning rate can be improved from $\mathcal{O}(λ_0)$ to $\mathcal{O}(1/\|\bm{H}^{\infty}\|_2)$, implying that GD actually enjoys a faster convergence rate. Despite such improvements, the convergence rate is still tied to the least eigenvalue of the Gram matrix, leading to slow convergence. We then develop the positive definiteness of Gram matrices with general smooth activation functions and provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs, demonstrating that the learning rate can be $\mathcal{O}(1)$ and at this rate, the convergence rate is independent of the Gram matrix. In particular, for smooth activation functions, the convergence rate of NGD is quadratic. Numerical experiments are conducted to verify our theoretical results.

cs.LG↗

arXiv2025

Refined generalization analysis of the Deep Ritz Method and Physics-Informed Neural Networks

In this paper, we derive refined generalization bounds for the Deep Ritz Method (DRM) and Physics-Informed Neural Networks (PINNs). For the DRM, we focus on two prototype elliptic partial differential equations (PDEs): Poisson equation and static Schrödinger equation on the $d$-dimensional unit hypercube with the Neumann boundary condition. Furthermore, sharper generalization bounds are derived based on the localization techniques under the assumptions that the exact solutions of the PDEs lie in the Barron spaces or the general Sobolev spaces. For the PINNs, we investigate the general linear second order elliptic PDEs with Dirichlet boundary condition using the local Rademacher complexity in the multi-task learning setting. Finally, we discuss the generalization error in the setting of over-parameterization when solutions of PDEs belong to Barron space.

math.NA↗

arXiv2025

Decoupling Classifier for Boosting Few-shot Object Detection and Instance Segmentation

This paper focus on few-shot object detection~(FSOD) and instance segmentation~(FSIS), which requires a model to quickly adapt to novel classes with a few labeled instances. The existing methods severely suffer from bias classification because of the missing label issue which naturally exists in an instance-level few-shot scenario and is first formally proposed by us. Our analysis suggests that the standard classification head of most FSOD or FSIS models needs to be decoupled to mitigate the bias classification. Therefore, we propose an embarrassingly simple but effective method that decouples the standard classifier into two heads. Then, these two individual heads are capable of independently addressing clear positive samples and noisy negative samples which are caused by the missing label. In this way, the model can effectively learn novel classes while mitigating the effects of noisy negative samples. Without bells and whistles, our model without any additional computation cost and parameters consistently outperforms its baseline and state-of-the-art by a large margin on PASCAL VOC and MS-COCO benchmarks for FSOD and FSIS tasks. The Code is available at https://csgaobb.github.io/Projects/DCFS.

cs.CV↗

arXiv2025

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

In recent years, neural networks have achieved remarkable progress in various fields and have also drawn much attention in applying them on scientific problems. A line of methods involving neural networks for solving partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs) and the Deep Ritz Method (DRM), has emerged. Although these methods outperform classical numerical methods in certain cases, the optimization problems involving neural networks are typically non-convex and non-smooth, which can result in unsatisfactory solutions for PDEs. In contrast to deterministic neural networks, the hidden weights of random neural networks are sampled from some prior distribution and only the output weights participate in training. This makes training much simpler, but it remains unclear how to select the prior distribution. In this paper, we focus on Barron type functions and approximate them under Sobolev norms by random neural networks with clear prior distribution. In addition to the approximation error, we also derive bounds for the optimization and generalization errors of random neural networks for solving PDEs when the solutions are Barron type functions.

math.NA↗

arXiv2025

Capacity-achieving sparse superposition codes with spatially coupled VAMP decoder

Sparse superposition (SS) codes provide an efficient communication scheme over the Gaussian channel, utilizing the vector approximate message passing (VAMP) decoder for rotational invariant design matrices. Previous work has established that the VAMP decoder for SS achieves Shannon capacity when the design matrix satisfies a specific spectral criterion and exponential decay power allocation is used. In this work, we propose a spatially coupled VAMP (SC-VAMP) decoder for SS with spatially coupled design matrices. Based on state evolution (SE) analysis, we demonstrate that the SC-VAMP decoder is capacity-achieving when the design matrices satisfy the spectra criterion. Empirically, we show that the SC-VAMP decoder outperforms the VAMP decoder with exponential decay power allocation, achieving a lower section error rate. All codes are available on https://github.com/yztfu/SC-VAMP-for-Superposition-Code.git.

cs.IT↗

arXiv2025

Image Restoration Models with Optimal Transport and Total Variation Regularization

In this paper, we propose image restoration models using optimal transport (OT) and total variation regularization. We present theoretical results of the proposed models based on the relations between the dual Lipschitz norm from OT and the G-norm introduced by Yves Meyer. We design a numerical method based on the Primal-Dual Hybrid Gradient (PDHG) algorithm for the Wasserstain distance and the augmented Lagrangian method (ALM) for the total variation, and the convergence analysis of the proposed numerical method is established. We also consider replacing the total variation in our model by one of its modifications developed in \cite{zhu}, with the aim of suppressing the stair-casing effect and preserving image contrasts. Numerical experiments demonstrate the features of the proposed models.

math.NA↗

arXiv2025

Computation of a Unified Graph-Based Rate Optimization Problem

We define a graph-based rate optimization problem and consider its computation, which provides a unified approach to the computation of various theoretical limits, including the (conditional) graph entropy, rate-distortion functions and capacity-cost functions with side information. Compared with their classical counterparts, theoretical limits with side information are much more difficult to compute since their characterizations as optimization problems have larger and more complex feasible regions. Following the unified approach, we develop effective methods to resolve the difficulty. On the theoretical side, we derive graph characterizations for rate-distortion and capacity-cost functions with side information and simplify the characterizations in special cases by reducing the number of decision variables. On the computational side, we design an efficient alternating minimization algorithm for the graph-based problem, which deals with the inequality constraint by a flexible multiplier update strategy. Moreover, simplified graph characterizations are exploited and deflation techniques are introduced, so that the computing time is greatly reduced. Theoretical analysis shows that the algorithm converges to an optimal solution. By numerical experiments, the accuracy and efficiency of the algorithm are illustrated and its significant advantage over existing methods is demonstrated.

cs.IT↗

arXiv2025

Accelerating Data Access for Single Node in Distributed Storage Systems via MDS Codes

Maximum distance separable (MDS) array codes are widely employed in modern distributed storage systems to provide high data reliability with small storage overhead. Compared with the data access latency of the entire file, the data access latency of a single node in a distributed storage system is equally important. In this paper, we propose two algorithms to effectively reduce the data access latency on a single node in different scenarios for MDS codes. We show theoretically that our algorithms have an expected reduction ratio of $\frac{(n-k)(n-k+1)}{n(n+1)}$ and $\frac{n-k}{n}$ for the data access latency of a single node when it obeys uniform distribution and shifted-exponential distribution, respectively, where $n$ and $k$ are the numbers of all nodes and the number of data nodes respectively. In the worst-case analysis, we show that our algorithms have a reduction ratio of more than $60\%$ when $(n,k)=(3,2)$. Furthermore, in simulation experiments, we use the Monte Carlo simulation algorithm to demonstrate less data access latency compared with the baseline algorithm.

cs.IT↗

arXiv2025

Distributed Nonparametric Estimation: from Sparse to Dense Samples per Terminal

Consider the communication-constrained problem of nonparametric function estimation, in which each distributed terminal holds multiple i.i.d. samples. Under certain regularity assumptions, we characterize the minimax optimal rates for all regimes, and identify phase transitions of the optimal rates as the samples per terminal vary from sparse to dense. This fully solves the problem left open by previous works, whose scopes are limited to regimes with either dense samples or a single sample per terminal. To achieve the optimal rates, we design a layered estimation protocol by exploiting protocols for the parametric density estimation problem. We show the optimality of the protocol using information-theoretic methods and strong data processing inequalities, and incorporating the classic balls and bins model. The optimal rates are immediate for various special cases such as density estimation, Gaussian, binary, Poisson and heteroskedastic regression models.

cs.LG↗

arXiv2025

Convergence analysis of wide shallow neural operators within the framework of Neural Tangent Kernel

Neural operators are aiming at approximating operators mapping between Banach spaces of functions, achieving much success in the field of scientific computing. Compared to certain deep learning-based solvers, such as Physics-Informed Neural Networks (PINNs), Deep Ritz Method (DRM), neural operators can solve a class of Partial Differential Equations (PDEs). Although much work has been done to analyze the approximation and generalization error of neural operators, there is still a lack of analysis on their training error. In this work, we conduct the convergence analysis of gradient descent for the wide shallow neural operators and physics-informed shallow neural operators within the framework of Neural Tangent Kernel (NTK). The core idea lies on the fact that over-parameterization and random initialization together ensure that each weight vector remains near its initialization throughout all iterations, yielding the linear convergence of gradient descent. In this work, we demonstrate that under the setting of over-parametrization, gradient descent can find the global minimum regardless of whether it is in continuous time or discrete time.

cs.LG↗

arXiv2024

Computing Approximate Graph Edit Distance via Optimal Transport

Given a graph pair $(G^1, G^2)$, graph edit distance (GED) is defined as the minimum number of edit operations converting $G^1$ to $G^2$. GED is a fundamental operation widely used in many applications, but its exact computation is NP-hard, so the approximation of GED has gained a lot of attention. Data-driven learning-based methods have been found to provide superior results compared to classical approximate algorithms, but they directly fit the coupling relationship between a pair of vertices from their vertex features. We argue that while pairwise vertex features can capture the coupling cost (discrepancy) of a pair of vertices, the vertex coupling matrix should be derived from the vertex-pair cost matrix through a more well-established method that is aware of the global context of the graph pair, such as optimal transport. In this paper, we propose an ensemble approach that integrates a supervised learning-based method and an unsupervised method, both based on optimal transport. Our learning method, GEDIOT, is based on inverse optimal transport that leverages a learnable Sinkhorn algorithm to generate the coupling matrix. Our unsupervised method, GEDGW, models GED computation as a linear combination of optimal transport and its variant, Gromov-Wasserstein discrepancy, for node and edge operations, respectively, which can be solved efficiently without needing the ground truth. Our ensemble method, GEDHOT, combines GEDIOT and GEDGW to further boost the performance. Extensive experiments demonstrate that our methods significantly outperform the existing methods in terms of the performance of GED computation, edit path generation, and model generalizability.

cs.LG↗

arXiv2024

A Simplified Algorithm for Joint Real-Time Synchronization, NLoS Identification, and Multi-Agent Localization

Real-time, high-precision localization in large-scale wireless networks faces two primary challenges: clock offsets caused by network asynchrony and non-line-of-sight (NLoS) conditions. To tackle these challenges, we propose a low-complexity real-time algorithm for joint synchronization and NLoS identification-based localization. For precise synchronization, we resolve clock offsets based on accumulated time-of-arrival measurements from all the past time instances, modeling it as a large-scale linear least squares (LLS) problem. To alleviate the high computational burden of solving this LLS, we introduce the blockwise recursive Moore-Penrose inverse (BRMP) technique, a generalized recursive least squares approach, and derive a simplified formulation of BRMP tailored specifically for the real-time synchronization problem. Furthermore, we formulate joint NLoS identification and localization as a robust least squares regression (RLSR) problem and address it by using an efficient iterative approach. Simulations show that the proposed algorithm achieves sub-nanosecond synchronization accuracy and centimeter-level localization precision, while maintaining low computational overhead.

eess.SP↗

arXiv2024

Adaptive Refinement Protocols for Distributed Distribution Estimation under $\ell^p$-Losses

Consider the communication-constrained estimation of discrete distributions under $\ell^p$ losses, where each distributed terminal holds multiple independent samples and uses limited number of bits to describe the samples. We obtain the minimax optimal rates of the problem in most parameter regimes. An elbow effect of the optimal rates at $p=2$ is clearly identified. To show the optimal rates, we first design estimation protocols to achieve them. The key ingredient of these protocols is to introduce adaptive refinement mechanisms, which first generate rough estimate by partial information and then establish refined estimate in subsequent steps guided by the rough estimate. The protocols leverage successive refinement, sample compression, thresholding and random hashing methods to achieve the optimal rates in different parameter regimes. The optimality of the protocols is shown by deriving compatible minimax lower bounds.

cs.LG↗

arXiv2024

Multiplexed Streaming Codes for Messages with Different Decoding Delays in Channel with Burst and Random Erasures

In a real-time transmission scenario, messages are transmitted through a channel that is subject to packet loss. The destination must recover the messages within the required deadline. In this paper, we consider a setup where two different types of messages with distinct decoding deadlines are transmitted through a channel model that introduces either one burst erasure of length at most $B$, or $N$ random erasures in any fixed-sized sliding window. The message with a short decoding deadline $T_{\mathrm{u}}$ is referred to as an urgent message, while the other one with a decoding deadline $T_{\mathrm{v}}$ ($T_{\mathrm{v}} > T_{\mathrm{u}}$) is referred to as a less urgent message. We consider the scenario where $T_{\mathrm{v}} > T_{\mathrm{u}} + B$ and propose a non-trivial achievable region $\mathcal{R}$ for the aforementioned channel model. We propose a novel merging approach to encode two message streams of different urgency levels into a single flow and present explicit constructions for encoding, contributing to the establishment of the achievability of region $\mathcal{R}$. Our comprehensive analysis demonstrates that this region encompasses the rate pairs of existing encoding schemes and coincides with the capacity region in burst channel scenarios. Lastly, we investigate the property of the achievable region $\mathcal{R}$, proving that it is the largest one obtained from all the rate pairs under the merging method.

cs.IT↗