arXiv subjects

Kai Fong Ernest Chong

Publications and source records attributed to Kai Fong Ernest Chong.

15 recordsLinked to original sources

arXiv2025

Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping

Leveraging over-the-air computations for model aggregation is an effective approach to cope with the communication bottleneck in federated edge learning. By exploiting the superposition properties of multi-access channels, this approach facilitates an integrated design of communication and computation, thereby enhancing system privacy while reducing implementation costs. However, the inherent electromagnetic interference in radio channels often exhibits heavy-tailed distributions, giving rise to exceptionally strong noise in globally aggregated gradients that can significantly deteriorate the training performance. To address this issue, we propose a novel gradient clipping method, termed Median Anchored Clipping (MAC), to combat the detrimental effects of heavy-tailed noise. We also derive analytical expressions for the convergence rate of model training with analog over-the-air federated learning under MAC, which quantitatively demonstrates the effect of MAC on training performance. Extensive experimental results show that the proposed MAC algorithm effectively mitigates the impact of heavy-tailed noise, hence substantially enhancing system robustness.

cs.LG↗

arXiv2024

The face numbers of homology spheres

The $g$-theorem is a momentous result in combinatorics that gives a complete numerical characterization of the face numbers of simplicial convex polytopes. The $g$-conjecture asserts that the same numerical conditions given in the $g$-theorem also characterizes the face numbers of all simplicial spheres, or even more generally, all simplicial homology spheres. In this paper, we prove the $g$-conjecture for simplicial $\mathbb{R}$-homology spheres. A key idea in our proof is a new algebra structure for polytopal complexes. Given a polytopal $d$-complex $Δ$, we use ideas from rigidity theory to construct a graded Artinian $\mathbb{R}$-algebra $Ψ(Δ,ν)$ of stresses on a PL realization $ν$ of $Δ$ in $\mathbb{R}^d$, where overlapping realized $d$-faces are allowed. In particular, we prove that if $Δ$ is a simplicial $\mathbb{R}$-homology sphere, then for generic PL realizations $ν$, the stress algebra $Ψ(Δ,ν)$ is Gorenstein and has the weak Lefschetz property.

math.CO↗

arXiv2024

The Role of Federated Learning in a Wireless World with Foundation Models

Foundation models (FMs) are general-purpose artificial intelligence (AI) models that have recently enabled multiple brand-new generative AI applications. The rapid advances in FMs serve as an important contextual backdrop for the vision of next-generation wireless networks, where federated learning (FL) is a key enabler of distributed network intelligence. Currently, the exploration of the interplay between FMs and FL is still in its nascent stage. Naturally, FMs are capable of boosting the performance of FL, and FL could also leverage decentralized data and computing resources to assist in the training of FMs. However, the exceptionally high requirements that FMs have for computing resources, storage, and communication overhead would pose critical challenges to FL-enabled wireless networks. In this article, we explore the extent to which FMs are suitable for FL over wireless networks, including a broad overview of research challenges and opportunities. In particular, we discuss multiple new paradigms for realizing future intelligent networks that integrate FMs and FL. We also consolidate several broad research directions associated with these paradigms.

cs.NI↗

arXiv2024

Spectral Co-Distillation for Personalized Federated Learning

Personalized federated learning (PFL) has been widely investigated to address the challenge of data heterogeneity, especially when a single generic model is inadequate in satisfying the diverse performance requirements of local clients simultaneously. Existing PFL methods are inherently based on the idea that the relations between the generic global and personalized local models are captured by the similarity of model weights. Such a similarity is primarily based on either partitioning the model architecture into generic versus personalized components, or modeling client relationships via model weights. To better capture similar (yet distinct) generic versus personalized model representations, we propose \textit{spectral distillation}, a novel distillation method based on model spectrum information. Building upon spectral distillation, we also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training. Moreover, to utilize the local idle time in conventional PFL, we propose a wait-free local training protocol. Through extensive experiments on multiple datasets over diverse heterogeneous data settings, we demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.

cs.LG↗

arXiv2023

GenKL: An Iterative Framework for Resolving Label Ambiguity and Label Non-conformity in Web Images Via a New Generalized KL Divergence

Web image datasets curated online inherently contain ambiguous in-distribution (ID) instances and out-of-distribution (OOD) instances, which we collectively call non-conforming (NC) instances. In many recent approaches for mitigating the negative effects of NC instances, the core implicit assumption is that the NC instances can be found via entropy maximization. For "entropy" to be well-defined, we are interpreting the output prediction vector of an instance as the parameter vector of a multinomial random variable, with respect to some trained model with a softmax output layer. Hence, entropy maximization is based on the idealized assumption that NC instances have predictions that are "almost" uniformly distributed. However, in real-world web image datasets, there are numerous NC instances whose predictions are far from being uniformly distributed. To tackle the limitation of entropy maximization, we propose $(α, β)$-generalized KL divergence, $\mathcal{D}_{\text{KL}}^{α, β}(p\|q)$, which can be used to identify significantly more NC instances. Theoretical properties of $\mathcal{D}_{\text{KL}}^{α, β}(p\|q)$ are proven, and we also show empirically that a simple use of $\mathcal{D}_{\text{KL}}^{α, β}(p\|q)$ outperforms all baselines on the NC instance identification task. Building upon $(α,β)$-generalized KL divergence, we also introduce a new iterative training framework, GenKL, that identifies and relabels NC instances. When evaluated on three web image datasets, Clothing1M, Food101/Food101N, and mini WebVision 1.0, we achieved new state-of-the-art classification accuracies: $81.34\%$, $85.73\%$ and $78.99\%$/$92.54\%$ (top-1/top-5), respectively.

cs.CV↗

arXiv2023

Abstract Visual Reasoning: An Algebraic Approach for Solving Raven's Progressive Matrices

We introduce algebraic machine reasoning, a new reasoning framework that is well-suited for abstract reasoning. Effectively, algebraic machine reasoning reduces the difficult process of novel problem-solving to routine algebraic computation. The fundamental algebraic objects of interest are the ideals of some suitably initialized polynomial ring. We shall explain how solving Raven's Progressive Matrices (RPMs) can be realized as computational problems in algebra, which combine various well-known algebraic subroutines that include: Computing the Gröbner basis of an ideal, checking for ideal containment, etc. Crucially, the additional algebraic structure satisfied by ideals allows for more operations on ideals beyond set-theoretic operations. Our algebraic machine reasoning framework is not only able to select the correct answer from a given answer set, but also able to generate the correct answer with only the question matrix given. Experiments on the I-RAVEN dataset yield an overall $93.2\%$ accuracy, which significantly outperforms the current state-of-the-art accuracy of $77.0\%$ and exceeds human performance at $84.4\%$ accuracy.

cs.CV↗

arXiv2022

FedCorr: Multi-Stage Federated Learning for Label Noise Correction

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose $\texttt{FedCorr}$, a general multi-stage framework to tackle heterogeneous label noise in FL, without making any assumptions on the noise models of local clients, while still maintaining client data privacy. In particular, (1) $\texttt{FedCorr}$ dynamically identifies noisy clients by exploiting the dimensionalities of the model prediction subspaces independently measured on all clients, and then identifies incorrect labels on noisy clients based on per-sample losses. To deal with data heterogeneity and to increase training stability, we propose an adaptive local proximal regularization term that is based on estimated local noise levels. (2) We further finetune the global model on identified clean clients and correct the noisy labels for the remaining noisy clients after finetuning. (3) Finally, we apply the usual training on all clients to make full use of all local data. Experiments conducted on CIFAR-10/100 with federated synthetic label noise, and on a real-world noisy dataset, Clothing1M, demonstrate that $\texttt{FedCorr}$ is robust to label noise and substantially outperforms the state-of-the-art methods at multiple noise levels.

cs.LG↗

arXiv2021

Dynamic Attention-based Communication-Efficient Federated Learning

Federated learning (FL) offers a solution to train a global machine learning model while still maintaining data privacy, without needing access to data stored locally at the clients. However, FL suffers performance degradation when client data distribution is non-IID, and a longer training duration to combat this degradation may not necessarily be feasible due to communication limitations. To address this challenge, we propose a new adaptive training algorithm $\texttt{AdaFL}$, which comprises two components: (i) an attention-based client selection mechanism for a fairer training scheme among the clients; and (ii) a dynamic fraction method to balance the trade-off between performance stability and communication efficiency. Experimental results show that our $\texttt{AdaFL}$ algorithm outperforms the usual $\texttt{FedAvg}$ algorithm, and can be incorporated to further improve various state-of-the-art FL algorithms, with respect to three aspects: model accuracy, performance stability, and communication efficiency.

cs.LG↗

arXiv2021

Training Classifiers that are Universally Robust to All Label Noise Levels

For classification tasks, deep neural networks are prone to overfitting in the presence of label noise. Although existing methods are able to alleviate this problem at low noise levels, they encounter significant performance reduction at high noise levels, or even at medium noise levels when the label noise is asymmetric. To train classifiers that are universally robust to all noise levels, and that are not sensitive to any variation in the noise model, we propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. In particular, we shall assume that a small subset of any given noisy dataset is known to have correct labels, which we treat as "positive", while the remaining noisy subset is treated as "unlabeled". Our framework consists of the following two components: (1) We shall generate, via iterative updates, an augmented clean subset with additional reliable "positive" samples filtered from "unlabeled" samples; (2) We shall train a teacher model on this larger augmented clean set. With the guidance of the teacher model, we then train a student model on the whole dataset. Experiments were conducted on the CIFAR-10 dataset with synthetic label noise at multiple noise levels for both symmetric and asymmetric noise. The results show that our framework generally outperforms at medium to high noise levels. We also evaluated our framework on Clothing1M, a real-world noisy dataset, and we achieved 2.94% improvement in accuracy over existing state-of-the-art methods.

cs.LG↗

arXiv2020

A closer look at the approximation capabilities of neural networks

The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions $σ$, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function $f$ to any given approximation threshold $\varepsilon$, if and only if $σ$ is non-polynomial. In this paper, we give a direct algebraic proof of the theorem. Furthermore we shall explicitly quantify the number of hidden units required for approximation. Specifically, if $X\subseteq \mathbb{R}^n$ is compact, then a neural network with $n$ input units, $m$ output units, and a single hidden layer with $\binom{n+d}{d}$ hidden units (independent of $m$ and $\varepsilon$), can uniformly approximate any polynomial function $f:X \to \mathbb{R}^m$ whose total degree is at most $d$ for each of its $m$ coordinate functions. In the general case that $f$ is any continuous function, we show there exists some $N\in \mathcal{O}(\varepsilon^{-n})$ (independent of $m$), such that $N$ hidden units would suffice to approximate $f$. We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights. We highlight several consequences: (i) For any $δ> 0$, the UAP still holds if we restrict all non-bias weights $w$ in the last layer to satisfy $|w| < δ$. (ii) There exists some $λ>0$ (depending only on $f$ and $σ$), such that the UAP still holds if we restrict all non-bias weights $w$ in the first layer to satisfy $|w|>λ$. (iii) If the non-bias weights in the first layer are \emph{fixed} and randomly chosen from a suitable range, then the UAP holds with probability $1$.

cs.LG↗

arXiv2019

Flag complexes and homology

We prove several relations on the $f$-vectors and Betti numbers of flag complexes. For every flag complex $Δ$, we show that there exists a balanced complex with the same $f$-vector as $Δ$, and whose top-dimensional Betti number is at least that of $Δ$, thereby extending a theorem of Frohmader by additionally taking homology into consideration. We obtain upper bounds on the top-dimensional Betti number of $Δ$ in terms of its face numbers. We also give a quantitative refinement of a theorem of Meshulam by establishing lower bounds on the $f$-vector of $Δ$, in terms of the top-dimensional Betti number of $Δ$. This result has a continuous analog: If $Δ$ is a $(d-1)$-dimensional flag complex whose $(d-1)$-th reduced homology group has dimension $a\geq 0$ (over some field), then the $f$-polynomial of $Δ$ satisfies the coefficient-wise inequality $f_Δ(x) \geq (1 + (\sqrt[d]{a}+1)x)^d$.

math.CO↗

arXiv2014

Hilbert functions of colored quotient rings and a generalization of the Clements-Lindström theorem

Given a polynomial ring $S = \Bbbk[x_1, \dots, x_n]$ over a field $\Bbbk$, and a monomial ideal $M$ of $S$, we say the quotient ring $R = S/M$ is Macaulay-Lex if for every graded ideal of $R$, there exists a lexicographic ideal of $R$ with the same Hilbert function. In this paper, we introduce a class of quotient rings with combinatorial significance, which we call colored quotient rings. This class of rings include Clements-Lindström rings and colored squarefree rings as special cases that are known to be Macaulay-Lex. We construct two new classes of Macaulay-Lex rings, characterize all colored quotient rings that are Macaulay-Lex, and give a simultaneous generalization of both the Clements-Lindström theorem and the Frankl-Füredi-Kalai theorem. We also show that the $f$-vectors of $(a_1, \dots, a_n)$-colored simplicial complexes or multicomplexes are never characterized by "reverse-lexicographic" complexes or multicomplexes when $n>1$ and $(a_1, \dots, a_n) \neq (1, \dots, 1)$.

math.AC↗

arXiv2014

Generalized Macaulay representations and the flag $f$-vectors of generalized colored complexes

A colored complex of type $\mathbf{a} = (a_1, \dots, a_n)$ is a simplicial complex $Δ$ on a vertex set $V$, together with an ordered partition $(V_1, \dots, V_n)$ of $V$, such that every face $F$ of $Δ$ satisfies $|F \cap V_i| \leq a_i$. For each $\mathbf{b} = (b_1, \dots, b_n) \leq \mathbf{a}$, let $f_{\mathbf{b}}$ be the number of faces $F$ of $Δ$ such that $|F \cap V_i| = b_i$. The array of integers $\{f_{\mathbf{b}}\}_{\mathbf{b} \leq \mathbf{a}}$ is called the fine $f$-vector of $Δ$, and it is a refinement of the $f$-vector of $Δ$. In this paper, we generalize the notion of Macaulay representations and give a numerical characterization of the fine $f$-vectors of colored complexes of arbitrary type, in terms of these generalized Macaulay representations. As part of the proof, we introduce the property of $\mathbf{a}$-Macaulay decomposability for simplicial complexes, which implies vertex-decomposability, and we show that every pure color-shifted balanced complex $Δ$ of type $\mathbf{a}$ is $\mathbf{a}$-Macaulay decomposable. Combined with previously known results, we also obtain a numerical characterization of the flag $f$-vectors of completely balanced Cohen-Macaulay complexes.

math.CO↗

arXiv2013

An application of liaison theory to the Eisenbud-Green-Harris conjecture

In this paper, we apply liaison theory to the Eisenbud-Green-Harris conjecture and prove that the conjecture holds for a certain subclass of homogeneous ideals in the linkage class of a complete intersection ideal. In the case of three variables, we prove that the conjecture holds for Gorenstein ideals.

math.AC↗

arXiv2010

Fountain Codes with Varying Probability Distributions

Fountain codes are rateless erasure-correcting codes, i.e., an essentially infinite stream of encoded packets can be generated from a finite set of data packets. Several fountain codes have been proposed recently to minimize overhead, many of which involve modifications of the Luby transform (LT) code. These fountain codes, like the LT code, have the implicit assumption that the probability distribution is fixed throughout the encoding process. In this paper, we will use the theory of posets to show that this assumption is unnecessary, and by dropping it, we can achieve overhead reduction by as much as 64% lower than LT codes. We also present the fundamental theory of probability distribution designs for fountain codes with non-constant probability distributions that minimize overhead.

cs.IT↗