arXiv subjects

Yuhua Li

Publications and source records attributed to Yuhua Li.

50 records · Page 3Linked to original sources

arXiv2024

MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning

Humans exhibit a remarkable ability to learn quickly from a limited number of labeled samples, a capability that starkly contrasts with that of current machine learning systems. Unsupervised Few-Shot Learning (U-FSL) seeks to bridge this divide by reducing reliance on annotated datasets during initial training phases. In this work, we first quantitatively assess the impacts of Masked Image Modeling (MIM) and Contrastive Learning (CL) on few-shot learning tasks. Our findings highlight the respective limitations of MIM and CL in terms of discriminative and generalization abilities, which contribute to their underperformance in U-FSL contexts. To address these trade-offs between generalization and discriminability in unsupervised pretraining, we introduce a novel paradigm named Masked Image Contrastive Modeling (MICM). MICM creatively combines the targeted object learning strength of CL with the generalized visual feature learning capability of MIM, significantly enhancing its efficacy in downstream few-shot learning inference. Extensive experimental analyses confirm the advantages of MICM, demonstrating significant improvements in both generalization and discrimination capabilities for few-shot learning. Our comprehensive quantitative evaluations further substantiate the superiority of MICM, showing that our two-stage U-FSL framework based on MICM markedly outperforms existing leading baselines.

cs.CV↗

arXiv2024

FedBAT: Communication-Efficient Federated Learning via Learnable Binarization

Federated learning is a promising distributed machine learning paradigm that can effectively exploit large-scale data without exposing users' privacy. However, it may incur significant communication overhead, thereby potentially impairing the training efficiency. To address this challenge, numerous studies suggest binarizing the model updates. Nonetheless, traditional methods usually binarize model updates in a post-training manner, resulting in significant approximation errors and consequent degradation in model accuracy. To this end, we propose Federated Binarization-Aware Training (FedBAT), a novel framework that directly learns binary model updates during the local training process, thus inherently reducing the approximation errors. FedBAT incorporates an innovative binarization operator, along with meticulously designed derivatives to facilitate efficient learning. In addition, we establish theoretical guarantees regarding the convergence of FedBAT. Extensive experiments are conducted on four popular datasets. The results show that FedBAT significantly accelerates the convergence and exceeds the accuracy of baselines by up to 9\%, even surpassing that of FedAvg in some cases.

cs.LG↗

arXiv2024

Compositional Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) is proposed to continually learn from novel classes with only a few samples after the (pre-)training on base classes with sufficient data. However, this remains a challenge. In contrast, humans can easily recognize novel classes with a few samples. Cognitive science demonstrates that an important component of such human capability is compositional learning. This involves identifying visual primitives from learned knowledge and then composing new concepts using these transferred primitives, making incremental learning both effective and interpretable. To imitate human compositional learning, we propose a cognitive-inspired method for the FSCIL task. We define and build a compositional model based on set similarities, and then equip it with a primitive composition module and a primitive reuse module. In the primitive composition module, we propose to utilize the Centered Kernel Alignment (CKA) similarity to approximate the similarity between primitive sets, allowing the training and evaluation based on primitive compositions. In the primitive reuse module, we enhance primitive reusability by classifying inputs based on primitives replaced with the closest primitives from other classes. Experiments on three datasets validate our method, showing it outperforms current state-of-the-art methods with improved interpretability. Our code is available at https://github.com/Zoilsen/Comp-FSCIL.

cs.CV↗

arXiv2024

Delve into Base-Novel Confusion: Redundancy Exploration for Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) aims to acquire knowledge from novel classes with limited samples while retaining information about base classes. Existing methods address catastrophic forgetting and overfitting by freezing the feature extractor during novel-class learning. However, these methods usually tend to cause the confusion between base and novel classes, i.e., classifying novel-class samples into base classes. In this paper, we delve into this phenomenon to study its cause and solution. We first interpret the confusion as the collision between the novel-class and the base-class region in the feature space. Then, we find the collision is caused by the label-irrelevant redundancies within the base-class feature and pixel space. Through qualitative and quantitative experiments, we identify this redundancy as the shortcut in the base-class training, which can be decoupled to alleviate the collision. Based on this analysis, to alleviate the collision between base and novel classes, we propose a method for FSCIL named Redundancy Decoupling and Integration (RDI). RDI first decouples redundancies from base-class space to shrink the intra-base-class feature space. Then, it integrates the redundancies as a dummy class to enlarge the inter-base-class feature space. This process effectively compresses the base-class feature space, creating buffer space for novel classes and alleviating the model's confusion between the base and novel classes. Extensive experiments across benchmark datasets, including CIFAR-100, miniImageNet, and CUB-200-2011 demonstrate that our method achieves state-of-the-art performance.

cs.CV↗

arXiv2024

Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning

Cross-domain few-shot learning (CDFSL) aims to acquire knowledge from limited training data in the target domain by leveraging prior knowledge transferred from source domains with abundant training samples. CDFSL faces challenges in transferring knowledge across dissimilar domains and fine-tuning models with limited training data. To address these challenges, we initially extend the analysis of loss landscapes from the parameter space to the representation space, which allows us to simultaneously interpret the transferring and fine-tuning difficulties of CDFSL models. We observe that sharp minima in the loss landscapes of the representation space result in representations that are hard to transfer and fine-tune. Moreover, existing flatness-based methods have limited generalization ability due to their short-range flatness. To enhance the transferability and facilitate fine-tuning, we introduce a simple yet effective approach to achieve long-range flattening of the minima in the loss landscape. This approach considers representations that are differently normalized as minima in the loss landscape and flattens the high-loss region in the middle by randomly sampling interpolated representations. We implement this method as a new normalization layer that replaces the original one in both CNNs and ViTs. This layer is simple and lightweight, introducing only a minimal number of additional parameters. Experimental results on 8 datasets demonstrate that our approach outperforms state-of-the-art methods in terms of average accuracy. Moreover, our method achieves performance improvements of up to 9\% compared to the current best approaches on individual datasets. Our code will be released.

cs.CV↗

arXiv2024

Masked Graph Autoencoder with Non-discrete Bandwidths

Masked graph autoencoders have emerged as a powerful graph self-supervised learning method that has yet to be fully explored. In this paper, we unveil that the existing discrete edge masking and binary link reconstruction strategies are insufficient to learn topologically informative representations, from the perspective of message propagation on graph neural networks. These limitations include blocking message flows, vulnerability to over-smoothness, and suboptimal neighborhood discriminability. Inspired by these understandings, we explore non-discrete edge masks, which are sampled from a continuous and dispersive probability distribution instead of the discrete Bernoulli distribution. These masks restrict the amount of output messages for each edge, referred to as "bandwidths". We propose a novel, informative, and effective topological masked graph autoencoder using bandwidth masking and a layer-wise bandwidth prediction objective. We demonstrate its powerful graph topological learning ability both theoretically and empirically. Our proposed framework outperforms representative baselines in both self-supervised link prediction (improving the discrete edge reconstructors by at most 20%) and node classification on numerous datasets, solely with a structure-learning pretext. Our implementation is available at https://github.com/Newiz430/Bandana.

cs.LG↗

arXiv2023

Normalized solutions for some quasilinear elliptic equation with critical Sobolev exponent

Consider the equation \begin{equation*} -Δ_p u =λ|u|^{p-2}u+μ|u|^{q-2}u+|u|^{p^\ast-2}u\ \ {\rm in}\ \R^N \end{equation*} under the normalized constraint $$\int_{ \R^N}|u|^p=c^p,$$ where $-Δ_pu={\rm div} (|\nabla u|^{p-2}\nabla u)$, $1 0$ and $λ\in\R$. In the purely $L^p$-subcritical case, we obtain the existence of ground state solution by virtue of truncation technique, and obtain multiplicity of normalized solutions. In the purely $L^p$-critical and supercritical case, we drive the existence of positive ground state solution, respectively. Finally, we investigate the asymptotic behavior of ground state solutions obtained above as $μ\to0^+$.

math.AP↗

arXiv2023

CSGCL: Community-Strength-Enhanced Graph Contrastive Learning

Graph Contrastive Learning (GCL) is an effective way to learn generalized graph representations in a self-supervised manner, and has grown rapidly in recent years. However, the underlying community semantics has not been well explored by most previous GCL methods. Research that attempts to leverage communities in GCL regards them as having the same influence on the graph, leading to extra representation errors. To tackle this issue, we define ''community strength'' to measure the difference of influence among communities. Under this premise, we propose a Community-Strength-enhanced Graph Contrastive Learning (CSGCL) framework to preserve community strength throughout the learning process. Firstly, we present two novel graph augmentation methods, Communal Attribute Voting (CAV) and Communal Edge Dropping (CED), where the perturbations of node attributes and edges are guided by community strength. Secondly, we propose a dynamic ''Team-up'' contrastive learning scheme, where community strength is used to progressively fine-tune the contrastive objective. We report extensive experiment results on three downstream tasks: node classification, node clustering, and link prediction. CSGCL achieves state-of-the-art performance compared with other GCL methods, validating that community strength brings effectiveness and generality to graph representations. Our code is available at https://github.com/HanChen-HUST/CSGCL.

cs.SI↗

arXiv2023

Supercontinuum generation in dispersion engineered highly doped silica glass waveguides

We investigate the effect of a lower index oxide layer inclusion within a highly doped silica glass slot waveguide for optimized supercontinuum generation at telecom wavelengths. By controlling the thickness of the oxide slot, we demonstrate that one can engineer the waveguide dispersion profile so that to obtain supercontinua with vastly different spectral broadening dynamics and bandwidths. Using this approach, we designed and fabricated a waveguide with a low and flat dispersion profile of less than 43 ps/km/nm across a wavelength range spanning over 1000 nm. We show that, when pumped at the telecom C-band, we can generate a supercontinuum that spans over 1.5 octave, from approximately 817 nm to 2183 nm. The numerical simulations, whose parameters are derived from the measured waveguide dimension and material indices, exhibit good agreement with experimental measurements, where one can observe both a qualitative and quantitative match in the supercontinuum overall spectrum and specific features (e.g. soliton and dispersive wave locations). This study represents an important step forward in the control and manipulation of dispersive and nonlinear dynamics in highly doped silica glass waveguides, paving the way towards advanced on-chip broadband light manipulation.

physics.optics↗

arXiv2022

Margin-Based Few-Shot Class-Incremental Learning with Class-Level Overfitting Mitigation

Few-shot class-incremental learning (FSCIL) is designed to incrementally recognize novel classes with only few training samples after the (pre-)training on base classes with sufficient samples, which focuses on both base-class performance and novel-class generalization. A well known modification to the base-class training is to apply a margin to the base-class classification. However, a dilemma exists that we can hardly achieve both good base-class performance and novel-class generalization simultaneously by applying the margin during the base-class training, which is still under explored. In this paper, we study the cause of such dilemma for FSCIL. We first interpret this dilemma as a class-level overfitting (CO) problem from the aspect of pattern learning, and then find its cause lies in the easily-satisfied constraint of learning margin-based patterns. Based on the analysis, we propose a novel margin-based FSCIL method to mitigate the CO problem by providing the pattern learning process with extra constraint from the margin-based patterns themselves. Extensive experiments on CIFAR100, Caltech-USCD Birds-200-2011 (CUB200), and miniImageNet demonstrate that the proposed method effectively mitigates the CO problem and achieves state-of-the-art performance.

cs.CV↗

arXiv2021

Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training

Distributed asynchronous offline training has received widespread attention in recent years because of its high performance on large-scale data and complex models. As data are distributed from cloud-centric to edge nodes, a big challenge for distributed machine learning systems is how to handle native and natural non-independent and identically distributed (non-IID) data for training. Previous asynchronous training methods do not have a satisfying performance on non-IID data because it would result in that the training process fluctuates greatly which leads to an abnormal convergence. We propose a gradient scheduling algorithm with partly averaged gradients and global momentum (GSGM) for non-IID data distributed asynchronous training. Our key idea is to apply global momentum and local average to the biased gradient after scheduling, in order to make the training process steady. Experimental results show that for non-IID data training under the same experimental conditions, GSGM on popular optimization algorithms can achieve a 20% increase in training stability with a slight improvement in accuracy on Fashion-Mnist and CIFAR-10 datasets. Meanwhile, when expanding distributed scale on CIFAR-100 dataset that results in sparse data distribution, GSGM can perform a 37% improvement on training stability. Moreover, only GSGM can converge well when the number of computing nodes grows to 30, compared to the state-of-the-art distributed asynchronous algorithms. At the same time, GSGM is robust to different degrees of non-IID data.

cs.DC↗

arXiv2021

Radio frequency spectrum analyzer with a 5 THz bandwidth based on nonlinear optics in a CMOS compatible high-index doped silica waveguide

We report an all-optical radio-frequency (RF) spectrum analyzer with a bandwidth greater than 5 terahertz (THz), based on a 50-cm long spiral waveguide in a CMOS-compatible high-index doped silica platform. By carefully mapping out the dispersion profile of the waveguides for different thicknesses, we identify the optimal design to achieve near zero dispersion in the C-band. To demonstrate the capability of the RF spectrum analyzer, we measure the optical output of a femtosecond fiber laser with an ultrafast optical RF spectrum in the terahertz regime.

physics.optics↗

arXiv2008

Isolated sub-100-attosecond pulse generation via controlling electron dynamics

A new method to coherently control the electron dynamics is proposed using a few-cycle laser pulse in combination with a controlling field. It is shown that this method not only broadens the attosecond pulse bandwidth, but also reduces the chirp, then an isolated 80-as pulse is straightforwardly obtained and even shorter pulse is achievable by increasing the intensity of the controlling field. Such ultrashort pulses allow one to investigate ultrafast electronic processes which have never be achieved before. In addition, the few-cycle synthesized pulse is expected to manipulate a wide range of laser-atom interactions.

physics.atom-ph↗

arXiv2008

Carrier-envelope phase measurement from half-cycle high harmonics

We present a method to distinguish the high harmonics generated in individual half-cycle of the driving laser pulse by mixing a weak ultraviolet pulse, enabling one to observe the cutoff of each half-cycle harmonic. We show that the detail information of the driving laser pulse, including the laser intensity, pulse duration and carrier-envelope phase, can be {\it in situ} retrieved from the harmonic spectrogram. In addition, our results show that this method also distinguishes the half-cycle high harmonics for a pulse longer than 10 fs, suggesting a potential to extend the CEP measurement to the multi-cycle regime.

physics.atom-ph↗