arXiv subjects

Fei Yang

Publications and source records attributed to Fei Yang.

At least 73 records · Page 4Linked to original sources

arXiv2024

Parabolic and near-parabolic renormalizations for local degree three

The invariant class under parabolic and near-parabolic renormalizations constructed by Inou and Shishikura has been proved to be extremely useful in recent years. It leads to several important progresses on the dynamics of certain holomorphic maps with critical points of local degree two. In this paper, we construct a new class consisting of holomorphic maps with critical points of local degree three which is invariant under parabolic and near-parabolic renormalizations. As potential applications, some results of cubic unicritical polynomials can be obtained similarly as the quadratic case. For example, the existence of cubic unicritical Julia sets with positive area, the characterizations of the topology and geometry of cubic irrationally indifferent attractors etc.

math.DS↗

arXiv2024

LocInv: Localization-aware Inversion for Text-Guided Image Editing

Large-scale Text-to-Image (T2I) diffusion models demonstrate significant generation capabilities based on textual prompts. Based on the T2I diffusion models, text-guided image editing research aims to empower users to manipulate generated images by altering the text prompts. However, existing image editing techniques are prone to editing over unintentional regions that are beyond the intended target area, primarily due to inaccuracies in cross-attention maps. To address this problem, we propose Localization-aware Inversion (LocInv), which exploits segmentation maps or bounding boxes as extra localization priors to refine the cross-attention maps in the denoising phases of the diffusion process. Through the dynamic updating of tokens corresponding to noun words in the textual input, we are compelling the cross-attention maps to closely align with the correct noun and adjective words in the text prompt. Based on this technique, we achieve fine-grained image editing over particular objects while preventing undesired changes to other regions. Our method LocInv, based on the publicly available Stable Diffusion, is extensively evaluated on a subset of the COCO dataset, and consistently obtains superior results both quantitatively and qualitatively.The code will be released at https://github.com/wangkai930418/DPL

cs.CV↗

arXiv2024

Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment

Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks. However, training these models can incur significant expenses, often requiring tens of thousands of GPUs for months of continuous operation. Typically, this training is carried out in specialized GPU clusters equipped with homogeneous high-speed Remote Direct Memory Access (RDMA) network interface cards (NICs). The acquisition and maintenance of such dedicated clusters is challenging. Current LLM training frameworks, like Megatron-LM and Megatron-DeepSpeed, focus primarily on optimizing training within homogeneous cluster settings. In this paper, we introduce Holmes, a training framework for LLMs that employs thoughtfully crafted data and model parallelism strategies over the heterogeneous NIC environment. Our primary technical contribution lies in a novel scheduling method that intelligently allocates distinct computational tasklets in LLM training to specific groups of GPU devices based on the characteristics of their connected NICs. Furthermore, our proposed framework, utilizing pipeline parallel techniques, demonstrates scalability to multiple GPU clusters, even in scenarios without high-speed interconnects between nodes in distinct clusters. We conducted comprehensive experiments that involved various scenarios in the heterogeneous NIC environment. In most cases, our framework achieves performance levels close to those achievable with homogeneous RDMA-capable networks (InfiniBand or RoCE), significantly exceeding training efficiency within the pure Ethernet environment. Additionally, we verified that our framework outperforms other mainstream LLM frameworks under heterogeneous NIC environment in terms of training efficiency and can be seamlessly integrated with them.

cs.CL↗

arXiv2024

Machine Unlearning for Document Classification

Document understanding models have recently demonstrated remarkable performance by leveraging extensive collections of user documents. However, since documents often contain large amounts of personal data, their usage can pose a threat to user privacy and weaken the bonds of trust between humans and AI services. In response to these concerns, legislation advocating ``the right to be forgotten" has recently been proposed, allowing users to request the removal of private information from computer systems and neural network models. A novel approach, known as machine unlearning, has emerged to make AI models forget about a particular class of data. In our research, we explore machine unlearning for document classification problems, representing, to the best of our knowledge, the first investigation into this area. Specifically, we consider a realistic scenario where a remote server houses a well-trained model and possesses only a small portion of training data. This setup is designed for efficient forgetting manipulation. This work represents a pioneering step towards the development of machine unlearning methods aimed at addressing privacy concerns in document analysis applications. Our code is publicly available at \url{https://github.com/leitro/MachineUnlearning-DocClassification}.

cs.CV↗

arXiv2024

The high type quadratic Siegel disks are Jordan domains

Let $α$ be an irrational number of sufficiently high type and suppose $P_α(z)=e^{2πiα}z+z^2$ has a Siegel disk $Δ_α$ centered at the origin. We prove that the boundary of $Δ_α$ is a Jordan curve, and that it contains the critical point $-e^{2πiα}/2$ if and only if $α$ is a Herman number.

math.DS↗

arXiv2024

Parallel Proportional Fusion of Spiking Quantum Neural Network for Optimizing Image Classification

The recent emergence of the hybrid quantum-classical neural network (HQCNN) architecture has garnered considerable attention due to the potential advantages associated with integrating quantum principles to enhance various facets of machine learning algorithms and computations. However, the current investigated serial structure of HQCNN, wherein information sequentially passes from one network to another, often imposes limitations on the trainability and expressivity of the network. In this study, we introduce a novel architecture termed Parallel Proportional Fusion of Quantum and Spiking Neural Networks (PPF-QSNN). The dataset information is simultaneously fed into both the spiking neural network and the variational quantum circuits, with the outputs amalgamated in proportion to their individual contributions. We systematically assess the impact of diverse PPF-QSNN parameters on network performance for image classification, aiming to identify the optimal configuration. Numerical results on the MNIST dataset unequivocally illustrate that our proposed PPF-QSNN outperforms both the existing spiking neural network and the serial quantum neural network across metrics such as accuracy, loss, and robustness. This study introduces a novel and effective amalgamation approach for HQCNN, thereby laying the groundwork for the advancement and application of quantum advantage in artificial intelligent computations.

quant-ph↗

arXiv2024

FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization

Large language models (LLMs) have demonstrated state-of-the-art performance across various tasks. However, the latency of inference and the large GPU memory consumption of LLMs restrict their deployment performance. Recently, there have been some efficient attempts to quantize LLMs, yet inference with large batch size or long sequence still has the issue of being compute-bound. Fine-grained quantization methods have showcased their proficiency in achieving low-bit quantization for LLMs, while requiring FP16 data type for linear layer computations, which is time-consuming when dealing with large batch size or long sequence. In this paper, we introduce a method called FlattenQuant, which significantly reduces the maximum value of the tensor by flattening the large channels in the tensor, to achieve low bit per-tensor quantization with minimal accuracy loss. Our experiments show that FlattenQuant can directly use 4 bits to achieve 48.29% of the linear layer calculation in LLMs, with the remaining layers using 8 bits. The 4-bit matrix multiplication introduced in the FlattenQuant method can effectively address the compute-bound caused by large matrix calculation. Our work achieves up to 2$\times$ speedup and 2.3$\times$ memory reduction for LLMs with negligible loss in accuracy.

cs.LG↗

arXiv2023

Hybrid GRU-CNN Bilinear Parameters Initialization for Quantum Approximate Optimization Algorithm

The Quantum Approximate Optimization Algorithm (QAOA), a pivotal paradigm in the realm of variational quantum algorithms (VQAs), offers promising computational advantages for tackling combinatorial optimization problems. Well-defined initial circuit parameters, responsible for preparing a parameterized quantum state encoding the solution, play a key role in optimizing QAOA. However, classical optimization techniques encounter challenges in discerning optimal parameters that align with the optimal solution. In this work, we propose a hybrid optimization approach that integrates Gated Recurrent Units (GRU), Convolutional Neural Networks (CNN), and a bilinear strategy as an innovative alternative to conventional optimizers for predicting optimal parameters of QAOA circuits. GRU serves to stochastically initialize favorable parameters for depth-1 circuits, while CNN predicts initial parameters for depth-2 circuits based on the optimized parameters of depth-1 circuits. To assess the efficacy of our approach, we conducted a comparative analysis with traditional initialization methods using QAOA on Erdős-Rényi graph instances, revealing superior optimal approximation ratios. We employ the bilinear strategy to initialize QAOA circuit parameters at greater depths, with reference parameters obtained from GRU-CNN optimization. This approach allows us to forecast parameters for a depth-12 QAOA circuit, yielding a remarkable approximation ratio of 0.998 across 10 qubits, which surpasses that of the random initialization strategy and the PPN2 method at a depth of 10. The proposed hybrid GRU-CNN bilinear optimization method significantly improves the effectiveness and accuracy of parameters initialization, offering a promising iterative framework for QAOA that elevates its performance.

quant-ph↗

arXiv2023

Topological phases in the periodically kicked Chern insulators

Novel topological properties that arose in the periodically driven system are unique, in which there are two kinds of quasienergy gaps, the zero quasienergy gap and the $π$ quasienergy gap. The corresponding edge modes would traverse either the zero quasienergy gap or the $π$ quasienergy gap, or traverse both two quasienergy gaps. And the characterization of these two kinds of edge modes might not be the same. However, in this paper, we find that both the zero edge modes and the $π$ edge modes in the Floquet Chern insulators can be characterized by the same topological invariant, where the corresponding Dirac mass term is periodically kicked. Particularly, we take the periodically kicked Qi-Wu-Zhang model as an illustrative example. In this model, the topology is characterized by the Floquet Chern number $C_F$, and there are six different topological phases in total, denoted as $C_F=\{-1_0,-2,-1_π,1_π,2,1_0\}$. Furthermore, we find that the Floquet operator associated with the periodically kicked Qi-Wu-Zhang model reduces to a Dirac Hamiltonian in the low-energy limit. Then, the phase diagram is uncovered by examining the topology of this Dirac Hamiltonian. Additionally, we explore the orders of topological phase transitions in the context of Floquet stationary states by analyzing the von Neumann entropy of these states. Our work provides further insights into the topological phases in periodically driven systems.

cond-mat.mes-hall↗

arXiv2023

Rational maps with smooth degenerate Herman rings

We prove the existence of rational maps having smooth degenerate Herman rings. This answers a question of Eremenko affirmatively. The proof is based on the construction of smooth Siegel disks by Avila, Buff and Chéritat as well as the classical Siegel-to-Herman quasiconformal surgery. A crucial ingredient in the proof is the surgery's continuity, which relies on the control of the loss of the area of quadratic filled-in Julia sets by Buff and Chéritat. As a by-product, we prove the existence of rational maps having a nowhere dense Julia set of positive area for which these maps have no irrationally indifferent periodic points, no Herman rings, and are not renormalizable.

math.DS↗

arXiv2023

Exploring Post-Training Quantization of Protein Language Models

Recent advancements in unsupervised protein language models (ProteinLMs), like ESM-1b and ESM-2, have shown promise in different protein prediction tasks. However, these models face challenges due to their high computational demands, significant memory needs, and latency, restricting their usage on devices with limited resources. To tackle this, we explore post-training quantization (PTQ) for ProteinLMs, focusing on ESMFold, a simplified version of AlphaFold based on ESM-2 ProteinLM. Our study is the first attempt to quantize all weights and activations of ProteinLMs. We observed that the typical uniform quantization method performs poorly on ESMFold, causing a significant drop in TM-Score when using 8-bit quantization. We conducted extensive quantization experiments, uncovering unique challenges associated with ESMFold, particularly highly asymmetric activation ranges before Layer Normalization, making representation difficult using low-bit fixed-point formats. To address these challenges, we propose a new PTQ method for ProteinLMs, utilizing piecewise linear quantization for asymmetric activation values to ensure accurate approximation. We demonstrated the effectiveness of our method in protein structure prediction tasks, demonstrating that ESMFold can be accurately quantized to low-bit widths without compromising accuracy. Additionally, we applied our method to the contact prediction task, showcasing its versatility. In summary, our study introduces an innovative PTQ method for ProteinLMs, addressing specific quantization challenges and potentially leading to the development of more efficient ProteinLMs with significant implications for various protein-related applications.

cs.LG↗

arXiv2023

Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing

Large-scale text-to-image generative models have been a ground-breaking development in generative AI, with diffusion models showing their astounding ability to synthesize convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are susceptible to unintended modifications of regions outside the targeted area, such as on the background or on distractor objects which have some semantic or visual relationship with the targeted object. According to our experimental findings, inaccurate cross-attention maps are at the root of this problem. Based on this observation, we propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. By updating the dynamic tokens for nouns in the textual input with the proposed leakage repairment losses, we achieve fine-grained image editing over particular objects while preventing undesired changes to other image regions. Our method DPL, based on the publicly available Stable Diffusion, is extensively evaluated on a wide range of images, and consistently obtains superior results both quantitatively (CLIP score, Structure-Dist) and qualitatively (on user-evaluation). We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.

cs.CV↗

arXiv2023

ScrollNet: Dynamic Weight Importance for Continual Learning

The principle underlying most existing continual learning (CL) methods is to prioritize stability by penalizing changes in parameters crucial to old tasks, while allowing for plasticity in other parameters. The importance of weights for each task can be determined either explicitly through learning a task-specific mask during training (e.g., parameter isolation-based approaches) or implicitly by introducing a regularization term (e.g., regularization-based approaches). However, all these methods assume that the importance of weights for each task is unknown prior to data exposure. In this paper, we propose ScrollNet as a scrolling neural network for continual learning. ScrollNet can be seen as a dynamic network that assigns the ranking of weight importance for each task before data exposure, thus achieving a more favorable stability-plasticity tradeoff during sequential task learning by reassigning this ranking for different tasks. Additionally, we demonstrate that ScrollNet can be combined with various CL methods, including regularization-based and replay-based approaches. Experimental results on CIFAR100 and TinyImagenet datasets show the effectiveness of our proposed method. We release our code at https://github.com/FireFYF/ScrollNet.git.

cs.CV↗

arXiv2023

Non-Hermitian Chiral Edge Modes With Complex Fermi Velocity

Recently, much attention has been paid to uncovering the influence of dissipation on a quantum system, particularly on how the non-Hermitian (NH) terms modify the band topology of topological materials and reshape the profile of the wavefunctions of a system (or the NH skin effect). In this paper, a specific NH skin effect that induced by local dissipation is studied for chiral edge modes, in which the NH term corresponds to the local imaginary Fermi velocity of the chiral edge modes. By solving the NH Schrödinger equation of the non-Hermitian chiral edge modes (nhCEs) with complex Fermi velocity, we uncovered the remarkable complex spectra and the wavefunctions of the nhCEs. We find that the complex spectra of these modes is a straight line in the topological materials, and its chirality can separates the modes with positive energy from those with negative energy, in which they are localized at different positions. We also studied the nhCEs at the boundary of 2-dimensional (2D) topological materials, the 2D $p$-wave superconductor and the Qi-Wu-Zhang model, in which the general law of nhCEs was verified. We expect that our findings will pave the way for researching the transport properties of the chiral edge modes in the non-equilibrium context.

cond-mat.str-el↗

arXiv2023

Exemplar-free Continual Learning of Vision Transformers via Gated Class-Attention and Cascaded Feature Drift Compensation

We propose a new method for exemplar-free class incremental training of ViTs. The main challenge of exemplar-free continual learning is maintaining plasticity of the learner without causing catastrophic forgetting of previously learned tasks. This is often achieved via exemplar replay which can help recalibrate previous task classifiers to the feature drift which occurs when learning new tasks. Exemplar replay, however, comes at the cost of retaining samples from previous tasks which for many applications may not be possible. To address the problem of continual ViT training, we first propose gated class-attention to minimize the drift in the final ViT transformer block. This mask-based gating is applied to class-attention mechanism of the last transformer block and strongly regulates the weights crucial for previous tasks. Importantly, gated class-attention does not require the task-ID during inference, which distinguishes it from other parameter isolation methods. Secondly, we propose a new method of feature drift compensation that accommodates feature drift in the backbone when learning new tasks. The combination of gated class-attention and cascaded feature drift compensation allows for plasticity towards new tasks while limiting forgetting of previous ones. Extensive experiments performed on CIFAR-100, Tiny-ImageNet and ImageNet100 demonstrate that our exemplar-free method obtains competitive results when compared to rehearsal based ViT methods.

cs.CV↗

arXiv2023

Recognition of Mental Adjectives in An Efficient and Automatic Style

In recent years, commonsense reasoning has received more and more attention from academic community. We propose a new lexical inference task, Mental and Physical Classification (MPC), to handle commonsense reasoning in a reasoning graph. Mental words relate to mental activities, which fall into six categories: Emotion, Need, Perceiving, Reasoning, Planning and Personality. Physical words describe physical attributes of an object, like color, hardness, speed and malleability. A BERT model is fine-tuned for this task and active learning algorithm is adopted in the training framework to reduce the required annotation resources. The model using ENTROPY strategy achieves satisfactory accuracy and requires only about 300 labeled words. We also compare our result with SentiWordNet to check the difference between MPC and subjectivity classification task in sentiment analysis.

cs.CL↗

arXiv2023

Formally Verifying a Real World Smart Contract

Nowadays, smart contracts have become increasingly popular and, as with software development in general, testing is the standard method for verifying their correctness. However, smart contracts require a higher level of certainty regarding correctness because they are diffcult to modify once deployed and errors can result in significant financial losses. Therefore, formal verification is essential. In this article, we present our search for a tool capable of formally verifying a real-world smart contract written in a recent version of Solidity.

cs.SE↗

arXiv2023

Mating Siegel and parabolic quadratic polynomials

Let $f_θ(z)=e^{2πiθ}z+z^2$ be the quadratic polynomial having an indifferent fixed point at the origin. For any bounded type irrational number $θ\in\mathbb{R}\setminus\mathbb{Q}$ and any rational number $ν\in\mathbb{Q}$, we prove that $f_θ$ and $f_ν$ are conformally mateable, and that the mating is unique up to conjugacy by a Möbius map. This gives an affirmative (partial) answer to a question raised by Milnor in 2004. A crucial ingredient in the proof relies on an expansive property when iterating certain rational maps near Siegel disk boundaries. Combining this with the expanding property in repelling petals of parabolic points, we also prove that the Julia sets of a class of Siegel rational maps with parabolic points are locally connected.

math.DS↗