arXiv subjects

Tao Yang

Publications and source records attributed to Tao Yang.

At least 181 records · Page 10Linked to original sources

arXiv2024

CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices

Recently, there has been a demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, personal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs. However, our experiments show that directly combining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoRAs to LLMs, we propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Aware LoRA (CA-LoRA) framework. We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.

cs.CL↗

arXiv2024

Computational Identification and Stuart-Landau Modeling of Collective Dynamical Behaviors of Octuple Laminar Diffusion Flame Oscillators

Annular chambers, consisting of multiple flame nozzles, are frequently used in many industrial processes, for example, rocket engines and gas turbines. In the study, we proposed a novel approach to the problem of annular combustion with emphasis on the collective dynamical behaviors that its individuals do not have. A series of circular arrays of octuple flickering laminar buoyant diffusion flames were investigated computationally and theoretically. Five distinct dynamical modes, such as the merged, in-phase mode, rotation, flickering death, partially flickering death, and anti-phase modes, were computationally identified and interpreted from the perspective of vortex dynamics. A unified regime diagram was obtained in terms of the normalized flame frequency f/f_0 and the combined parameter (α-1)Gr^1/2, where α=l/D is the ratio of the flame separation distance l to the flame nozzle diameter D and Gr is the Grashof number. The bifurcation transition from the in-phase mode and the anti-phase mode to the totally or partially flickering death occurs at (α-1)Gr^1/2=655+-55. In addition, a Stuart-Landau model with a time-delay coupling was utilized to reproduce the general features and collective modes of the octuple oscillators flame systems.

physics.flu-dyn↗

arXiv2024

Acoustic Bound States in the Continuum in Coupled Helmholtz Resonators

Resonant states underlie a variety of metastructures that exhibit remarkable capabilities for effective control of acoustic waves at subwavelength scales. The development of metamaterials relies on the rigorous mode engineering providing the implementation of the desired properties. At the same time, the application of metamaterials is still limited as their building blocks are frequently characterized by complicated geometry and can't be tuned easily. In this work, we consider a simple system of coupled Helmholtz resonators and study their properties associated with the tuning of coupling strength and symmetry breaking. We numerically and experimentally demonstrate the excitation of quasi-bound state in the continuum in the resonators placed in a free space and in a rectangular cavity. It is also shown that tuning the intrinsic losses via introducing porous inserts can lead to spectral splitting or merging of quasi-\textit{bound states in the continuum} and occurrence of \textit{exceptional points}. The obtained results will open new opportunities for the development of simple and easy-tunable metastructures based on Helmholtz resonances.

physics.app-ph↗

arXiv2024

Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization

Diffusion models have demonstrated impressive performance in various image generation, editing, enhancement and translation tasks. In particular, the pre-trained text-to-image stable diffusion models provide a potential solution to the challenging realistic image super-resolution (Real-ISR) and image stylization problems with their strong generative priors. However, the existing methods along this line often fail to keep faithful pixel-wise image structures. If extra skip connections between the encoder and the decoder of a VAE are used to reproduce details, additional training in image space will be required, limiting the application to tasks in latent space such as image stylization. In this work, we propose a pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR and personalized image stylization. Specifically, a pixel-aware cross attention module is introduced to enable diffusion models perceiving image local structures in pixel-wise level, while a degradation removal module is used to extract degradation insensitive features to guide the diffusion process together with image high level information. An adjustable noise schedule is introduced to further improve the image restoration results. By simply replacing the base diffusion model with a stylized one, PASD can generate diverse stylized images without collecting pairwise training data, and by shifting the base model with an aesthetic one, PASD can bring old photos back to life. Extensive experiments in a variety of image enhancement and stylization tasks demonstrate the effectiveness of our proposed PASD approach. Our source codes are available at \url{https://github.com/yangxy/PASD/}.

cs.CV↗

arXiv2024

Susceptibility indicator for chiral topological orders emergent from correlated fermions

Chiral topological orders formed in correlated fermion systems have been widely explored. However, the mechanism on how they emerge from interacting fermions is still unclear. Here, we propose a susceptibility condition. Under this condition, we show that chiral topological orders can spontaneously take place in correlated fermion systems. The condition leads to a low-energy effective theory of bosons with strong frustration, mimicking the flat band systems. The frustration then melts the long-range orders and results in topological orders with time-reversal symmetry breaking. We apply the theory to strongly-correlated semiconductors doped to the metallic phase. A novel excitonic topological order with semionic excitations and chiral excitonic edge state is revealed. We also discuss the application to frustrated magnets. The theory predicts a chiral spin liquid state, which is numerically confirmed by our tensor network calculations. These results demonstrate an unprecedented indicator for chiral topological orders, which bridges the existing gap between interacting fermions and correlated topological matter.

cond-mat.str-el↗

arXiv2024

Distributed Event-Triggered Bandit Convex Optimization with Time-Varying Constraints

This paper considers the distributed bandit convex optimization problem with time-varying inequality constraints over a network of agents, where the goal is to minimize network regret and cumulative constraint violation. Existing distributed online algorithms require that each agent broadcasts its decision to its neighbors at each iteration. To better utilize the limited communication resources, we propose a distributed event-triggered online primal--dual algorithm with two-point bandit feedback. Under several classes of appropriately chosen decreasing parameter sequences and non-increasing event-triggered threshold sequences, we establish dynamic network regret and network cumulative constraint violation bounds. These bounds are comparable to the results achieved by distributed event-triggered online algorithms with full-information feedback. Finally, a numerical example is provided to verify the theoretical results.

math.OC↗

arXiv2024

Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

Disentangled representation learning strives to extract the intrinsic factors within observed data. Factorizing these representations in an unsupervised manner is notably challenging and usually requires tailored loss functions or specific structural designs. In this paper, we introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias to facilitate the learning of disentangled representations. We propose to encode an image to a set of concept tokens and treat them as the condition of the latent diffusion for image reconstruction, where cross-attention over the concept tokens is used to bridge the interaction between the encoder and diffusion. Without any additional regularization, this framework achieves superior disentanglement performance on the benchmark datasets, surpassing all previous methods with intricate designs. We have conducted comprehensive ablation studies and visualization analysis, shedding light on the functioning of this model. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs. We anticipate that our findings will inspire more investigation on exploring diffusion for disentangled representation learning towards more sophisticated data analysis and understanding.

cs.CV↗

arXiv2024

Weighted KL-Divergence for Document Ranking Model Refinement

Transformer-based retrieval and reranking models for text document search are often refined through knowledge distillation together with contrastive learning. A tight distribution matching between the teacher and student models can be hard as over-calibration may degrade training effectiveness when a teacher does not perform well. This paper contrastively reweights KL divergence terms to prioritize the alignment between a student and a teacher model for proper separation of positive and negative documents. This paper analyzes and evaluates the proposed loss function on the MS MARCO and BEIR datasets to demonstrate its effectiveness in improving the relevance of tested student models.

cs.IR↗

arXiv2024

Breaking through the learning plateaus of in-context learning in Transformer

In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually seperate a component within the model's internal representation that is exclusively affected by the model's weights. We call this the "weights component", and the remainder is identified as the "context component". By conducting meticulous and controlled experiments on synthetic tasks, we note that the persistence of learning plateaus correlates with compromised functionality of the weights component. Recognizing the impaired performance of the weights component as a fundamental behavior drives learning plateaus, we have developed three strategies to expedite the learning of Transformers. The effectiveness of these strategies is further confirmed in natural language processing tasks. In conclusion, our research demonstrates the feasibility of cultivating a powerful in-context learning ability within AI systems in an eco-friendly manner.

cs.LG↗

arXiv2024

SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution

Owe to the powerful generative priors, the pre-trained text-to-image (T2I) diffusion models have become increasingly popular in solving the real-world image super-resolution problem. However, as a consequence of the heavy quality degradation of input low-resolution (LR) images, the destruction of local structures can lead to ambiguous image semantics. As a result, the content of reproduced high-resolution image may have semantic errors, deteriorating the super-resolution performance. To address this issue, we present a semantics-aware approach to better preserve the semantic fidelity of generative real-world image super-resolution. First, we train a degradation-aware prompt extractor, which can generate accurate soft and hard semantic prompts even under strong degradation. The hard semantic prompts refer to the image tags, aiming to enhance the local perception ability of the T2I model, while the soft semantic prompts compensate for the hard ones to provide additional representation information. These semantic prompts encourage the T2I model to generate detailed and semantically accurate results. Furthermore, during the inference process, we integrate the LR images into the initial sampling noise to mitigate the diffusion model's tendency to generate excessive random details. The experiments show that our method can reproduce more realistic image details and hold better the semantics. The source code of our method can be found at https://github.com/cswry/SeeSR.

cs.CV↗

arXiv2024

Eccentricity enables the earliest warning and localization of gravitational waves with ground-based detectors

The early and precise localization of gravitational waves (GWs) is pivotal in detecting their electromagnetic (EM) counterparts, especially for binary neutron stars (BNS) and neutron star-black hole binaries (NSBH). In this letter, we pioneer the exploration of utilizing the higher harmonic modes induced by the eccentricity of compact binaries to localize GWs with ground-based detectors even before the quadrupole baseline $\ell=2$ mode enters the detector band. Our theoretical analysis marks a first in proposing a strategy for gaining the earliest possible warning and maximizing preparation time for observing pre- and/or post-merger EM counterparts. We simulate three typical binaries from GWTC-3 with eccentricities ranging from 0.05 to 0.4. Our results reveal that the third-generation (3G) detectors (low frequency cut-off $f_0=5$ Hz) can accumulate sufficient signal-to-noise ratios through higher modes before the onset of the baseline $\ell=2$ mode entry into the band. Notably, relying solely on the higher modes, the 3G detector network ET+2CE achieves an average localization on the order of $1-10^2~\rm deg^2$ around 1-1.8 hours before the merger of a GW170817-like BNS, and $10-10^3~\rm deg^2$ approximately 18-30 minutes prior to the merger of a GW200115-like NSBH. A $100~\rm deg^2$ localization is attainable even 2-4 hours prior to a BNS merger. Moreover, in the near face-on orientations which are generally more favorable for EM counterpart detection, the localization can be further improved.

gr-qc↗

arXiv2024

Distributed online constrained convex optimization with event-triggered communication

This paper focuses on the distributed online convex optimization problem with time-varying inequality constraints over a network of agents, where each agent collaborates with its neighboring agents to minimize the cumulative network-wide loss over time. To reduce communication overhead between the agents, we propose a distributed event-triggered online primal-dual algorithm over a time-varying directed graph. With several classes of appropriately chose decreasing parameter sequences and non-increasing event-triggered threshold sequences, we establish dynamic network regret and network cumulative constraint violation bounds. Finally, a numerical simulation example is provided to verify the theoretical results.

math.OC↗

arXiv2024

Approximate Cluster-Based Sparse Document Retrieval with Segmented Maximum Term Weights

This paper revisits cluster-based retrieval that partitions the inverted index into multiple groups and skips the index partially at cluster and document levels during online inference using a learned sparse representation. It proposes an approximate search scheme with two parameters to control the rank-safeness competitiveness of pruning with segmented maximum term weights within each cluster. Cluster-level maximum weight segmentation allows an improvement in the rank score bound estimation and threshold-based pruning to be approximately adaptive to bound estimation tightness, resulting in better relevance and efficiency. The experiments with MS MARCO passage ranking and BEIR datasets demonstrate the usefulness of the proposed scheme with a comparison to the baselines. This paper presents the design of this approximate retrieval scheme with rank-safeness analysis, compares clustering and segmentation options, and reports evaluation results.

cs.IR↗

arXiv2024

Parallel Proportional Fusion of Spiking Quantum Neural Network for Optimizing Image Classification

The recent emergence of the hybrid quantum-classical neural network (HQCNN) architecture has garnered considerable attention due to the potential advantages associated with integrating quantum principles to enhance various facets of machine learning algorithms and computations. However, the current investigated serial structure of HQCNN, wherein information sequentially passes from one network to another, often imposes limitations on the trainability and expressivity of the network. In this study, we introduce a novel architecture termed Parallel Proportional Fusion of Quantum and Spiking Neural Networks (PPF-QSNN). The dataset information is simultaneously fed into both the spiking neural network and the variational quantum circuits, with the outputs amalgamated in proportion to their individual contributions. We systematically assess the impact of diverse PPF-QSNN parameters on network performance for image classification, aiming to identify the optimal configuration. Numerical results on the MNIST dataset unequivocally illustrate that our proposed PPF-QSNN outperforms both the existing spiking neural network and the serial quantum neural network across metrics such as accuracy, loss, and robustness. This study introduces a novel and effective amalgamation approach for HQCNN, thereby laying the groundwork for the advancement and application of quantum advantage in artificial intelligent computations.

quant-ph↗

arXiv2024

Enhance Reasoning for Large Language Models in the Game Werewolf

This paper presents an innovative framework that integrates Large Language Models (LLMs) with an external Thinker module to enhance the reasoning capabilities of LLM-based agents. Unlike augmenting LLMs with prompt engineering, Thinker directly harnesses knowledge from databases and employs various optimization techniques. The framework forms a reasoning hierarchy where LLMs handle intuitive System-1 tasks such as natural language processing, while the Thinker focuses on cognitive System-2 tasks that require complex logical analysis and domain-specific knowledge. Our framework is presented using a 9-player Werewolf game that demands dual-system reasoning. We introduce a communication protocol between LLMs and the Thinker, and train the Thinker using data from 18800 human sessions and reinforcement learning. Experiments demonstrate the framework's effectiveness in deductive reasoning, speech generation, and online game evaluation. Additionally, we fine-tune a 6B LLM to surpass GPT4 when integrated with the Thinker. This paper also contributes the largest dataset for social deduction games to date.

cs.AI↗

arXiv2024

Are LLMs Effective Backbones for Fine-tuning? An Experimental Investigation of Supervised LLMs on Chinese Short Text Matching

The recent success of Large Language Models (LLMs) has garnered significant attention in both academia and industry. Prior research on LLMs has primarily focused on enhancing or leveraging their generalization capabilities in zero- and few-shot settings. However, there has been limited investigation into effectively fine-tuning LLMs for a specific natural language understanding task in supervised settings. In this study, we conduct an experimental analysis by fine-tuning LLMs for the task of Chinese short text matching. We explore various factors that influence performance when fine-tuning LLMs, including task modeling methods, prompt formats, and output formats.

cs.CL↗

arXiv2024

Superfluid Oscillator Circuit with Quantum Current Regulator

We examine the properties of atomic current in a superfluid oscillating circuit consisting of a mesoscopic channel that connects two reservoirs of a Bose-Einstein condensate. We investigate the presence of a critical current in the channel and examine how the amplitude of the oscillations in the number imbalance between the two reservoirs varies with system parameters. In addition to highlighting that the dissipative resistance stems from the formation of vortex pairs, we also illustrate the role of these vortex pairs as a quantum current regulator. The dissipation strength is discrete based on the number imbalance, which corresponds to the emergence of vortex pairs in the system. Our findings indicate that the circuit demonstrates characteristics of both voltage-limiting and current-limiting mechanisms. To model the damping behavior of the atomic superfluid circuit, we develop an equivalent LC oscillator circuit with a quantum current regulator.

quant-ph↗

arXiv2024

Robust and Scalable Model Editing for Large Language Models

Large language models (LLMs) can make predictions using parametric knowledge--knowledge encoded in the model weights--or contextual knowledge--knowledge presented in the context. In many scenarios, a desirable behavior is that LLMs give precedence to contextual knowledge when it conflicts with the parametric knowledge, and fall back to using their parametric knowledge when the context is irrelevant. This enables updating and correcting the model's knowledge by in-context editing instead of retraining. Previous works have shown that LLMs are inclined to ignore contextual knowledge and fail to reliably fall back to parametric knowledge when presented with irrelevant context. In this work, we discover that, with proper prompting methods, instruction-finetuned LLMs can be highly controllable by contextual knowledge and robust to irrelevant context. Utilizing this feature, we propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. To better evaluate the robustness of model editors, we collect a new dataset, that contains irrelevant questions that are more challenging than the ones in existing datasets. Empirical results show that our method outperforms current state-of-the-art methods by a large margin. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs (and vice versa). The source code can be found at https://github.com/thunlp/EREN.

cs.CL↗