arXiv subjects

Lu Chen

Publications and source records attributed to Lu Chen.

At least 217 records · Page 12Linked to original sources

arXiv2024

A BiRGAT Model for Multi-intent Spoken Language Understanding with Hierarchical Semantic Frames

Previous work on spoken language understanding (SLU) mainly focuses on single-intent settings, where each input utterance merely contains one user intent. This configuration significantly limits the surface form of user utterances and the capacity of output semantics. In this work, we first propose a Multi-Intent dataset which is collected from a realistic in-Vehicle dialogue System, called MIVS. The target semantic frame is organized in a 3-layer hierarchical structure to tackle the alignment and assignment problems in multi-intent cases. Accordingly, we devise a BiRGAT model to encode the hierarchy of ontology items, the backbone of which is a dual relational graph attention network. Coupled with the 3-way pointer-generator decoder, our method outperforms traditional sequence labeling and classification-based schemes by a large margin.

cs.CL↗

arXiv2024

Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

The growing prevalence of visually rich documents, such as webpages and scanned/digital-born documents (images, PDFs, etc.), has led to increased interest in automatic document understanding and information extraction across academia and industry. Although various document modalities, including image, text, layout, and structure, facilitate human information retrieval, the interconnected nature of these modalities presents challenges for neural networks. In this paper, we introduce WebLM, a multimodal pre-training network designed to address the limitations of solely modeling text and structure modalities of HTML in webpages. Instead of processing document images as unified natural images, WebLM integrates the hierarchical structure of document images to enhance the understanding of markup-language-based documents. Additionally, we propose several pre-training tasks to model the interaction among text, structure, and image modalities effectively. Empirical results demonstrate that the pre-trained WebLM significantly surpasses previous state-of-the-art pre-trained models across several webpage understanding tasks. The pre-trained models and code are available at https://github.com/X-LANCE/weblm.

cs.CL↗

arXiv2024

Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution

Faithfulness, expressiveness, and elegance is the constant pursuit in machine translation. However, traditional metrics like \textit{BLEU} do not strictly align with human preference of translation quality. In this paper, we explore leveraging reinforcement learning with human feedback (\textit{RLHF}) to improve translation quality. It is non-trivial to collect a large high-quality dataset of human comparisons between translations, especially for low-resource languages. To address this issue, we propose a cost-effective preference learning strategy, optimizing reward models by distinguishing between human and machine translations. In this manner, the reward model learns the deficiencies of machine translation compared to human and guides subsequent improvements in machine translation. Experimental results demonstrate that \textit{RLHF} can effectively enhance translation quality and this improvement benefits other translation directions not trained with \textit{RLHF}. Further analysis indicates that the model's language capabilities play a crucial role in preference learning. A reward model with strong language capabilities can more sensitively learn the subtle differences in translation quality and align better with real human translation preferences.

cs.CL↗

arXiv2024

SparDL: Distributed Deep Learning Training with Efficient Sparse Communication

Top-k sparsification has recently been widely used to reduce the communication volume in distributed deep learning. However, due to the Sparse Gradient Accumulation (SGA) dilemma, the performance of top-k sparsification still has limitations. Recently, a few methods have been put forward to handle the SGA dilemma. Regrettably, even the state-of-the-art method suffers from several drawbacks, e.g., it relies on an inefficient communication algorithm and requires extra transmission steps. Motivated by the limitations of existing methods, we propose a novel efficient sparse communication framework, called SparDL. Specifically, SparDL uses the Spar-Reduce-Scatter algorithm, which is based on an efficient Reduce-Scatter model, to handle the SGA dilemma without additional communication operations. Besides, to further reduce the latency cost and improve the efficiency of SparDL, we propose the Spar-All-Gather algorithm. Moreover, we propose the global residual collection algorithm to ensure fast convergence of model training. Finally, extensive experiments are conducted to validate the superiority of SparDL.

cs.LG↗

arXiv2024

A Unified Causal View of Instruction Tuning

Instruction tuning on a mixture of tasks has improved zero-shot capabilities in natural language processing (NLP). Nevertheless, existing methods often learn features that exhibit correlations between instruction-formatted samples and target labels, rather than causal relationships. Termed as ``spurious correlation'' in statistics, such a correlation may change drastically in a new task, making the effect from the learned features to be misleading. To this end, we develop a meta Structural Causal Model (meta-SCM) to integrate different NLP tasks under a single causal structure of the data. Specifically, the meta-SCM introduces multiple latent factors that represent properties of source context, only some of which causally influence the target labels for a specific task. The key idea is to learn task-required causal factors and only use those to make predictions for a given task. Theoretically, we prove the causal factor can be identified without mixing information from others. Guided by the identifiability, we propose a Structural Instruction Tuning (SIT) method to learn the task-required causal representations that can mimic the causal factors for each task. The utility of our approach is verified by improvements of zero-shot ability on a range of unseen datasets and tasks.

cs.CL↗

arXiv2024

GPU-Accelerated Batch-Dynamic Subgraph Matching

Subgraph matching has garnered increasing attention for its diverse real-world applications. Given the dynamic nature of real-world graphs, addressing evolving scenarios without incurring prohibitive overheads has been a focus of research. However, existing approaches for dynamic subgraph matching often proceed serially, retrieving incremental matches for each updated edge individually. This approach falls short when handling batch data updates, leading to a decrease in system throughput. Leveraging the parallel processing power of GPUs, which can execute a massive number of cores simultaneously, has been widely recognized for performance acceleration in various domains. Surprisingly, systematic exploration of subgraph matching in the context of batch-dynamic graphs, particularly on a GPU platform, remains untouched. In this paper, we bridge this gap by introducing an efficient framework, GAMMA (GPU-Accelerated Batch-Dynamic Subgraph Matching). Our approach features a DFS-based warp-centric batch-dynamic subgraph matching algorithm. To ensure load balance in the DFS-based search, we propose warp-level work stealing via shared memory. Additionally, we introduce coalesced search to reduce redundant computations. Comprehensive experiments demonstrate the superior performance of GAMMA. Compared to state-of-the-art algorithms, GAMMA showcases a performance improvement up to hundreds of times.

cs.DC↗

arXiv2024

MouSi: Poly-Visual-Expert Vision-Language Models

Current large vision-language models (VLMs) often encounter challenges such as insufficient capabilities of a single visual component and excessively long visual tokens. These issues can limit the model's effectiveness in accurately interpreting complex visual information and over-lengthy contextual information. Addressing these challenges is crucial for enhancing the performance and applicability of VLMs. This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc. This technique introduces a fusion network to unify the processing of outputs from different visual experts, while bridging the gap between image encoders and pre-trained LLMs. In addition, we explore different positional encoding schemes to alleviate the waste of positional encoding caused by lengthy image feature sequences, effectively addressing the issue of position overflow and length limitations. For instance, in our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1. Experimental results demonstrate that VLMs with multiple experts exhibit consistently superior performance over isolated visual encoders and mark a significant performance boost as more experts are integrated. We have open-sourced the training code used in this report. All of these resources can be found on our project website.

cs.CV↗

arXiv2024

Text Classification Based on Knowledge Graphs and Improved Attention Mechanism

To resolve the semantic ambiguity in texts, we propose a model, which innovatively combines a knowledge graph with an improved attention mechanism. An existing knowledge base is utilized to enrich the text with relevant contextual concepts. The model operates at both character and word levels to deepen its understanding by integrating the concepts. We first adopt information gain to select import words. Then an encoder-decoder framework is used to encode the text along with the related concepts. The local attention mechanism adjusts the weight of each concept, reducing the influence of irrelevant or noisy concepts during classification. We improve the calculation formula for attention scores in the local self-attention mechanism, ensuring that words with different frequencies of occurrence in the text receive higher attention scores. Finally, the model employs a Bi-directional Gated Recurrent Unit (Bi-GRU), which is effective in feature extraction from texts for improved classification accuracy. Its performance is demonstrated on datasets such as AGNews, Ohsumed, and TagMyNews, achieving accuracy of 75.1%, 58.7%, and 68.5% respectively, showing its effectiveness in classifying tasks.

cs.CL↗

arXiv2024

FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis

Text-to-SQL, which provides zero-code interface for operating relational databases, has gained much attention in financial analysis; because, financial professionals may not well-skilled in SQL programming. However, until now, there is no practical Text-to-SQL benchmark dataset for financial analysis, and existing Text-to-SQL methods have not considered the unique characteristics of databases in financial applications, such as commonly existing wide tables. To address these issues, we collect a practical Text-to-SQL benchmark dataset and propose a model-agnostic Large Language Model (LLMs)-based Text-to-SQL framework for financial analysis. The benchmark dataset, BULL, is collected from the practical financial analysis business of Hundsun Technologies Inc., including databases for fund, stock, and macro economy. Besides, the proposed LLMs-based Text-to-SQL framework, FinSQL, provides a systematic treatment for financial Text-to-SQL from the perspectives of prompt construction, parameter-efficient fine-tuning and output calibration. Extensive experimental results on BULL demonstrate that FinSQL achieves the state-of-the-art Text-to-SQL performance at a small cost; furthermore, FinSQL can bring up to 36.64% performance improvement in scenarios requiring few-shot cross-database model transfer.

cs.CL↗

arXiv2024

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training. In this report, we attempt to address these two issues. (1) From a data perspective, we propose a method to measure the strength of preferences within the data, based on a voting mechanism of multiple reward models. Experimental results confirm that data with varying preference strengths have different impacts on reward model performance. We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset and fully leverage high-quality preference data. (2) From an algorithmic standpoint, we introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization. Furthermore, we employ meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples, and this approach can be utilized for iterative RLHF optimization.

cs.AI↗

arXiv2024

Efficient Temporal Butterfly Counting and Enumeration on Temporal Bipartite Graphs

Bipartite graphs characterize relationships between two different sets of entities, like actor-movie, user-item, and author-paper. The butterfly, a 4-vertices 4-edges (2,2)-biclique, is the simplest cohesive motif in a bipartite graph and is the fundamental component of higher-order substructures. Counting and enumerating the butterflies offer significant benefits across various applications, including fraud detection, graph embedding, and community search. While the corresponding motif, the triangle, in the unipartite graphs has been widely studied in both static and temporal settings, the extension of butterfly to temporal bipartite graphs remains unexplored. In this paper, we investigate the temporal butterfly counting and enumeration problem: count and enumerate the butterflies whose edges establish following a certain order within a given duration. Towards efficient computation, we devise a non-trivial baseline rooted in the state-of-the-art butterfly counting algorithm on static graphs, further, explore the intrinsic property of the temporal butterfly, and develop a new optimization framework with a compact data structure and effective priority strategy. The time complexity is proved to be significantly reduced without compromising on space efficiency. In addition, we generalize our algorithms to practical streaming settings and multi-core computing architectures. Our extensive experiments on 11 large-scale real-world datasets demonstrate the efficiency and scalability of our solutions.

cs.DS↗

arXiv2023

Stability of Hardy-Littlewood-Sobolev inequalities with explicit lower bounds

In this paper, we establish the stability for the Hardy-Littlewood-Sobolev (HLS) inequalities with explicit lower bounds. By establishing the relation between the stability of HLS inequalities and the stability of fractional Sobolev inequalities, we also give the stability of the fractional Sobolev inequalities with the lower bounds. This extends the stability of Sobolev inequalities with the explicit lower bounds established by Dolbeault, Esteban, Figalli, Frank and Loss in [16] to the fractional order case. Our proofs are based on the competing symmetries, the continuous Steiner symmetrization inequality for the HLS integral and the dual stability theory.

math.AP↗

arXiv2023

Learning Neural Operators on Riemannian Manifolds

In Artificial Intelligence (AI) and computational science, learning the mappings between functions (called operators) defined on complex computational domains is a common theoretical challenge. Recently, Neural Operator emerged as a promising framework with a discretisation-independent model structure to break the fixed-dimension limitation of classical deep learning models. However, existing operator learning methods mainly focus on regular computational domains, and many components of these methods rely on Euclidean structural data. In real-life applications, many operator learning problems are related to complex computational domains such as complex surfaces and solids, which are non-Euclidean and widely referred to as Riemannian manifolds. Here, we report a new concept, Neural Operator on Riemannian Manifolds (NORM), which generalises Neural Operator from being limited to Euclidean spaces to being applicable to Riemannian manifolds, and can learn the mapping between functions defined on any real-life complex geometries, while preserving the discretisation-independent model structure. NORM shifts the function-to-function mapping to finite-dimensional mapping in the Laplacian eigenfunctions' subspace of geometry, and holds universal approximation property in learning operators on Riemannian manifolds even with only one fundamental block. The theoretical and experimental analysis prove that NORM is a significant step forward in operator learning and has the potential to solve complex problems in many fields of applications sharing the same nature and theoretical principle.

math.NA↗

arXiv2023

MUST: An Effective and Scalable Framework for Multimodal Search of Target Modality

We investigate the problem of multimodal search of target modality, where the task involves enhancing a query in a specific target modality by integrating information from auxiliary modalities. The goal is to retrieve relevant objects whose contents in the target modality match the specified multimodal query. The paper first introduces two baseline approaches that integrate techniques from the Database, Information Retrieval, and Computer Vision communities. These baselines either merge the results of separate vector searches for each modality or perform a single-channel vector search by fusing all modalities. However, both baselines have limitations in terms of efficiency and accuracy as they fail to adequately consider the varying importance of fusing information across modalities. To overcome these limitations, the paper proposes a novel framework, called MUST. Our framework employs a hybrid fusion mechanism, combining different modalities at multiple stages. Notably, we leverage vector weight learning to determine the importance of each modality, thereby enhancing the accuracy of joint similarity measurement. Additionally, the proposed framework utilizes a fused proximity graph index, enabling efficient joint search for multimodal queries. MUST offers several other advantageous properties, including pluggable design to integrate any advanced embedding techniques, user flexibility to customize weight preferences, and modularized index construction. Extensive experiments on real-world datasets demonstrate the superiority of MUST over the baselines in terms of both search accuracy and efficiency. Our framework achieves over 10x faster search times while attaining an average of 93% higher accuracy. Furthermore, MUST exhibits scalability to datasets containing more than 10 million data elements.

cs.DB↗

arXiv2023

A partial defense of algebraic relationalism

I defend algebraicism, according to which physical fields can be understood in terms of their structural relations without reference to a spacetime manifold, as a genuine relationalist view against the conventional wisdom that it is equivalent to substantivalism, according to which spacetime exists fundamentally. I criticize the standard version of algebraicism that is considered equivalent to substantivalism. Furthermore, I present alternative examples of algebraicism that better implement relationalism with their conceptual advantages over substantivalism or its standard algebraic counterpart.

physics.hist-ph↗

arXiv2023

HarsanyiNet: Computing Accurate Shapley Values in a Single Forward Propagation

The Shapley value is widely regarded as a trustworthy attribution metric. However, when people use Shapley values to explain the attribution of input variables of a deep neural network (DNN), it usually requires a very high computational cost to approximate relatively accurate Shapley values in real-world applications. Therefore, we propose a novel network architecture, the HarsanyiNet, which makes inferences on the input sample and simultaneously computes the exact Shapley values of the input variables in a single forward propagation. The HarsanyiNet is designed on the theoretical foundation that the Shapley value can be reformulated as the redistribution of Harsanyi interactions encoded by the network.

cs.LG↗

arXiv2023

Primordial Black Hole Mass Function with Mass Gap

Primordial black holes (PBHs) are mainly characterized by their mass function, in which there may be some huge suppression for certain mass spans. If this is the case, the absence of these PBHs will form mass gaps. In this paper, we investigate the PBH mass function with mass gap. Firstly, to obtain a data-supported PBH mass function with mass gap for subsolar masses PBHs, we fine-tune the coefficients of a model-independent power spectrum of primordial curvature perturbations. Then we take this unique PBH mass function into consideration and calculate the energy density spectrum of the stochastic gravitational wave background from PBH mergers. We find the location of its first peak almost has no relationship with the mass gap and is only determined by the probability distribution of frequencies at which PBH binaries merge. Apart from the first peak, there must be an accompanying smaller trough at higher frequency resulting from the mass gap. Therefore, the detection of this smaller trough will provide more information about inflation and PBH formation.

astro-ph.CO↗

arXiv2023

Planar thermal Hall effect from phonons in a Kitaev candidate material

Kitaev materials are a promising platform for the realization of quantum spin liquid states. The thermal Hall effect has emerged as a potential probe of exotic excitations within such states. In the Kitaev candidate material $α$-RuCl$_{3}$, however, the thermal Hall conductivity $κ_{xy}$ has been attributed not only to exotic Majorana fermions or chiral magnons, but also to phonons. It has been shown theoretically that the former two types of heat carriers can generate a "planar" thermal Hall effect, whereby the magnetic field is parallel to the heat current, as observed experimentally, but it is unknown whether phonons also could. Here we show that a planar thermal Hall effect is present in another Kitaev candidate material, Na$_{2}$Co$_{2}$TeO$_{6}$. On the basis of a striking similarity between the temperature and field dependence of $κ_{xy}$ and that of the phonon-dominated thermal conductivity $κ_{xx}$, we argue that the planar thermal Hall effect in Na$_{2}$Co$_{2}$TeO$_{6}$ is generated by phonons. The phonon contributed planar $κ_{xy}$ also shows a strong sample dependence, which indicates an extrinsic origin of the mechanism. By conducting a complete study with different in-plane configurations of heat current $J$ and magnetic field $H$, $i.e.$ $H$ $\|$ $J$ and $H$ $\bot$ $J$, we observe a large difference in $κ_{xy}$ between these two configurations, which reveals that the direction of the heat current $J$ may play an important role in determining the planar thermal Hall effect. Our observation calls for a re-evaluation of the planar thermal Hall effect observed in $α$-RuCl$_{3}$.

cond-mat.str-el↗