arXiv subjects

Lu Xu

Publications and source records attributed to Lu Xu.

At least 55 records · Page 3Linked to original sources

arXiv2024

Scalar conservation law in a bounded domain with strong source at boundary

We consider a scalar conservation law with source in a bounded open interval $Ω\subseteq\mathbb R$. The equation arises from the macroscopic evolution of an interacting particle system. The source term models an external effort driving the solution to a given function $\varrho$ with an intensity function $V:Ω\to\mathbb R_+$ that grows to infinity at $\partialΩ$. We define the entropy solution $u \in L^\infty$ and prove the uniqueness. When $V$ is integrable, $u$ satisfies the boundary conditions introduced in [F. Otto, C. R. Acad. Sci. Paris 1996], which allows the solution to attain values at $\partialΩ$ different from the given boundary data. When the integral of $V$ blows up, $u$ satisfies an energy estimate and presents essential continuity at $\partialΩ$ in a weak sense.

math.AP↗

arXiv2024

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.

cs.CV↗

arXiv2024

End-to-end training of Multimodal Model and ranking Model

Traditional recommender systems heavily rely on ID features, which often encounter challenges related to cold-start and generalization. Modeling pre-extracted content features can mitigate these issues, but is still a suboptimal solution due to the discrepancies between training tasks and model parameters. End-to-end training presents a promising solution for these problems, yet most of the existing works mainly focus on retrieval models, leaving the multimodal techniques under-utilized. In this paper, we propose an industrial multimodal recommendation framework named EM3: End-to-end training of Multimodal Model and ranking Model, which sufficiently utilizes multimodal information and allows personalized ranking tasks to directly train the core modules in the multimodal model to obtain more task-oriented content features, without overburdening resource consumption. First, we propose Fusion-Q-Former, which consists of transformers and a set of trainable queries, to fuse different modalities and generate fixed-length and robust multimodal embeddings. Second, in our sequential modeling for user content interest, we utilize Low-Rank Adaptation technique to alleviate the conflict between huge resource consumption and long sequence length. Third, we propose a novel Content-ID-Contrastive learning task to complement the advantages of content and ID by aligning them with each other, obtaining more task-oriented content embeddings and more generalized ID embeddings. In experiments, we implement EM3 on different ranking models in two scenario, achieving significant improvements in both offline evaluation and online A/B test, verifying the generalizability of our method. Ablation studies and visualization are also performed. Furthermore, we also conduct experiments on two public datasets to show that our proposed method outperforms the state-of-the-art methods.

cs.IR↗

arXiv2024

HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting

Holistic understanding of urban scenes based on RGB images is a challenging yet important problem. It encompasses understanding both the geometry and appearance to enable novel view synthesis, parsing semantic labels, and tracking moving objects. Despite considerable progress, existing approaches often focus on specific aspects of this task and require additional inputs such as LiDAR scans or manually annotated 3D bounding boxes. In this paper, we introduce a novel pipeline that utilizes 3D Gaussian Splatting for holistic urban scene understanding. Our main idea involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints. Our approach offers the ability to render new viewpoints in real-time, yielding 2D and 3D semantic information with high accuracy, and reconstruct dynamic scenes, even in scenarios where 3D bounding box detection are highly noisy. Experimental results on KITTI, KITTI-360, and Virtual KITTI 2 demonstrate the effectiveness of our approach.

cs.CV↗

arXiv2024

Parameter-Efficient Conversational Recommender System as a Language Processing Task

Conversational recommender systems (CRS) aim to recommend relevant items to users by eliciting user preference through natural language conversation. Prior work often utilizes external knowledge graphs for items' semantic information, a language model for dialogue generation, and a recommendation module for ranking relevant items. This combination of multiple components suffers from a cumbersome training process, and leads to semantic misalignment issues between dialogue generation and item recommendation. In this paper, we represent items in natural language and formulate CRS as a natural language processing task. Accordingly, we leverage the power of pre-trained language models to encode items, understand user intent via conversation, perform item recommendation through semantic matching, and generate dialogues. As a unified model, our PECRS (Parameter-Efficient CRS), can be optimized in a single stage, without relying on non-textual metadata such as a knowledge graph. Experiments on two benchmark CRS datasets, ReDial and INSPIRED, demonstrate the effectiveness of PECRS on recommendation and conversation. Our code is available at: https://github.com/Ravoxsg/efficient_unified_crs.

cs.CL↗

arXiv2024

Incompressible limit for weakly asymmetric simple exclusion processes coupled through collision

We establish the incompressible limit of weakly asymmetric simple exclusion processes coupled through particle collisions. The incompressible limit depends on various parameters in the particle system and is linked to fluid dynamics equations. Our main contributions to previous results are the extension of the parameter space and the focus on local particle jumps. Our proof uses the relative entropy method. The main novelties in the proof are a Boltzmann-Gibbs principle (a replacement lemma) and a spectral gap estimate.

math.PR↗

arXiv2024

The Memory Perturbation Equation: Understanding Model's Sensitivity to Data

Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.

cs.LG↗

arXiv2023

Decomposed Prompt Tuning via Low-Rank Reparameterization

While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low intrinsic rank characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method.

cs.CL↗

arXiv2023

A stronger constant rank theorem

Motivated from one-dimensional rigidity results of entire solutions to Liouville equation, we consider the semilinear equation \begin{align} \label{liouvilleequationab} Δu=G(u) \quad \mbox{in $\mathbb{R}^n$}, \end{align}where $G>0, G'<0$ and $GG^{''}\le A(G')^2$, with $A>0$. Let $u$ be a smooth convex solution and $σ_k(D^2 u)$ be the $k$-th elementary symmetric polynomial with respect to $D^2u$. We prove stronger constant rank theorems in the following sense. (1) When $A\le 2$, if $σ_2(D^2u)$ takes a local minimum, then $D^2 u$ has constant rank $1$. (2) When $A\le \frac{n}{n-1}$, if $σ_n(D^2 u)$ takes a local minimum, then $σ_n(D^2 u)$ is always zero in the domain.

math.AP↗

arXiv2023

Learning Large Margin Sparse Embeddings for Open Set Medical Diagnosis

Fueled by deep learning, computer-aided diagnosis achieves huge advances. However, out of controlled lab environments, algorithms could face multiple challenges. Open set recognition (OSR), as an important one, states that categories unseen in training could appear in testing. In medical fields, it could derive from incompletely collected training datasets and the constantly emerging new or rare diseases. OSR requires an algorithm to not only correctly classify known classes, but also recognize unknown classes and forward them to experts for further diagnosis. To tackle OSR, we assume that known classes could densely occupy small parts of the embedding space and the remaining sparse regions could be recognized as unknowns. Following it, we propose Open Margin Cosine Loss (OMCL) unifying two mechanisms. The former, called Margin Loss with Adaptive Scale (MLAS), introduces angular margin for reinforcing intra-class compactness and inter-class separability, together with an adaptive scaling factor to strengthen the generalization capacity. The latter, called Open-Space Suppression (OSS), opens the classifier by recognizing sparse embedding space as unknowns using proposed feature space descriptors. Besides, since medical OSR is still a nascent field, two publicly available benchmark datasets are proposed for comparison. Extensive ablation studies and feature visualization demonstrate the effectiveness of each design. Compared with state-of-the-art methods, MLAS achieves superior performances, measured by ACC, AUROC, and OSCR.

cs.CV↗

arXiv2023

Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction

The DocRED dataset is one of the most popular and widely used benchmarks for document-level relation extraction (RE). It adopts a recommend-revise annotation scheme so as to have a large-scale annotated dataset. However, we find that the annotation of DocRED is incomplete, i.e., false negative samples are prevalent. We analyze the causes and effects of the overwhelming false negative problem in the DocRED dataset. To address the shortcoming, we re-annotate 4,053 documents in the DocRED dataset by adding the missed relation triples back to the original DocRED. We name our revised DocRED dataset Re-DocRED. We conduct extensive experiments with state-of-the-art neural models on both datasets, and the experimental results show that the models trained and evaluated on our Re-DocRED achieve performance improvements of around 13 F1 points. Moreover, we conduct a comprehensive analysis to identify the potential areas for further improvement. Our dataset is publicly available at https://github.com/tonytan48/Re-DocRED.

cs.CL↗

arXiv2023

Class-Adaptive Self-Training for Relation Extraction with Incompletely Annotated Training Data

Relation extraction (RE) aims to extract relations from sentences and documents. Existing relation extraction models typically rely on supervised machine learning. However, recent studies showed that many RE datasets are incompletely annotated. This is known as the false negative problem in which valid relations are falsely annotated as 'no_relation'. Models trained with such data inevitably make similar mistakes during the inference stage. Self-training has been proven effective in alleviating the false negative problem. However, traditional self-training is vulnerable to confirmation bias and exhibits poor performance in minority classes. To overcome this limitation, we proposed a novel class-adaptive re-sampling self-training framework. Specifically, we re-sampled the pseudo-labels for each class by precision and recall scores. Our re-sampling strategy favored the pseudo-labels of classes with high precision and low recall, which improved the overall recall without significantly compromising precision. We conducted experiments on document-level and biomedical relation extraction datasets, and the results showed that our proposed self-training framework consistently outperforms existing competitive methods on the Re-DocRED and ChemDisgene datasets when the training data are incompletely annotated. Our code is released at https://github.com/DAMO-NLP-SG/CAST.

cs.CL↗

arXiv2023

Efficient and scalable scheme for overcoming the pulse energy bottleneck of single-cycle laser sources

We propose a novel scheme called advanced dual-chirped optical parametric amplification (DC-OPA) that employs two kinds of nonlinear crystals (BiB$_3$O$_6$ and MgO-doped lithium niobate) to overcome the bottleneck of pulse energy scalability for single-cycle mid-infrared (MIR) laser pulses. In experiments, the advanced DC-OPA scheme achieved carrier-to-envelope phase-stable MIR laser pulses for a bandwidth of over one octave (1.4-3.1 $μ$m) with an output pulse energy of 53 mJ. The pulse duration was compressed to 8.58 fs, which corresponds to 1.05 cycles with a central wavelength of 2.44 $μ$m and peak power of 6 TW. To our knowledge, the obtained values for the pulse energy and peak power are the highest achieved for optical parametric amplification of single-cycle MIR laser pulses. Thanks to the energy scalability of the advanced DC-OPA scheme, it is potentially applicable to multi-TW sub-cycle laser pulses.

physics.optics↗

arXiv2023

Deep Electron Cloud-activity and Field-activity Relationships

Chemists have been pursuing the general mathematical laws to explain and predict molecular properties for a long time. However, most of the traditional quantitative structure-activity relationship (QSAR) models have limited application domains, e.g., they tend to have poor generalization performance when applied to molecules with parent structures different from those of the trained molecules. This paper attempts to develop a new QSAR method that is theoretically possible to predict various properties of molecules with diverse structures. The proposed deep electron cloud-activity relationships (DECAR) and deep field-activity relationships (DFAR) methods consist of three essentials: (1) A large number of molecule entities with activity data as training objects and responses; (2) three-dimensional electron cloud density (ECD) or related field data by the accurate density functional theory methods as input descriptors; (3) a deep learning model that is sufficiently flexible and powerful to learn the large data described above. DECAR and DFAR are used to distinguish 977 sweet and 1965 non-sweet molecules (with 6-fold data augmentation) and the classification performance is demonstrated to be significantly better than the traditional least squares support vector machine (LS-SVM) models using traditional descriptors. DECAR and DFAR would provide a possible way to establish a widely applicable, cumulative, and shareable artificial intelligence-driven QSAR system. They are likely to promote the development of an interactive platform to collect and share the accurate ECD and field data of millions of molecules with annotated activities. With enough input data, we envision the appearance of several deep networks trained for various molecular activities. Finally, we could anticipate a single DECAR or DFAR network to learn and infer various properties of interest for chemical molecules.

physics.chem-ph↗

arXiv2023

Better Sampling of Negatives for Distantly Supervised Named Entity Recognition

Distantly supervised named entity recognition (DS-NER) has been proposed to exploit the automatically labeled training data instead of human annotations. The distantly annotated datasets are often noisy and contain a considerable number of false negatives. The recent approach uses a weighted sampling approach to select a subset of negative samples for training. However, it requires a good classifier to assign weights to the negative samples. In this paper, we propose a simple and straightforward approach for selecting the top negative samples that have high similarities with all the positive samples for training. Our method achieves consistent performance improvements on four distantly supervised NER datasets. Our analysis also shows that it is critical to differentiate the true negatives from the false negatives.

cs.CL↗

arXiv2023

An Infinite-dimensional KAM Theorem with Normal Degeneracy

In this paper, we consider a classical Hamiltonian normal form with degeneracy in normal direction. In previous results, one needs to assume that the perturbation satisfies certain non-degenerate conditions in order to remove the degeneracy in the normal form. In stead of that, we introduce a topological degree condition and a weak convexity condition, which are easy to be verified, and we prove the persistence of lower dimensional tori without any restriction on perturbation but only smallness and analyticity.

math.DS↗

arXiv2023

Equilibrium perturbations for stochastic interacting systems

We consider the equilibrium perturbations for two stochastic systems: the $d$-dimensional generalized exclusion process and the one-dimensional chain of anharmonic oscillators. We add a perturbation of order $N^{-α}$ to the equilibrium profile and speed up the process by $N^{1+κ}$ for parameters $0<κ\leα$. Under some additional constraints on $κ$ and $α$, we show the perturbed quantities evolve according to the Burgers equation in the exclusion process, and to two decoupled Burgers equations in the anharmonic chain, both in the smooth regime.

math.PR↗

arXiv2022

Quasi-static limit for a hyperbolic conservation law

We study the quasi-static limit for the $L^\infty$ entropy weak solution of scalar one-dimensional hyperbolic equations with strictly concave or convex flux and time dependent boundary conditions. The quasi-stationary profile evolves with the quasi-static equation, whose entropy solution is determined by the stationary profile corresponding to the boundary data at a given time.

math.AP↗