arXiv subjects

Lu Chen

Publications and source records attributed to Lu Chen.

At least 181 records · Page 10Linked to original sources

arXiv2024

AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures

Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.

cs.AI↗

arXiv2024

Linear stability and resolvent analyses of a bluff-body stabilized flame with conjugate heat transfer

Conjugate heat transfer is a challenging fluid-structure coupling problem that can significantly influence flame stabilization and thermoacoustic instabilities. To properly capture combustion phenomena that involve conjugate heat transfer, careful modeling of chemical reactions in the fluid domain and heat transfer in the solid body is necessary and remains an active research topic. To this end, we derived a strongly-coupled method with a monolithic weak formulation to investigate the conjugate heat transfer between an anchored flame and a thermal conductive cylinder by means of linear stability analysis and resolvent analysis. We conduct parameter continuation with the Damkohler number to construct a bifurcation diagram and identify multiple baseflow states, including blow-off, anchored flame, and flashback. Linear stability analysis reveals the presence of a single unstable, non-oscillatory eigenmode for the base states on the anchored flame branch. This eigenmode plays a pivotal role in driving the bifurcation. Subsequently, resolvent analysis is performed to examine the amplification behavior of the fluid-solid coupled system under external forcing, showing that heat fluctuations are maximized when heat transfer between the fluid and solid is minimized.

physics.flu-dyn↗

arXiv2024

Contrasting Adversarial Perturbations: The Space of Harmless Perturbations

Existing works have extensively studied adversarial examples, which are minimal perturbations that can mislead the output of deep neural networks (DNNs) while remaining imperceptible to humans. However, in this work, we reveal the existence of a harmless perturbation space, in which perturbations drawn from this space, regardless of their magnitudes, leave the network output unchanged when applied to inputs. Essentially, the harmless perturbation space emerges from the usage of non-injective functions (linear or non-linear layers) within DNNs, enabling multiple distinct inputs to be mapped to the same output. For linear layers with input dimensions exceeding output dimensions, any linear combination of the orthogonal bases of the nullspace of the parameter consistently yields no change in their output. For non-linear layers, the harmless perturbation space may expand, depending on the properties of the layers and input samples. Inspired by this property of DNNs, we solve for a family of general perturbation spaces that are redundant for the DNN's decision, and can be used to hide sensitive data and serve as a means of model identification. Our work highlights the distinctive robustness of DNNs (i.e., consistency under large magnitude perturbations) in contrast to adversarial examples (vulnerability for small imperceptible noises).

cs.LG↗

arXiv2024

Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework

Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG's confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach. Our code and benchmark dataset are available at https://github.com/ict-bigdatalab/RC-RAG.

cs.CL↗

arXiv2024

Acquire Precise and Comparable Fundus Image Quality Score: FTHNet and FQS Dataset

The retinal fundus images are utilized extensively in the diagnosis, and their quality can directly affect the diagnosis results. However, due to the insufficient dataset and algorithm application, current fundus image quality assessment (FIQA) methods are not powerful enough to meet ophthalmologists` demands. In this paper, we address the limitations of datasets and algorithms in FIQA. First, we establish a new FIQA dataset, Fundus Quality Score(FQS), which includes 2246 fundus images with two labels: a continuous Mean Opinion Score varying from 0 to 100 and a three-level quality label. Then, we propose a FIQA Transformer-based Hypernetwork (FTHNet) to solve these tasks with regression results rather than classification results in conventional FIQA works. The FTHNet is optimized for the FIQA tasks with extensive experiments. Results on our FQS dataset show that the FTHNet can give quality scores for fundus images with PLCC of 0.9423 and SRCC of 0.9488, significantly outperforming other methods with fewer parameters and less computation complexity.We successfully build a dataset and model addressing the problems of current FIQA methods. Furthermore, the model deployment experiments demonstrate its potential in automatic medical image quality control. All experiments are carried out with 10-fold cross-validation to ensure the significance of the results.

eess.IV↗

arXiv2024

SciDFM: A Large Language Model with Mixture-of-Experts for Science

Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0.

cs.CL↗

arXiv2024

SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The codes and data are publicly available on https://github.com/OpenDFM/SciEval.

cs.CL↗

arXiv2024

ES-Gaussian: Gaussian Splatting Mapping via Error Space-Based Gaussian Completion

Accurate and affordable indoor 3D reconstruction is critical for effective robot navigation and interaction. Traditional LiDAR-based mapping provides high precision but is costly, heavy, and power-intensive, with limited ability for novel view rendering. Vision-based mapping, while cost-effective and capable of capturing visual data, often struggles with high-quality 3D reconstruction due to sparse point clouds. We propose ES-Gaussian, an end-to-end system using a low-altitude camera and single-line LiDAR for high-quality 3D indoor reconstruction. Our system features Visual Error Construction (VEC) to enhance sparse point clouds by identifying and correcting areas with insufficient geometric detail from 2D error maps. Additionally, we introduce a novel 3DGS initialization method guided by single-line LiDAR, overcoming the limitations of traditional multi-view setups and enabling effective reconstruction in resource-constrained environments. Extensive experimental results on our new Dreame-SR dataset and a publicly available dataset demonstrate that ES-Gaussian outperforms existing methods, particularly in challenging scenarios. The project page is available at https://chenlu-china.github.io/ES-Gaussian/.

cs.CV↗

arXiv2024

DIMS: Distributed Index for Similarity Search in Metric Spaces

Similarity search finds objects that are similar to a given query object based on a similarity metric. As the amount and variety of data continue to grow, similarity search in metric spaces has gained significant attention. Metric spaces can accommodate any type of data and support flexible distance metrics, making similarity search in metric spaces beneficial for many real-world applications, such as multimedia retrieval, personalized recommendation, trajectory analytics, data mining, decision planning, and distributed servers. However, existing studies mostly focus on indexing metric spaces on a single machine, which faces efficiency and scalability limitations with increasing data volume and query amount. Recent advancements in similarity search turn towards distributed methods, while they face challenges including inefficient local data management, unbalanced workload, and low concurrent search efficiency. To this end, we propose DIMS, an efficient Distributed Index for similarity search in Metric Spaces. First, we design a novel three-stage heterogeneous partition to achieve workload balance. Then, we present an effective three-stage indexing structure to efficiently manage objects. We also develop concurrent search methods with filtering and validation techniques that support efficient distributed similarity search. Additionally, we devise a cost-based optimization model to balance communication and computation cost. Extensive experiments demonstrate that DIMS significantly outperforms existing distributed similarity search approaches.

cs.DB↗

arXiv2024

AlignSum: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference

Text summarization tasks commonly employ Pre-trained Language Models (PLMs) to fit diverse standard datasets. While these PLMs excel in automatic evaluations, they frequently underperform in human evaluations, indicating a deviation between their generated summaries and human summarization preferences. This discrepancy is likely due to the low quality of fine-tuning datasets and the limited availability of high-quality human-annotated data that reflect true human preference. To address this challenge, we introduce a novel human summarization preference alignment framework AlignSum. This framework consists of three parts: Firstly, we construct a Data Pymarid with extractive, abstractive, and human-annotated summary data. Secondly, we conduct the Gaussian Resampling to remove summaries with extreme lengths. Finally, we implement the two-stage hierarchical fine-tuning with Data Pymarid after Gaussian Resampling. We apply AlignSum to PLMs on the human-annotated CNN/DailyMail and BBC XSum datasets. Experiments show that with AlignSum, PLMs like BART-Large surpass 175B GPT-3 in both automatic and human evaluations. This demonstrates that AlignSum significantly enhances the alignment of language models with human summarization preferences.

cs.CL↗

arXiv2024

Biglobal linear stability analysis of a ducted 2D premixed flame: intrinsic thermoacoustic mode and role of exceptional point

In this work, we aim to establish a detailed description of the physical mechanisms of intrinsic thermoacoustic modes and their interplay with duct acoustic modes. The biglobal linear stability analysis of an anchored laminar flame in an acoustic duct is carried out by exploiting the linearized compressible reactive flow equations. The pure intrinsic thermoacoustic mode is first identified and characterized under conditions where the eigenfrequencies of duct acoustic modes are sufficiently high and no mode interplay is present. Parameter variations, considering the inflow Mach number and acoustic reflection coefficients, are conducted to study the behaviors of eigenmode trajectories in the proximity of an exceptional point. Near the exceptional point, trajectory veering and mode switching are observed, and the extreme sensitivity renders the mode trajectory highly dependent on the choices of parameter and parameter variation path. One important finding is that different parameter variation paths can lead to inconsistent results in mode origin identification. Hence, we propose to characterize the thermoacoustic modes based on the flow structures. Specifically, the Helmholtz decomposition is employed to extract the potential and solenoidal components of the thermoacoustic modes. Intrinsic thermoacoustic modes and duct acoustic modes exhibit distinct flow structures, which are clearly distinguishable through the decomposed fields. Near the exceptional point, thermoacoustic modes display characteristics common to both intrinsic and duct modes. The results from compressible reactive flow analysis reveal that the flow structure of eigenmodes offers new insights into intrinsic thermoacoustic modes and mode interplay near the exceptional point.

physics.flu-dyn↗

arXiv2024

Optimal concentration level of anisotropic Trudinger-Moser functionals on any bounded domain

Let $F$ be convex and homogeneous of degree $1$, its polar $F^{o}$ represent a finsler metric on $\mathbb{R}^{n}$, and $Ω$ be any bounded open set in $\mathbb{R}^{n}$. In this paper, we first construct the theoretical structure of anisotropic harmonic transplantation. Using the anisotropic harmonic transplantation, co-area formula, limiting Sobolev approximation method, delicate estimate of level set of Green function, we investigate the optimal concentration level of the Trudinger-Moser functional \[ \int_Ωe^{λ_{n}|u|^{\frac{n}{n-1}}}dx \] under the anisotropic Dirichlet norm constraint $\int_ΩF^{n}\left( \nabla{u}\right) dx\leq1$, where $λ_{n}=n^{\frac{n}{n-1}}κ_{n}^{\frac{1}{n-1}}\ $ denotes the sharp constant of anisotropic Trudinger-Moser inequality in bounded domain and $κ_{n}$ is the Lebesgue measure of the unit Wulff ball. As an application. we can immediately deduce the existence of extremals for anisotropic Trudinger-Moser inequality on bounded domain. Finally, we also consider the optimal concentration level of the anisotropic singular Trudinger-Moser functional. The method is based on the limiting Hardy-Sobolev approximation method and constructing a suitable normalized anisotropic concentrating sequence.

math.AP↗

arXiv2024

Defining and Extracting generalizable interaction primitives from DNNs

Faithfully summarizing the knowledge encoded by a deep neural network (DNN) into a few symbolic primitive patterns without losing much information represents a core challenge in explainable AI. To this end, Ren et al. (2024) have derived a series of theorems to prove that the inference score of a DNN can be explained as a small set of interactions between input variables. However, the lack of generalization power makes it still hard to consider such interactions as faithful primitive patterns encoded by the DNN. Therefore, given different DNNs trained for the same task, we develop a new method to extract interactions that are shared by these DNNs. Experiments show that the extracted interactions can better reflect common knowledge shared by different DNNs.

cs.LG↗

arXiv2024

Optimal asymptotic lower bound for stability of fractional Sobolev inequality and the stability of Log-Sobolev inequality on the sphere

We establish the optimal asymptotic lower bound for the stability of fractional Sobolev inequality: \begin{equation}\label{Sob sta ine} \left\|(-Δ)^{s/2} U \right\|_2^2 - \mathcal S_{s,n} \| U\|_{\frac{2n}{n-2s}}^2\geq C_{n,s} d^{2}(U, \mathcal{M}_s), \end{equation} where $\mathcal{M}_s$ is the set of maximizers of the fractional Sobolev inequality of order $s$, $s\in (0, 1)$ and $C_{n,s}$ denotes the optimal lower bound of stability. We prove that the optimal lower bound $C_{n,s}$ behaves asymptotically at the order of $\frac{1}{n}$ when $n\rightarrow +\infty$ for any fixed $s\in (0,1)$. This extends the work by Dolbeault-Esteban-Figalli-Frank-Loss [19] on the stability of the first order Sobolev inequality and quantify the asymptotic behavior for lower bound of stability of fractional Sobolev inequality established by the current author's previous work in [15] in the case of $s\in (0, 1)$. Moreover, $C_{n,s}$ behaves asymptotically at the order of $s$ when $s\rightarrow 0$ for any given dimension $n$. (See Theorem 1.1.) As an application of this asymptotic estimate as $s\to 0$ and through the end-point differentiation method, we also derive the global stability for the log-Sobolev inequality on the sphere established by Beckner in [3,4] with the optimal asymptotic lower bound on the sphere. (see Theorem 1.6). This sharpens the earlier work by the authors in [14] where only the local stability for the log-Sobolev inequality on the sphere was proved. We also obtain the asymptotically optimal lower bound for the Hardy-Littlewood-Sobolev inequality when $s\to 0$ for fixed dimension $n$ and when $n\to \infty$ for fixed $s\in (0, 1)$ (See Theorem 1.4 and the subsequent Remark 1.5).

math.AP↗

arXiv2024

Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

Large Language Models (LLMs) often generate erroneous outputs, known as hallucinations, due to their limitations in discerning questions beyond their knowledge scope. While addressing hallucination has been a focal point in research, previous efforts primarily concentrate on enhancing correctness without giving due consideration to the significance of rejection mechanisms. In this paper, we conduct a comprehensive examination of the role of rejection, introducing the notion of model reliability along with corresponding metrics. These metrics measure the model's ability to provide accurate responses while adeptly rejecting questions exceeding its knowledge boundaries, thereby minimizing hallucinations. To improve the inherent reliability of LLMs, we present a novel alignment framework called Reinforcement Learning from Knowledge Feedback (RLKF). RLKF leverages knowledge feedback to dynamically determine the model's knowledge boundary and trains a reliable reward model to encourage the refusal of out-of-knowledge questions. Experimental results on mathematical questions affirm the substantial efficacy of RLKF in significantly enhancing LLM reliability.

cs.CL↗

arXiv2024

GlobalPointer: Large-Scale Plane Adjustment with Bi-Convex Relaxation

Plane adjustment (PA) is crucial for many 3D applications, involving simultaneous pose estimation and plane recovery. Despite recent advancements, it remains a challenging problem in the realm of multi-view point cloud registration. Current state-of-the-art methods can achieve globally optimal convergence only with good initialization. Furthermore, their high time complexity renders them impractical for large-scale problems. To address these challenges, we first exploit a novel optimization strategy termed \textit{Bi-Convex Relaxation}, which decouples the original problem into two simpler sub-problems, reformulates each sub-problem using a convex relaxation technique, and alternately solves each one until the original problem converges. Building on this strategy, we propose two algorithmic variants for solving the plane adjustment problem, namely \textit{GlobalPointer} and \textit{GlobalPointer++}, based on point-to-plane and plane-to-plane errors, respectively. Extensive experiments on both synthetic and real datasets demonstrate that our method can perform large-scale plane adjustment with linear time complexity, larger convergence region, and robustness to poor initialization, while achieving similar accuracy as prior methods. The code is available at https://github.com/wu-cvgl/GlobalPointer.

cs.CV↗

arXiv2024

Symmetries as Isomorphisms

Symmetries and isomorphisms play similar conceptual roles when we consider how models represent physical situations, but they are formally distinct, as two models related by symmetries are not typically isomorphic. I offer a rigorous categorical strategy that formulate symmetries as isomorphisms between models and apply it to classical electromagnetism, and evaluate its philosophical significance in relation to the recent debate between `sophistication' and `reduction'. In addition to traditional spacetime models, I also consider algebraic models, in which case we can use the method of natural operators to address the problem of ontological nonperspicuity faced by the categorical strategy. Finally, I briefly expound on the significance of symmetries as isomorphisms in the framework of Univalent Foundations, in which isomorphic structures are formally identified.

physics.hist-ph↗

arXiv2024

Univalence and Ontic Structuralism

The persistent challenge of formulating ontic structuralism in a rigorous manner, which prioritizes structures over the entities they contain, calls for a transformation of traditional logical frameworks. I argue that Univalent Foundations (UF), which feature the axiom that all isomorphic structures are identical, offer such a foundation and are more attractive than other proposed structuralist frameworks. Furthermore, I delve into the significance in the case of the hole argument and, very briefly, the nature of symmetries.

physics.hist-ph↗