arXiv subjects

Derong Xu

Publications and source records attributed to Derong Xu.

49 records · Page 3Linked to original sources

arXiv2024

Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decoding

The impressive capabilities of large language models (LLMs) have attracted extensive interests of applying LLMs to medical field. However, the complex nature of clinical environments presents significant hallucination challenges for LLMs, hindering their widespread adoption. In this paper, we address these hallucination issues in the context of Medical Information Extraction (MIE) tasks by introducing ALternate Contrastive Decoding (ALCD). We begin by redefining MIE tasks as an identify-and-classify process. We then separate the identification and classification functions of LLMs by selectively masking the optimization of tokens during fine-tuning. During the inference stage, we alternately contrast output distributions derived from sub-task models. This approach aims to selectively enhance the identification and classification capabilities while minimizing the influence of other inherent abilities in LLMs. Additionally, we propose an alternate adaptive constraint strategy to more effectively adjust the scale and scope of contrastive tokens. Through comprehensive experiments on two different backbones and six diverse medical information extraction tasks, ALCD demonstrates significant improvements in resolving hallucination issues compared to conventional decoding methods.

cs.CL↗

arXiv2024

Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models

Model editing aims to precisely alter the behaviors of large language models (LLMs) in relation to specific knowledge, while leaving unrelated knowledge intact. This approach has proven effective in addressing issues of hallucination and outdated information in LLMs. However, the potential of using model editing to modify knowledge in the medical field remains largely unexplored, even though resolving hallucination is a pressing need in this area. Our observations indicate that current methods face significant challenges in dealing with specialized and complex knowledge in medical domain. Therefore, we propose MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing. MedLaSA harnesses the strengths of both adding extra parameters and locate-then-edit methods for medical model editing. We utilize causal tracing to identify the association of knowledge in neurons across different layers, and generate a corresponding scale set from the association value for each piece of knowledge. Subsequently, we incorporate scalable adapters into the dense layers of LLMs. These adapters are assigned scaling values based on the corresponding specific knowledge, which allows for the adjustment of the adapter's weight and rank. The more similar the content, the more consistent the scale between them. This ensures precise editing of semantically identical knowledge while avoiding impact on unrelated knowledge. To evaluate the editing impact on the behaviours of LLMs, we propose two model editing studies for medical domain: (1) editing factual knowledge for medical specialization and (2) editing the explanatory ability for complex knowledge. We build two novel medical benchmarking datasets and introduce a series of challenging and comprehensive metrics. Extensive experiments on medical LLMs demonstrate the editing efficiency of MedLaSA, without affecting unrelated knowledge.

cs.CL↗

arXiv2024

Online regularization of Poincaré map of storage rings with Shannon entropy

Shannon entropy, as a chaos indicator, is used for online Poincaré map regularization and dynamic aperture optimization in the National Synchrotron Light Source-II (NSLS-II) ring. Although various chaos indicators are widely used in studying nonlinear dynamical systems, including modern particle accelerators, it is the first time to use a measurable one in a real-world machine for online nonlinear optimization. Poincaré maps, constructed with the turn-by-turn beam trajectory readings from beam position monitors, are commonly used to observe the nonlinearity in ring-based accelerators. However, such observations typically only provide a qualitative interpretation. We analyze their entropy to quantify the chaos in measured Poincaré maps. After some canonical transformations on the Poincaré maps, not only can the commonly used nonlinear characterizations be extracted, but more importantly, the chaos can be quantitatively calibrated with Shannon entropy, and then used as the online optimization objectives.

physics.acc-ph↗

arXiv2024

Enhanced beam-beam modeling to include longitudinal variation during weak-strong simulation

Beam-beam interactions pose substantial challenges in the design and operation of circular colliders, significantly affecting their performance. In particular, the weak-strong simulation approach is pivotal for investigating single-particle dynamics during the collider design phase. This paper evaluates the limitations of existing models in weak-strong simulations, noting that while they accurately account for energy changes due to slingshot effects, they fail to incorporate longitudinal coordinate changes ($z$-variation). To address this gap, we introduce two novel transformations that enhance Hirata's original framework by including both $z$-variation and slingshot effect-induced energy changes. Through rigorous mathematical analysis and extensive weak-strong simulation studies, we validate the efficacy of these enhancements in achieving a more precise simulation of beam-beam interactions. Our results reveal that although $z$-variation constitutes a higher-order effect and does not substantially affect the emittance growth rate within the specific design parameters of the Electron-Ion Collider (EIC), the refined model offers improved accuracy, particularly in scenarios involving the interaction between beam-beam effects and other random diffusion processes, as well as in simulations incorporating realistic lattice models.

physics.acc-ph↗

arXiv2024

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.

cs.CL↗

arXiv2024

Assessing global crabbing scheme feasibility for Electron-Ion Collider

The Electron-Ion Collider (EIC) plans to utilize the local crabbing crossing scheme. This paper explores the feasibility of adopting a single crab cavity with adjusted voltage, inspired by the successful global crabbing scheme in KEKB, to restore effective head-on collisions. Using weak-strong simulations, the study assesses the potential of this global crabbing scheme for the EIC while emphasizing the need for adiabatic cavity ramping to prevent luminosity loss. Additionally, the research outlines potential risks associated with beam dynamics in implementing this scheme.

physics.acc-ph↗

arXiv2024

Advancing Electron Injection Dynamics and Mitigation Approaches in the Electron-Ion Collider Swap-out Injection Scheme

The Electron-Ion Collider (EIC) will use swap-out injection scheme for the Electron Storage Ring (ESR) to overcome limitations in polarization lifetime. However, the pursuit of highest luminosity with the required 28 nC electron bunches encounters stability challenges in the Rapid Cycling Synchrotron (RCS). One method is to inject multiple RCS bunches into a same ESR bucket. In this paper we perform simulation studies investigating proton emittance growth and electron emittance blowup in this injection scheme. Mitigation strategies are explored. These findings promise enhanced EIC stability and performance, shaping potential future operational improvements.

physics.acc-ph↗

arXiv2024

Large Language Model based Long-tail Query Rewriting in Taobao Search

In the realm of e-commerce search, the significance of semantic matching cannot be overstated, as it directly impacts both user experience and company revenue. Along this line, query rewriting, serving as an important technique to bridge the semantic gaps inherent in the semantic matching process, has attached wide attention from the industry and academia. However, existing query rewriting methods often struggle to effectively optimize long-tail queries and alleviate the phenomenon of "few-recall" caused by semantic gap. In this paper, we present BEQUE, a comprehensive framework that Bridges the sEmantic gap for long-tail QUEries. In detail, BEQUE comprises three stages: multi-instruction supervised fine tuning (SFT), offline feedback, and objective alignment. We first construct a rewriting dataset based on rejection sampling and auxiliary tasks mixing to fine-tune our large language model (LLM) in a supervised fashion. Subsequently, with the well-trained LLM, we employ beam search to generate multiple candidate rewrites, and feed them into Taobao offline system to obtain the partial order. Leveraging the partial order of rewrites, we introduce a contrastive learning method to highlight the distinctions between rewrites, and align the model with the Taobao online objectives. Offline experiments prove the effectiveness of our method in bridging semantic gap. Online A/B tests reveal that our method can significantly boost gross merchandise volume (GMV), number of transaction (#Trans) and unique visitor (UV) for long-tail queries. BEQUE has been deployed on Taobao, one of most popular online shopping platforms in China, since October 2023.

cs.IR↗

arXiv2024

Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models

Knowledge graph completion (KGC) is a widely used method to tackle incompleteness in knowledge graphs (KGs) by making predictions for missing links. Description-based KGC leverages pre-trained language models to learn entity and relation representations with their names or descriptions, which shows promising results. However, the performance of description-based KGC is still limited by the quality of text and the incomplete structure, as it lacks sufficient entity descriptions and relies solely on relation names, leading to sub-optimal results. To address this issue, we propose MPIKGC, a general framework to compensate for the deficiency of contextualized knowledge and improve KGC by querying large language models (LLMs) from various perspectives, which involves leveraging the reasoning, explanation, and summarization capabilities of LLMs to expand entity descriptions, understand relations, and extract structures, respectively. We conducted extensive evaluation of the effectiveness and improvement of our framework based on four description-based KGC models and four datasets, for both link prediction and triplet classification tasks.

cs.CL↗

arXiv2023

Social Context-aware GCN for Video Character Search via Scene-prior Enhancement

With the increasing demand for intelligent services of online video platforms, video character search task has attracted wide attention to support downstream applications like fine-grained retrieval and summarization. However, traditional solutions only focus on visual or coarse-grained social information and thus cannot perform well when facing complex scenes, such as changing camera view or character posture. Along this line, we leverage social information and scene context as prior knowledge to solve the problem of character search in complex scenes. Specifically, we propose a scene-prior-enhanced framework, named SoCoSearch. We first integrate multimodal clues for scene context to estimate the prior probability of social relationships, and then capture characters' co-occurrence to generate an enhanced social context graph. Afterwards, we design a social context-aware GCN framework to achieve feature passing between characters to obtain robust representation for the character search task. Extensive experiments have validated the effectiveness of SoCoSearch in various metrics.

cs.MM↗

arXiv2023

Nonlinear optics from hybrid dispersive orbits

In this paper we present an expansion of the technique of characterizing nonlinear optics from off-energy orbits (NOECO) to cover harmonic sextupoles in storage rings. The existing NOECO technique has been successfully used to correct the chromatic sextupole errors on the MAX-IV machine, however, it doesn't account for harmonic sextupoles, which are widely used on many other machines. Through generating vertical dispersion with chromatic skew quadrupoles, a measurable dependency of nonlinear optics on harmonic sextupoles can be observed from hybrid horizontal and vertical dispersive orbits. Proof of concept of our expanded technique was accomplished by simulations and beam measurements on the National Synchrotron Light Source II (NSLS-II) storage ring.

physics.acc-ph↗

arXiv2022

Synchro-betatron Resonance of Crab Crossing Scheme with Large Crossing Angle and Finite Bunch Length

Crab crossing scheme is an essential collision scheme to achieve high luminosity for the future colliders with large crossing angles. However, when bunch length of one or both colliding beams is comparable with the wavelength of the crab cavity voltage, the nonlinear dependence of the crabbing kick may present a challenge to the beam dynamics of the colliding beams and impact the beam quality as well as the luminosity lifetime. In this paper, the results of nonlinear dynamics in the crab crossing scheme are presented, using both analytical and numerical studies. The result indicates that higher-order synchro-betatron resonances may be excited in the crab crossing scheme with large crossing angle, which causes the beam quality deterioration and luminosity degradation. The studies also reveal possible countermeasures to suppress the synchro-beta resonance, hence mitigate the degradation of beam quality and luminosity.

physics.acc-ph↗

arXiv2022

Combined effects of Crab Dispersion and Momentum Dispersion in Colliders with Local Crab Crossing Scheme

In this paper, we present the effects of linear transverse-longitudinal coupling on beam size at Interaction Point (IP) of a collider with local crab crossing scheme, when time dependent transverse deflection (crab kicks) and dispersive orbit intertwine near IP. The analytic propagation formula and the closed orbit form of the crab dispersion and momentum dispersion are derived. The non-zero momentum dispersion at crab cavities and the non-ideal phase from crab cavities to IP are detailed with the derived propagation formula to predict the beam size distortion at IP with or without the beam-beam interaction. The linear results are compared with nonlinear simulation using the weak-strong beam-beam code.

physics.acc-ph↗