arXiv subjects

Yuqi Zhu

Publications and source records attributed to Yuqi Zhu.

35 records · Page 2Linked to original sources

arXiv2025

LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding

Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs' long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs' capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K-1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering.

cs.SE↗

arXiv2025

KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges, especially when interacting with environments through generating executable actions. This inadequacy primarily stems from the lack of built-in action knowledge in language agents, which fails to effectively guide the planning trajectories during task solving and results in planning hallucination. To address this issue, we introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge. Specifically, KnowAgent employs an action knowledge base and a knowledgeable self-learning strategy to constrain the action path during planning, enabling more reasonable trajectory synthesis, and thereby enhancing the planning performance of language agents. Experimental results on HotpotQA and ALFWorld based on various backbone models demonstrate that KnowAgent can achieve comparable or superior performance to existing baselines. Further analysis indicates the effectiveness of KnowAgent in terms of planning hallucinations mitigation. Code is available in https://github.com/zjunlp/KnowAgent.

cs.CL↗

arXiv2025

Agent Planning with World Knowledge Model

Recent endeavors towards directly using large language models (LLMs) as agent models to execute interactive planning tasks have shown commendable results. Despite their achievements, however, they still struggle with brainless trial-and-error in global planning and generating hallucinatory actions in local planning due to their poor understanding of the ``real'' physical world. Imitating humans' mental world knowledge model which provides global prior knowledge before the task and maintains local dynamic knowledge during the task, in this paper, we introduce parametric World Knowledge Model (WKM) to facilitate agent planning. Concretely, we steer the agent model to self-synthesize knowledge from both expert and sampled trajectories. Then we develop WKM, providing prior task knowledge to guide the global planning and dynamic state knowledge to assist the local planning. Experimental results on three complex real-world simulated datasets with three state-of-the-art open-source LLMs, Mistral-7B, Gemma-7B, and Llama-3-8B, demonstrate that our method can achieve superior performance compared to various strong baselines. Besides, we analyze to illustrate that our WKM can effectively alleviate the blind trial-and-error and hallucinatory action issues, providing strong support for the agent's understanding of the world. Other interesting findings include: 1) our instance-level task knowledge can generalize better to unseen tasks, 2) weak WKM can guide strong agent model planning, and 3) unified WKM training has promising potential for further development. The code is available at https://github.com/zjunlp/WKM.

cs.CL↗

arXiv2024

LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.

cs.CL↗

arXiv2024

Showing LLM-Generated Code Selectively Based on Confidence of LLMs

Large Language Models (LLMs) have shown impressive abilities in code generation, but they may generate erroneous programs. Reading a program takes ten times longer than writing it. Showing these erroneous programs to developers will waste developers' energies and introduce security risks to software. To address the above limitations, we propose HonestCoder, a novel LLM-based code generation approach. HonestCoder selectively shows the generated programs to developers based on LLMs' confidence. The confidence provides valuable insights into the correctness of generated programs. To achieve this goal, we propose a novel approach to estimate LLMs' confidence in code generation. It estimates confidence by measuring the multi-modal similarity between LLMs-generated programs. We collect and release a multilingual benchmark named TruthCodeBench, which consists of 2,265 samples and covers two popular programming languages (i.e., Python and Java). We apply HonestCoder to four popular LLMs (e.g., DeepSeek-Coder and Code Llama) and evaluate it on TruthCodeBench. Based on the experiments, we obtain the following insights. (1) HonestCoder can effectively estimate LLMs' confidence and accurately determine the correctness of generated programs. For example, HonestCoder outperforms the state-of-the-art baseline by 27.79% in AUROC and 63.74% in AUCPR. (2) HonestCoder can decrease the number of erroneous programs shown to developers. Compared to eight baselines, it can show more correct programs and fewer erroneous programs to developers. (3) Compared to showing code indiscriminately, HonestCoder only adds slight time overhead (approximately 0.4 seconds per requirement). (4) We discuss future directions to facilitate the application of LLMs in software development. We hope this work can motivate broad discussions about measuring the reliability of LLMs' outputs in performing code-related tasks.

cs.SE↗

arXiv2024

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

Large Language Models (LLMs) like GPT-4, MedPaLM-2, and Med-Gemini achieve performance competitively with human experts across various medical benchmarks. However, they still face challenges in making professional diagnoses akin to physicians, particularly in efficiently gathering patient information and reasoning the final diagnosis. To this end, we introduce the RuleAlign framework, designed to align LLMs with specific diagnostic rules. We develop a medical dialogue dataset comprising rule-based communications between patients and physicians and design an alignment learning approach through preference learning. Experimental results demonstrate the effectiveness of the proposed approach. We hope that our work can serve as an inspiration for exploring the potential of LLMs as AI physicians.

cs.CL↗

arXiv2024

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released.

cs.CL↗

arXiv2024

DevEval: Evaluating Code Generation in Practical Software Projects

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

cs.SE↗

arXiv2023

Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models

Recently, Large Language Models (LLMs) have shown impressive abilities in code generation. However, existing LLMs' decoding strategies are designed for Natural Language (NL) generation, overlooking the differences between NL and programming languages (PL). Due to this oversight, a better decoding strategy for code generation remains an open question. In this paper, we conduct the first systematic study to explore a decoding strategy specialized in code generation. With an analysis of loss distributions of code tokens, we find that code tokens can be divided into two categories: challenging tokens that are difficult to predict and confident tokens that can be easily inferred. Among them, the challenging tokens mainly appear at the beginning of a code block. Inspired by the above findings, we propose a simple yet effective method: Adaptive Temperature (AdapT) sampling, which dynamically adjusts the temperature coefficient when decoding different tokens. We apply a larger temperature when sampling for challenging tokens, allowing LLMs to explore diverse choices. We employ a smaller temperature for confident tokens avoiding the influence of tail randomness noises. We apply AdapT sampling to LLMs with different sizes and conduct evaluations on two popular datasets. Results show that AdapT sampling significantly outperforms state-of-the-art decoding strategy.

cs.SE↗

arXiv2023

Rydberg-atom-based single-photon detection for haloscope axion searches

We propose a Rydberg-atom-based single-photon detector for signal readout in dark matter haloscope experiments between 40 $μ$eV and 200 $μ$eV (10 GHz and 50 GHz). At these frequencies, standard haloscope readout using linear amplifiers is limited by quantum measurement noise, which can be avoided by using a single-photon detector. Our single-photon detection scheme can offer scan rate enhancements up to a factor of $10^4$ over traditional linear amplifier readout, and is compatible with many different haloscope cavities. We identify multiple haloscope designs that could use our Rydberg-atom-based single-photon detector to search for QCD axions with masses above 40 $μ$eV (10 GHz), currently a minimally explored parameter space.

hep-ex↗

arXiv2023

How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?

Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models. In this paper, we investigate principal methodologies, in-context learning and data generation, for few-shot relation extraction via GPT-3.5 through exhaustive experiments. To enhance few-shot performance, we further propose task-related instructions and schema-constrained data generation. We observe that in-context learning can achieve performance on par with previous prompt learning approaches, and data generation with the large language model can boost previous solutions to obtain new state-of-the-art few-shot results on four widely-studied relation extraction datasets. We hope our work can inspire future research for the capabilities of large language models in few-shot relation extraction. Code is available in https://github.com/zjunlp/DeepKE/tree/main/example/llm.

cs.CL↗

arXiv2023

An improved synthetic signal injection routine for the Haloscope At Yale Sensitive To Axion Cold dark matter (HAYSTAC)

Microwave cavity haloscopes are among the most sensitive direct detection experiments searching for dark matter axions via their coupling to photons. When the power of the expected microwave signal due to axion-photon conversion is on the order of $10^{-24}$~W, having the ability to validate the detector response and analysis procedure by injecting realistic synthetic axion signals becomes helpful. Here we present a method based on frequency hopping spread spectrum for synthesizing axion signals in a microwave cavity haloscope experiment. It allows us to generate a narrow and asymmetric shape in frequency space that mimics an axion's spectral distribution, which is derived from a Maxwell-Boltzmann distribution. In addition, we show that the synthetic axion's power can be calibrated with reference to the system noise. Compared to the synthetic axion injection in HAYSTAC phase I, we demonstrated synthetic signal injection with a more realistic lineshape and calibrated power.

physics.ins-det↗

arXiv2023

New Results from HAYSTAC's Phase II Operation with a Squeezed State Receiver

A search for dark matter axions with masses $>10 μeV/c^{2}$ has been performed using the HAYSTAC experiment's squeezed state receiver to achieve sub-quantum limited noise. This report includes details of the design and operation of the experiment previously used to search for axions in the mass ranges $16.96-17.12$ and $17.14-17.28 μeV/c^{2}$($4.100-4.140$GHz) and $4.145-4.178$GHz) as well as upgrades to facilitate an extended search at higher masses. These upgrades include improvements to the data acquisition routine which have reduced the effective dead time by a factor of 5, allowing for the new region to be scanned $\sim$1.6 times faster with comparable sensitivity. No statistically significant evidence of an axion signal is found in the range $18.44-18.71μeV/c^{2}$($4.459-4.523$GHz), leading to an aggregate upper limit exclusion at the $90\%$ level on the axion-photon coupling of $2.06\times g_γ^{KSVZ}$.

hep-ex↗

arXiv2022

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

Large-scale pre-trained language models have achieved impressive results on a wide range of downstream tasks recently. However, fine-tuning an extremely large-scale pre-trained language model on limited target datasets is often plagued by overfitting and representation degradation. In this paper, we propose a Dynamic Parameter Selection (DPS) algorithm for the large-scale pre-trained models during fine-tuning, which adaptively selects a more promising subnetwork to perform staging updates based on gradients of back-propagation. Experiments on the GLUE benchmark show that DPS outperforms previous fine-tuning methods in terms of overall performance and stability, and consistently achieves better results with variable pre-trained language models. In addition, DPS brings a large magnitude of improvement in out-of-domain transferring experiments and low-resource scenarios, which shows that it can maintain stable general contextual features and reduce the representation collapse. We release our code at https://github.com/ZhangHaojie077/DPS

cs.CL↗

arXiv2022

EIT spectroscopy of high-lying Rydberg states in $^{39}K$

We present a study of the Rydberg spectrum in $^{39}K$ for $nS$ and $nD_{3/2}$ series connected to $5^2P_{1/2}$ using two-photon spectroscopy based on EIT in a heated vapor cell. We observed some 80 transitions from $5P_{1/2}$ to Rydberg states with principal quantum numbers $n\sim50-90$, and determined their transition frequencies and state energies with sub-GHz precision. Our spectroscopy results lay the groundwork for using Rydberg atoms as sensitive microwave photon detectors in searches for dark matter axions in the $\sim40-200$ $μ$eV mass range, which is a prime range for future axion searches suggested by theory studies.

physics.atom-ph↗

arXiv2021

Polarization Enhanced Deep Optical Dipole Trapping of $Λ$-cooled Polar Molecules

We demonstrate loading of SrF molecules into an optical dipole trap (ODT) via in-trap $Λ$-cooling. We find that this cooling can be optimized by a proper choice of relative ODT and $Λ$ beam polarizations. In this optimized configuration, we observe molecules with temperatures as low as 14(1) $μ$K in traps with depths up to 570 $μ$K. With optimized parameters, we transfer $\!\sim\!5$% of molecules from our radio-frequency magneto-optical trap into the ODT, at a density of $\sim\!2\times 10^{9}$ cm$^{-3}$, a phase space density of $\sim\!2\times 10^{-7}$, and with a trap lifetime of $\!\sim\!1$ s.

physics.atom-ph↗

arXiv2018

Continuous Production of Rovibronic Ground State RbCs Molecules via Short-Range Photoassociation to the $b^3Π_{1}-c^3Σ^+_{1}-B^1Π_1$ States

We have investigated rovibronic levels of the strongly mixed $b^3Π_1-c^3Σ^+_1-B^1Π_1$ states [the 2(1), 3(1), and 4(1) states in Hund's case (c) notation] of $^{85}$Rb$^{133}$Cs in the energy range of 13950$-$14200 cm$^{-1}$ using short-range photoassociation (PA). For selected PA states, vibrational and rotational branching in the $X^1Σ^+$ state have been investigated using resonance-enhanced multiphoton ionization and depletion spectroscopy. Efficient production of $^{85}$Rb$^{133}$Cs molecules in the rovibronic ground state $X^1Σ^+ (v=0, J=0)$, at up to $\sim$1$\times 10^4$ molecules/s, has been achieved.

physics.atom-ph↗