arXiv subjects

Ming Wen

Publications and source records attributed to Ming Wen.

At least 37 records · Page 2Linked to original sources

arXiv2025

PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities

Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR.

cs.CR↗

arXiv2025

Influence of Bi Alloying on GaAs Valence Band Structure

Bi alloying is predicted to transform GaAs from a semiconductor to a topological insulator or semi-metal. To date, studies of the GaAs$_{1-x}$Bi$_x$ alloy band structure have been limited, and the origins of Bi-induced enhancement of the spin-orbit splitting energy, $Δ_\mathrm{SO}$, are unresolved. Here, we present high-resolution angle-resolved photoemission spectroscopy (ARPES) of droplet-free epitaxial GaAs$_{1-x}$Bi$_x$ films with $x_{\mathrm{Bi}}$ = 0.06. In addition to quantifying the Bi-induced shifts of the light-hole and heavy-hole valence bands, we probe the origins of the Bi-enhanced $Δ_\mathrm{SO}$. Using exact-two-component density functional theory calculations, we identify the key role of Bi p-orbitals in the upward shift of the light-hole and heavy-hole bands that results in the Bi-enhanced $Δ_\mathrm{SO}$.

cond-mat.mtrl-sci↗

arXiv2025

What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

The increasing development of LLMs in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and ten sub-categories, and analyzed the root cause for common bug types. To better understand the performance of LLMs in real-world projects, we also manually created a real-world benchmark RWPB. We analyzed bugs on RWPB to highlight distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Our comprehensive and extensive study provides insights into the current limitations of LLM-based code generation and opportunities for enhancing the accuracy and quality of the generated code.

cs.SE↗

arXiv2025

An Empirical Study of Production Incidents in Generative AI Cloud Services

The ever-increasing demand for generative artificial intelligence (GenAI) has motivated cloud-based GenAI services such as Azure OpenAI Service and Amazon Bedrock. Like any large-scale cloud service, failures are inevitable in cloud-based GenAI services, resulting in user dissatisfaction and significant monetary losses. However, GenAI cloud services, featured by their massive parameter scales, hardware demands, and usage patterns, present unique challenges, including generated content quality issues and privacy concerns, compared to traditional cloud services. To understand the production reliability of GenAI cloud services, we analyzed production incidents from a leading GenAI cloud service provider spanning in the past four years. Our study (1) presents the general characteristics of GenAI cloud service incidents at different stages of the incident life cycle; (2) identifies the symptoms and impacts of these incidents on GenAI cloud service quality and availability; (3) uncovers why these incidents occurred and how they were resolved; (4) discusses open research challenges in terms of incident detection, triage, and mitigation, and sheds light on potential solutions.

cs.DC↗

arXiv2025

Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

Large language models (LLMs) typically require fine-tuning for domain-specific tasks, and LoRA offers a computationally efficient approach by training low-rank adapters. LoRA is also communication-efficient for federated LLMs when multiple users collaboratively fine-tune a global LLM model without sharing their proprietary raw data. However, even the transmission of local adapters between a server and clients risks serious privacy leakage. Applying differential privacy (DP) to federated LoRA encounters a dilemma: adding noise to both adapters amplifies synthetic noise on the model, while fixing one adapter impairs the learnability of fine-tuning. In this paper, we propose FedASK (Differentially Private Federated Low Rank Adaptation with Double Sketching) , a novel federated LoRA framework to enable effective updating of both low-rank adapters with robust differential privacy. Inspired by randomized SVD, our key idea is a two-stage sketching pipeline. This pipeline first aggregates carefully sketched, privacy-preserving local updates, and then reconstructs the global matrices on the server to facilitate effective updating of both adapters. We theoretically prove FedASK's differential privacy guarantee and its exact aggregation property. Comprehensive experiments demonstrate that FedASK consistently outperforms baseline methods across a variety of privacy settings and data distributions.

cs.CR↗

arXiv2025

Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation

We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality.

eess.AS↗

arXiv2025

CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building

Project building is pivotal to support various program analysis tasks, such as generating intermediate rep- resentation code for static analysis and preparing binary code for vulnerability reproduction. However, automating the building process for C/C++ projects is a highly complex endeavor, involving tremendous technical challenges, such as intricate dependency management, diverse build systems, varied toolchains, and multifaceted error handling mechanisms. Consequently, building C/C++ projects often proves to be difficult in practice, hindering the progress of downstream applications. Unfortunately, research on facilitating the building of C/C++ projects remains to be inadequate. The emergence of Large Language Models (LLMs) offers promising solutions to automated software building. Trained on extensive corpora, LLMs can help unify diverse build systems through their comprehension capabilities and address complex errors by leveraging tacit knowledge storage. Moreover, LLM-based agents can be systematically designed to dynamically interact with the environment, effectively managing dynamic building issues. Motivated by these opportunities, we first conduct an empirical study to systematically analyze the current challenges in the C/C++ project building process. Particularly, we observe that most popular C/C++ projects encounter an average of five errors when relying solely on the default build systems. Based on our study, we develop an automated build system called CXXCrafter to specifically address the above-mentioned challenges, such as dependency resolution. Our evaluation on open-source software demonstrates that CXXCrafter achieves a success rate of 78% in project building. Specifically, among the Top100 dataset, 72 projects are built successfully by both CXXCrafter and manual efforts, 3 by CXXCrafter only, and 14 manually only. ...

cs.SE↗

arXiv2025

CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution

Code benchmarks such as HumanEval are widely adopted to evaluate Large Language Models' (LLMs) coding capabilities. However, there is an unignorable programming language bias in existing code benchmarks -- over 95% code generation benchmarks are dominated by Python, leaving the LLMs' capabilities in other programming languages such as Java and C/C++ unknown. Moreover, coding task bias is also crucial. Most benchmarks focus on code generation capability, while benchmarks for code reasoning (given input, reasoning output; and given output, reasoning input), an essential coding capability, are insufficient. Yet, constructing multi-lingual benchmarks can be expensive and labor-intensive, and codes in contest websites such as Leetcode suffer from data contamination during training. To fill this gap, we propose CRUXEVAL-X, a multi-lingual code reasoning benchmark that contains 19 programming languages. It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total. In particular, the construction pipeline of CRUXEVAL-X works in a fully automated and test-guided manner, which iteratively generates and repairs based on execution feedback. Also, to cross language barriers (e.g., dynamic/static type systems in Python/C++), we formulated various transition rules between language pairs to facilitate translation. Our intensive evaluation of 24 representative LLMs reveals the correlation between language pairs. For example, TypeScript and JavaScript show a significant positive correlation, while Racket has less correlation with other languages. More interestingly, even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages, revealing the cross-language generalization of LLMs.

cs.AI↗

arXiv2024

Multi-Programming Language Sandbox for LLMs

We introduce MPLSandbox, an out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler and analysis tools for Large Language Models (LLMs). It can automatically identify the programming language of the code, compiling and executing it within an isolated sub-sandbox to ensure safety and stability. In addition, MPLSandbox also integrates both traditional and LLM-based code analysis tools, providing a comprehensive analysis of generated code. MPLSandbox can be effortlessly integrated into the training and deployment of LLMs to improve the quality and correctness of their generated code. It also helps researchers streamline their workflows for various LLM-based code-related tasks, reducing the development cost. To validate the effectiveness of MPLSandbox, we integrate it into training and deployment approaches, and also employ it to optimize workflows for a wide range of real-world code-related tasks. Our goal is to enhance researcher productivity on LLM-based code-related tasks by simplifying and automating workflows through delegation to MPLSandbox.

cs.SE↗

arXiv2024

Green/WeakCoupling: Implementation of fully self-consistent finite-temperature many-body perturbation theory for molecules and solids

The accurate ab initio simulation of molecules and periodic solids with diagrammatic perturbation theory is an important task in quantum chemistry, condensed matter physics, and materials science. In this article, we present the WeakCoupling module of the open-source software package Green, which implements fully self-consistent diagrammatic weak coupling simulations, capable of dealing with real materials in the finite-temperature formalism. The code is licensed under the permissive MIT license. We provide self-consistent GW (scGW) and self-consistent second-order Green's function perturbation theory (GF2) solvers, analysis tools, and post-processing methods. This paper summarizes the theoretical methods implemented and provides background, tutorials and practical instructions for running simulations.

cond-mat.mtrl-sci↗

arXiv2024

Quasiparticle and fully self-consistent GW methods: an unbiased analysis using Gaussian orbitals

We present a comparison of various approximations to self-consistency in the GW method, including the one-shot G0W0 method, different quasiparticle self-consistency schemes, and the fully self-consistent GW (scGW) approach. To ensure an unbiased and equitable comparison, we have implemented all the schemes with the same underlying Matsubara formalism, while employing Gaussian orbitals to describe the system. Aiming to assess and compare different GW schemes, we analyze band gaps in semiconductors and insulators, as well as ionization potentials in molecules. Our findings reveal that for solids, the different self-consistency schemes perform very similarly. However, for molecules, full self-consistency outperforms all other approximations, i.e., the one-shot and quasiparticle self-consistency GW schemes. Our work highlights the importance of implementation details when comparing different GW methods. By employing state-of-the-art fully self-consistent, finite temperature GW calculations, we have successfully addressed discrepancies in the existing literature regarding its performance. Our results also indicate that when stringent thresholds are imposed, the scGW method consistently yields accurate results.

cond-mat.str-el↗

arXiv2024

Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning

Code Pre-trained Models (CodePTMs) based vulnerability detection have achieved promising results over recent years. However, these models struggle to generalize as they typically learn superficial mapping from source code to labels instead of understanding the root causes of code vulnerabilities, resulting in poor performance in real-world scenarios beyond the training instances. To tackle this challenge, we introduce VulLLM, a novel framework that integrates multi-task learning with Large Language Models (LLMs) to effectively mine deep-seated vulnerability features. Specifically, we construct two auxiliary tasks beyond the vulnerability detection task. First, we utilize the vulnerability patches to construct a vulnerability localization task. Second, based on the vulnerability features extracted from patches, we leverage GPT-4 to construct a vulnerability interpretation task. VulLLM innovatively augments vulnerability classification by leveraging generative LLMs to understand complex vulnerability patterns, thus compelling the model to capture the root causes of vulnerabilities rather than overfitting to spurious features of a single task. The experiments conducted on six large datasets demonstrate that VulLLM surpasses seven state-of-the-art models in terms of effectiveness, generalization, and robustness.

cs.CR↗

arXiv2024

Comparing self-consistent GW and vertex corrected G0W0 (G0W0Γ) accuracy for molecular ionization potentials

We test the performance of self-consistent GW and several representative implementations of vertex corrected G0W0 (G0W0Γ). These approaches are tested on benchmark data sets covering full valence spectra (first ionization potentials and some inner valence shell excitations). For small molecules, when comparing against state of the art wave function techniques, our results show that performing full self-consistency in the GW scheme either systematically outperforms vertex corrected G0W0 or gives results of at least the same quality. Moreover, the G0W0Γ results in additional computational cost when compared to G0W0 or self-consistent GW and the G0W0Γ dependency on the starting mean-filed solution is frequently larger than the magnitude of the vertex correction. Consequently, for molecular systems self-consistent GW performed on imaginary axis and then followed by modern analytical continuation techniques offers a more reliable approach to make predictions of IP spectra.

physics.chem-ph↗

arXiv2024

Large Exciton Binding Energy in the Bulk van der Waals Magnet CrSBr

Excitons, bound electron-hole pairs, influence the optical properties in strongly interacting solid state systems. Excitons and their associated many-body physics are typically most stable and pronounced in monolayer materials. Bulk systems with large exciton binding energies, on the other hand, are rare and the mechanisms driving their stability are still relatively unexplored. Here, we report an exceptionally large exciton binding energy in single crystals of the bulk van der Waals antiferromagnet CrSBr. Utilizing state-of-the-art angle-resolved photoemission spectroscopy and self-consistent ab-initio GW calculations, we present direct spectroscopic evidence that robust electronic and structural anisotropy can significantly amplify the exciton binding energy within bulk crystals. Furthermore, the application of a vertical electric field enables broad tunability of the optical and electronic properties. Our results indicate that CrSBr is a promising material for the study of the role of anisotropy in strongly interacting bulk systems and for the development of exciton-based optoelectronics.

cond-mat.mtrl-sci↗

arXiv2023

An Extensive Study on Adversarial Attack against Pre-trained Models of Code

Transformer-based pre-trained models of code (PTMC) have been widely utilized and have achieved state-of-the-art performance in many mission-critical applications. However, they can be vulnerable to adversarial attacks through identifier substitution or coding style transformation, which can significantly degrade accuracy and may further incur security concerns. Although several approaches have been proposed to generate adversarial examples for PTMC, the effectiveness and efficiency of such approaches, especially on different code intelligence tasks, has not been well understood. To bridge this gap, this study systematically analyzes five state-of-the-art adversarial attack approaches from three perspectives: effectiveness, efficiency, and the quality of generated examples. The results show that none of the five approaches balances all these perspectives. Particularly, approaches with a high attack success rate tend to be time-consuming; the adversarial code they generate often lack naturalness, and vice versa. To address this limitation, we explore the impact of perturbing identifiers under different contexts and find that identifier substitution within for and if statements is the most effective. Based on these findings, we propose a new approach that prioritizes different types of statements for various tasks and further utilizes beam search to generate adversarial examples. Evaluation results show that it outperforms the state-of-the-art ALERT in terms of both effectiveness and efficiency while preserving the naturalness of the generated adversarial examples.

cs.CR↗

arXiv2023

Automatic Root Cause Analysis via Large Language Models for Cloud Incidents

Ensuring the reliability and availability of cloud services necessitates efficient root cause analysis (RCA) for cloud incidents. Traditional RCA methods, which rely on manual investigations of data sources such as logs and traces, are often laborious, error-prone, and challenging for on-call engineers. In this paper, we introduce RCACopilot, an innovative on-call system empowered by the large language model for automating RCA of cloud incidents. RCACopilot matches incoming incidents to corresponding incident handlers based on their alert types, aggregates the critical runtime diagnostic information, predicts the incident's root cause category, and provides an explanatory narrative. We evaluate RCACopilot using a real-world dataset consisting of a year's worth of incidents from Microsoft. Our evaluation demonstrates that RCACopilot achieves RCA accuracy up to 0.766. Furthermore, the diagnostic information collection component of RCACopilot has been successfully in use at Microsoft for over four years.

cs.SE↗

arXiv2023

A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair

ChatGPT has revolutionized many research and industrial fields. ChatGPT has shown great potential in software engineering to boost various traditional tasks such as program repair, code understanding, and code generation. However, whether automatic program repair (APR) applies to deep learning (DL) programs is still unknown. DL programs, whose decision logic is not explicitly encoded in the source code, have posed unique challenges to APR. While to repair DL programs, an APR approach needs to not only parse the source code syntactically but also needs to understand the code intention. With the best prior work, the performance of fault localization is still far less than satisfactory (only about 30\%). Therefore, in this paper, we explore ChatGPT's capability for DL program repair by asking three research questions. (1) Can ChatGPT debug DL programs effectively? (2) How can ChatGPT's repair performance be improved by prompting? (3) In which way can dialogue help facilitate the repair? On top of that, we categorize the common aspects useful for prompt design for DL program repair. Also, we propose various prompt templates to facilitate the performance and summarize the advantages and disadvantages of ChatGPT's abilities such as detecting bad code smell, code refactoring, and detecting API misuse/deprecation.

cs.SE↗

arXiv2023

Attention: Not Just Another Dataset for Patch-Correctness Checking

Automated Program Repair (APR) techniques have drawn wide attention from both academia and industry. Meanwhile, one main limitation with the current state-of-the-art APR tools is that patches passing all the original tests are not necessarily the correct ones wanted by developers, i.e., the plausible patch problem. To date, various Patch-Correctness Checking (PCC) techniques have been proposed to address this important issue. However, they are only evaluated on very limited datasets as the APR tools used for generating such patches can only explore a small subset of the search space of possible patches, posing serious threats to external validity to existing PCC studies. In this paper, we construct an extensive PCC dataset (the largest manually labeled PCC dataset to our knowledge) to revisit all state-of-the-art PCC techniques. More specifically, our PCC dataset includes 1,988 patches generated from the recent PraPR APR tool, which leverages highly-optimized bytecode-level patch executions and can exhaustively explore all possible plausible patches within its large predefined search space (including well-known fixing patterns from various prior APR tools). Our extensive study of representative PCC techniques on the new dataset has revealed various surprising findings and provided guidelines for future PCC research.

cs.SE↗