arXiv subjects

Weiyi Shang

Publications and source records attributed to Weiyi Shang.

At least 19 recordsLinked to original sources

arXiv2026

SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

cs.SE↗

arXiv2026

Do Privacy Policies Match with the Logs? An Empirical Study of Privacy Disclosure in Android Application Logs

Privacy policies are intended to inform users about how software systems collect and handle data, yet they often remain vague or incomplete. This paper presents an empirical study of patterns in log-related statements within privacy policies and their alignment with privacy disclosures observed in Android application logs. We analyzed 1,000 Android apps across multiple categories, generating 86,836,964 log entries. Our findings reveal that while most applications (88.0%) provide privacy policies, only 28.5% explicitly mention logging practices. Among those that reference logging, most clearly describe what information is logged; however, 27.7% of log-related statements remain overly simplistic or vague, offering limited insight into actual data collection. We further observed widespread privacy leakages in application logs, with 67.6% of apps leaking sensitive information not mentioned in their policies. Alarmingly, only 0.4% of applications demonstrated consistent alignment between declared policy contents and actual logged data. These findings highlight that current privacy policies provide incomplete or ambiguous descriptions of logging practices, which frequently do not align with actual logging behaviors.

cs.CR↗

arXiv2026

Is Your Private Information Logged? An Empirical Study on Android App Logs

With the rapid growth of mobile apps, users' concerns about their privacy have become increasingly prominent. Android app logs serve as crucial computer resources, aiding developers in debugging and monitoring the status of Android apps, while also containing a wealth of software system information. Previous studies have acknowledged privacy leaks in software logs and Android apps as significant issues without providing a comprehensive view of the privacy leaks in Android app logs. In this study, we build a comprehensive dataset of Android app logs and conduct an empirical study to analyze the status and severity of privacy leaks in Android app logs. Our study comprises three aspects: (1) Understanding real-world developers' concerns regarding privacy issues related to software logs; (2) Studying privacy leaks in the Android app logs; (3) Investigating the characteristics of privacy-leaking Android app logs and analyzing the reasons behind them. Our study reveals five different categories of concerns from real-world developers regarding privacy issues related to software logs and the prevalence of privacy leaks in Android app logs, with the majority stemming from developers' unawareness of such leaks. Additionally, our study provides developers with suggestions to safeguard their privacy from being logged.

cs.SE↗

arXiv2025

A Story About Cohesion and Separation: Label-Free Metric for Log Parser Evaluation

Log parsing converts log messages into structured event templates, allowing for automated log analysis and reducing manual inspection effort. To select the most compatible parser for a specific system, multiple evaluation metrics are commonly used for performance comparisons. However, existing evaluation metrics heavily rely on labeled log data, which limits prior studies to a fixed set of datasets and hinders parser evaluations and selections in the industry. Further, we discovered that different versions of ground-truth used in existing studies can lead to inconsistent performance conclusions. Motivated by these challenges, we propose a novel label-free template-level metric, PMSS (parser medoid silhouette score), to evaluate log parser performance. PMSS evaluates both parser grouping and template quality with medoid silhouette analysis and Levenshtein distance within a near-linear time complexity in general. To understand its relationship with label-based template-level metrics, FGA and FTA, we compared their evaluation outcomes for six log parsers on the standard corrected Loghub 2.0 dataset. Our results indicate that log parsers achieving the highest PMSS or FGA exhibit comparable performance, differing by only 2.1% on average in terms of the FGA score; the difference is 9.8% for FTA. PMSS is also significantly (p<1e-8) and positively correlated to both FGA and FTA: the Spearman's rho correlation coefficient of PMSS-FGA and PMSS-FTA are respectively 0.648 and 0.587, close to the coefficient between FGA and FTA (0.670). We further extended our discussion on how to interpret the conclusions from different metrics, identifying challenges in using PMSS, and provided guidelines on conducting parser selections with our metric. PMSS provides a valuable evaluation alternative when ground-truths are inconsistent or labels are unavailable.

cs.SE↗

arXiv2025

ADPerf: Investigating and Testing Performance in Autonomous Driving Systems

Obstacle detection is crucial to the operation of autonomous driving systems, which rely on multiple sensors, such as cameras and LiDARs, combined with code logic and deep learning models to detect obstacles for time-sensitive decisions. Consequently, obstacle detection latency is critical to the safety and effectiveness of autonomous driving systems. However, the latency of the obstacle detection module and its resilience to various changes in the LiDAR point cloud data are not yet fully understood. In this work, we present the first comprehensive investigation on measuring and modeling the performance of the obstacle detection modules in two industry-grade autonomous driving systems, i.e., Apollo and Autoware. Learning from this investigation, we introduce ADPerf, a tool that aims to generate realistic point cloud data test cases that can expose increased detection latency. Increasing latency decreases the availability of the detected obstacles and stresses the capabilities of subsequent modules in autonomous driving systems, i.e., the modules may be negatively impacted by the increased latency in obstacle detection. We applied ADPerf to stress-test the performance of widely used 3D obstacle detection modules in autonomous driving systems, as well as the propagation of such tests on trajectory prediction modules. Our evaluation highlights the need to conduct performance testing of obstacle detection components, especially 3D obstacle detection, as they can be a major bottleneck to increased latency of the autonomous driving system. Such an adverse outcome will also further propagate to other modules, reducing the overall reliability of autonomous driving systems.

cs.SE↗

arXiv2025

OFP-Repair: Repairing Floating-point Errors via Original-Precision Arithmetic

Errors in floating-point programs can lead to severe consequences, particularly in critical domains such as military, aerospace, and financial systems, making their repair a crucial research problem. In practice, some errors can be fixed using original-precision arithmetic, while others require high-precision computation. Developers often avoid addressing the latter due to excessive computational resources required. However, they sometimes struggle to distinguish between these two types of errors, and existing repair tools fail to assist in this differentiation. Most current repair tools rely on high-precision implementations, which are time-consuming to develop and demand specialized expertise. Although a few tools do not require high-precision programs, they can only fix a limited subset of errors or produce suboptimal results. To address these challenges, we propose a novel method, named OFP-Repair.On ACESO's dataset, our patches achieve improvements of three, seven, three, and eight orders of magnitude across four accuracy metrics. In real-world cases, our method successfully detects all five original-precision-repairable errors and fixes three, whereas ACESO only repairs one. Notably, these results are based on verified data and do not fully capture the potential of OFP-Repair. To further validate our method, we deploy it on a decade-old open bug report from GNU Scientific Library (GSL), successfully repairing five out of 15 bugs. The developers have expressed interest in our method and are considering integrating our tool into their development workflow. We are currently working on applying our patches to GSL. The results are highly encouraging, demonstrating the practical applicability of our technique.

cs.SE↗

arXiv2025

A Mathematics-Guided Approach to Floating-Point Error Detection

Floating-point program errors can lead to severe consequences, particularly in critical domains such as military applications. Only a small subset of inputs may induce substantial floating-point errors, prompting researchers to develop methods for identifying these error-inducing inputs. Although existing approaches have achieved some success, they still suffer from two major limitations: (1) High computational cost: The evaluation of error magnitude for candidate inputs relies on high-precision programs, which are prohibitively time-consuming. (2) Limited long-range convergence capability: Current methods exhibit inefficiency in search, making the process akin to finding a needle in a haystack. To address these two limitations, we propose a novel method, named MGDE, to detect error-inducing inputs based on mathematical guidance. By employing the Newton-Raphson method, which exhibits quadratic convergence properties, we achieve highly effective and efficient results. Since the goal of identifying error-inducing inputs is to uncover the underlying bugs, we use the number of bugs detected in floating-point programs as the primary evaluation metric in our experiments. As FPCC represents the most effective state-of-the-art approach to date, we use it as the baseline for comparison. The dataset of FPCC consists of 88 single-input floating-point programs. FPCC is able to detect 48 bugs across 29 programs, whereas our method successfully identifies 89 bugs across 44 programs. Moreover, FPCC takes 6.4096 times as long as our proposed method. We also deploy our method to multi-input programs, identifying a total of nine bugs with an average detection time of 0.6443 seconds per program. In contrast, FPCC fails to detect any bugs while requiring an average computation time of 100 seconds per program.

cs.SE↗

arXiv2025

Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths

Large Language Models (LLMs) have become central to automated code generation, yet existing approaches operate within a single-LLM paradigm: one model is selected and applied throughout the entire generation process. We observe that different LLMs exhibit complementary strengths: no single model dominates across all programming languages, algorithmic problem categories, or development stages. Multi-LLM collaboration, structured as per-stage, per-category routing rather than majority voting, produces higher-quality code than any individual model. Based on this observation, we propose PerfOrch, a multi-agent orchestration system that decomposes code generation into four collaborative agents: categorization, generation, debugging, and refinement. Each agent maintains a Memory module: a ranking matrix indexed by programming language and problem category, constructed from offline profiling and consulted at runtime to select the most suitable model for each task. We evaluate PerfOrch on two benchmarks, HumanEval-X and EffiBench-X, totaling 2,500 problems across five languages (Python, Java, C++, Go, and Rust). PerfOrch achieves average pass@1 rates of 97.19% on HumanEval-X and 95.83% on EffiBench-X, improving over the strongest single-model pipeline by 1.22-14.58 percentage points across languages. Notably, Memory rankings constructed solely from HumanEval-X profiling generalize to the entirely unseen EffiBench-X benchmark without re-profiling, demonstrating that the complementary-strength patterns PerfOrch exploits are properties of the models rather than artifacts of a specific problem distribution. Beyond correctness, PerfOrch improves execution time for 61-90% of solved problems with mean speedups of 4.7-29.9%, matching the refinement coverage of exhaustive multi-model evaluation at roughly half the token cost.

cs.SE↗

arXiv2025

Computing Floating-Point Errors by Injecting Perturbations

Floating-point programs form the foundation of modern science and engineering, providing the essential computational framework for a wide range of applications, such as safety-critical systems, aerospace engineering, and financial analysis. Floating-point errors can lead to severe consequences. Although floating-point errors widely exist, only a subset of inputs may trigger significant errors in floating-point programs. Therefore, it is crucial to determine whether a given input could produce such errors. Researchers tend to take the results of high-precision floating-point programs as oracles for detecting floating-point errors, which introduces two main limitations: (1) difficulty of implementation and (2) prolonged execution time. The two recent tools, ATOMU and FPCC, can partially address these issues. However, ATOMU suffers from false positives; while FPCC, though eliminating false positives, operates at a considerably slower speed. To address these two challenges, we propose a novel approach named PI-detector to computing floating-point errors effectively and efficiently. Our approach is based on the observation that floating-point errors stem from large condition numbers in atomic operations (such as addition and subtraction), which then propagate and accumulate. PI-detector injects small perturbations into the operands of individual atomic operations within the program and compares the outcomes of the original program with the perturbed version to compute floating-point errors. We evaluate PI-detector with datasets from ATOMU and HSED, as well as a complex linear system-solving program. Experimental results demonstrate that PI-detector can perform efficient and accurate floating-point error computation.

cs.SE↗

arXiv2025

An Empirical Study on the Capability of LLMs in Decomposing Bug Reports

Background: Bug reports are essential to the software development life cycle. They help developers track and resolve issues, but are often difficult to process due to their complexity, which can delay resolution and affect software quality. Aims: This study investigates whether large language models (LLMs) can assist developers in automatically decomposing complex bug reports into smaller, self-contained units, making them easier to understand and address. Method: We conducted an empirical study on 127 resolved privacy-related bug reports collected from Apache Jira. We evaluated ChatGPT and DeepSeek using different prompting strategies. We first tested both LLMs with zero-shot prompts, then applied improved prompts with demonstrations (using few-shot prompting) to measure their abilities in bug decomposition. Results: Our findings show that LLMs are capable of decomposing bug reports, but their overall performance still requires further improvement and strongly depends on the quality of the prompts. With zero-shot prompts, both studied LLMs (ChatGPT and DeepSeek) performed poorly. After prompt tuning, ChatGPT's true decomposition rate increased by 140\% and DeepSeek's by 163.64\%. Conclusions: LLMs show potential in helping developers analyze and decompose complex bug reports, but they still need improvement in terms of accuracy and bug understanding.

cs.SE↗

arXiv2024

DELA: A Novel Approach for Detecting Errors Induced by Large Atomic Condition Numbers

Numerical programs form the foundation of modern science and engineering, providing essential solutions to complex mathematical problems. Therefore, errors in numerical results would lead to harmful consequences, especially in safety-critical applications. Since only a few inputs may lead to substantial errors for numerical programs, it is essential to determine whether a given input could result in a significant error. Existing researchers tend to use the results of high-precision programs to assess whether there is a substantial error, which introduces three main challenges: difficulty of implementation, existence of potential faults in the detection of numerical errors, and long execution time. To address these limitations, we propose a novel approach named DELA. Our approach is based on the observation that most numerical errors stem from large condition numbers in atomic operations (such as subtraction), which then propagate and accumulate. DELA injects small perturbations into the results of individual atomic operations within the program and compares the outcomes of the original program with the perturbed version to detect errors. We evaluate DELA with datasets from ATOMU and HSED, as well as data from a complex linear system-solving program. Experimental results demonstrate that we can detect all the significant errors that were reported by prior research. DELA shows strong alignment with high-precision programs of ATOMU and HSED, with average Pearson and Spearman correlations of 0.86 and 0.61. Additionally, DELA effectively detects significant errors in complex programs, achieving correlation scores of 0.9763 and 0.8993. More importantly, in experiments with ATOMU and HSED, DELA's perturbed programs run within only 0.13% of the time needed by high-precision versions; while for the linear system-solving programs, DELA is 73.46 times faster than the high-precision programs.

cs.SE↗

arXiv2024

Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models

During software development, developers often make numerous modifications to the software to address existing issues or implement new features. However, certain changes may inadvertently have a detrimental impact on the overall system performance. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing, such as load testing, or component-level performance testing to detect performance regressions. However, performance testing for the entire system is often expensive and time-consuming, posing challenges to adapting to the rapid release cycles common in modern DevOps practices. System-level performance testing cannot be conducted until the system is fully built and deployed. On the other hand, component-level testing focuses on isolated components, neglecting overall system performance and the impact of system workloads. In this paper, we propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models. Our approach uses local performance data to identify deviations at the component level, and then propagate these deviations to the architectural model. We then use the architectural model to predict regressions in the performance of the overall system. We evaluate our approach on two open-source benchmark systems and show that it can effectively detect end-to-end system performance regressions from local performance deviations with different intensities and under various system workloads. More importantly, our approach can detect regressions as early as in the development phase, in contrast to existing approaches that require the system to be fully built and deployed. Our approach is lightweight and can complement traditional system performance testing when testing resources are scarce.

cs.SE↗

arXiv2024

Automatic String Data Validation with Pattern Discovery

In enterprise data pipelines, data insertions occur periodically and may impact downstream services if data quality issues are not addressed. Typically, such problems can be investigated and fixed by on-call engineers, but locating the cause of such problems and fixing errors are often time-consuming. Therefore, automatic data validation is a better solution to defend the system and downstream services by enabling early detection of errors and providing detailed error messages for quick resolution. This paper proposes a self-validate data management system with automatic pattern discovery techniques to verify the correctness of semi-structural string data in enterprise data pipelines. Our solution extracts patterns from historical data and detects erroneous incoming data in a top-down fashion. High-level information of historical data is analyzed to discover the format skeleton of correct values. Fine-grained semantic patterns are then extracted to strike a balance between generalization and specification of the discovered pattern, thus covering as many correct values as possible while avoiding over-fitting. To tackle cold start and rapid data growth, we propose an incremental update strategy and example generalization strategy. Experiments on large-scale industrial and public datasets demonstrate the effectiveness and efficiency of our method compared to alternative solutions. Furthermore, a case study on an industrial platform (Ant Group Inc.) with thousands of applications shows that our system captures meaningful data patterns in daily operations and helps engineers quickly identify errors.

cs.DB↗

arXiv2024

Easy over Hard: A Simple Baseline for Test Failures Causes Prediction

The test failure causes analysis is critical since it determines the subsequent way of handling different types of bugs, which is the prerequisite to get the bugs properly analyzed and fixed. After a test case fails, software testers have to inspect the test execution logs line by line to identify its root cause. However, manual root cause determination is often tedious and time-consuming, which can cost 30-40% of the time needed to fix a problem. Therefore, there is a need for automatically predicting the test failure causes to lighten the burden of software testers. In this paper, we present a simple but hard-to-beat approach, named NCChecker to automatically identify the failure causes for failed test logs. Our approach can help developers efficiently identify the test failure causes, and flag the most probable log lines of indicating the root causes for investigation. Our approach has three main stages: log abstraction, lookup table construction, and failure causes prediction. We first perform log abstraction to parse the unstructured log messages into structured log events. NCChecker then automatically maintains and updates a lookup table via employing our heuristic rules, which record the matching score between different log events and test failure causes. When it comes to the failure cause prediction stage, for a newly generated failed test log, NCChecker can easily infer its failed reason by checking out the associated log events' scores from the lookup table. We have developed a prototype and evaluated our tool on a real-world industrial dataset with more than 10K test logs. The extensive experiments show the promising performance of our model over a set of benchmarks. Moreover, our approach is highly efficient and memory-saving, and can successfully handle the data imbalance problem.

cs.SE↗

arXiv2024

EffiBench: Benchmarking the Efficiency of Automatically Generated Code

Code generation models have increasingly become integral to aiding software development. Although current research has thoroughly examined the correctness of the code produced by code generation models, a vital aspect that plays a pivotal role in green computing and sustainability efforts has often been neglected. This paper presents EffiBench, a benchmark with 1,000 efficiency-critical coding problems to assess the efficiency of code generated by code generation models. EffiBench contains a diverse set of LeetCode coding problems. Each problem is paired with an executable human-written canonical solution, which obtains the SOTA efficiency on the LeetCode solution leaderboard. With EffiBench, we empirically examine the ability of 42 large language models (35 open-source and 7 closed-source) to generate efficient code. Our evaluation results demonstrate that the efficiency of the code generated by LLMs is generally worse than the efficiency of human-written canonical solutions. For example, GPT-4 generated code has an average \textbf{3.12} times execution time that of the human-written canonical solutions. In the most extreme cases, the execution time and total memory usage of GPT-4 generated code are \textbf{13.89} and \textbf{43.92} times that of the canonical solutions. The source code of EffiBench is released on https://github.com/huangd1999/EffiBench. We also provide the LeaderBoard at https://huggingface.co/spaces/EffiBench/effibench-leaderboard.

cs.SE↗

arXiv2023

Comprehensive Evaluation of ChatGPT Reliability Through Multilingual Inquiries

ChatGPT is currently the most popular large language model (LLM), with over 100 million users, making a significant impact on people's lives. However, due to the presence of jailbreak vulnerabilities, ChatGPT might have negative effects on people's lives, potentially even facilitating criminal activities. Testing whether ChatGPT can cause jailbreak is crucial because it can enhance ChatGPT's security, reliability, and social responsibility. Inspired by previous research revealing the varied performance of LLMs in different language translations, we suspected that wrapping prompts in multiple languages might lead to ChatGPT jailbreak. To investigate this, we designed a study with a fuzzing testing approach to analyzing ChatGPT's cross-linguistic proficiency. Our study includes three strategies by automatically posing different formats of malicious questions to ChatGPT: (1) each malicious question involving only one language, (2) multilingual malicious questions, (3) specifying that ChatGPT responds in a language different from the prompts. In addition, we also combine our strategies by utilizing prompt injection templates to wrap the three aforementioned types of questions. We examined a total of 7,892 Q&A data points, discovering that multilingual wrapping can indeed lead to ChatGPT's jailbreak, with different wrapping methods having varying effects on jailbreak probability. Prompt injection can amplify the probability of jailbreak caused by multilingual wrapping. This work provides insights for OpenAI developers to enhance ChatGPT's support for language diversity and inclusion.

cs.SE↗

arXiv2023

Are They All Good? Studying Practitioners' Expectations on the Readability of Log Messages

Developers write logging statements to generate logs that provide run-time information for various tasks. The readability of log messages in the logging statements (i.e., the descriptive text) is rather crucial to the value of the generated logs. Immature log messages may slow down or even obstruct the process of log analysis. Despite the importance of log messages, there is still a lack of standards on what constitutes good readability in log messages and how to write them. In this paper, we conduct a series of interviews with 17 industrial practitioners to investigate their expectations on the readability of log messages. Through the interviews, we derive three aspects related to the readability of log messages, including Structure, Information, and Wording, along with several specific practices to improve each aspect. We validate our findings through a series of online questionnaire surveys and receive positive feedback from the participants. We then manually investigate the readability of log messages in large-scale open source systems and find that a large portion (38.1%) of the log messages have inadequate readability. Motivated by such observation, we further explore the potential of automatically classifying the readability of log messages using deep learning and machine learning models. We find that both deep learning and machine learning models can effectively classify the readability of log messages with a balanced accuracy above 80.0% on average. Our study provides comprehensive guidelines for composing log messages to further improve practitioners' logging practices.

cs.SE↗

arXiv2023

Did We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction

Due to the sheer size of software logs, developers rely on automated techniques for log analysis. One of the first and most important steps of automated log analysis is log abstraction, which parses the raw logs into a structured format. Prior log abstraction techniques aim to identify and abstract all the dynamic variables in logs and output a static log template for automated log analysis. However, these abstracted dynamic variables may also contain important information that is useful to different tasks in log analysis. In this paper, we investigate the characteristics of dynamic variables and their importance in practice, and explore the potential of a variable-aware log abstraction technique. Through manual investigations and surveys with practitioners, we find that different categories of dynamic variables record various information that can be important depending on the given tasks, the distinction of dynamic variables in log abstraction can further assist in log analysis. We then propose a deep learning based log abstraction approach, named VALB, which can identify different categories of dynamic variables and preserve the value of specified categories of dynamic variables along with the log templates (i.e., variable-aware log abstraction). Through the evaluation on a widely used log abstraction benchmark, we find that VALB outperforms other state-of-the-art log abstraction techniques on general log abstraction (i.e., when abstracting all the dynamic variables) and also achieves a high variable-aware log abstraction accuracy that further identifies the category of the dynamic variables. Our study highlights the potential of leveraging the important information recorded in the dynamic variables to further improve the process of log analysis.

cs.SE↗