arXiv subjects

Xin Zhou

Publications and source records attributed to Xin Zhou.

At least 109 records · Page 6Linked to original sources

arXiv2025

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Large language models (LLMs) are gaining increasing popularity in software engineering (SE) due to their unprecedented performance across various applications. These models are increasingly being utilized for a range of SE tasks, including requirements engineering and design, code analysis and generation, software maintenance, and quality assurance. As LLMs become more integral to SE, evaluating their effectiveness is crucial for understanding their potential in this field. In recent years, substantial efforts have been made to assess LLM performance in various SE tasks, resulting in the creation of several benchmarks tailored to this purpose. This paper offers a thorough review of 291 benchmarks, addressing three main aspects: what benchmarks are available, how benchmarks are constructed, and the future outlook for these benchmarks. We begin by examining SE tasks such as requirements engineering and design, coding assistant, software testing, AIOPs, software maintenance, and quality management. We then analyze the benchmarks and their development processes, highlighting the limitations of existing benchmarks. Additionally, we discuss the successes and failures of LLMs in different software tasks and explore future opportunities and challenges for SE-related benchmarks. We aim to provide a comprehensive overview of benchmark research in SE and offer insights to support the creation of more effective evaluation tools.

cs.SE↗

arXiv2025

More Than Generation: Unifying Generation and Depth Estimation via Text-to-Image Diffusion Models

Generative depth estimation methods leverage the rich visual priors stored in pre-trained text-to-image diffusion models, demonstrating astonishing zero-shot capability. However, parameter updates during training lead to catastrophic degradation in the image generation capability of the pre-trained model. We introduce MERGE, a unified model for image generation and depth estimation, starting from a fixed pre-trained text-to-image model. MERGE demonstrates that the pre-trained text-to-image model can do more than image generation, but also expand to depth estimation effortlessly. Specifically, MERGE introduces a play-and-plug framework that enables seamless switching between image generation and depth estimation modes through simple and pluggable converters. Meanwhile, we propose a Group Reuse Mechanism to encourage parameter reuse and improve the utilization of the additional learnable parameters. MERGE unleashes the powerful depth estimation capability of the pre-trained text-to-image model while preserving its original image generation ability. Compared to other unified models for image generation and depth estimation, MERGE achieves state-of-the-art performance across multiple depth estimation benchmarks. The code will be made available at https://github.com/H-EmbodVis/MERGE

cs.CV↗

arXiv2025

Beyond Surface Similarity: Evaluating LLM-Based Test Refactorings with Structural and Semantic Awareness

Large Language Models (LLMs) are increasingly used to refactor unit tests, improving readability and structure while preserving behavior. Evaluating such refactorings, however, remains difficult: metrics like CodeBLEU penalize beneficial renamings and edits, while semantic similarities overlook readability and modularity. We propose CTSES, a first step toward human-aligned evaluation of refactored tests. CTSES combines CodeBLEU, METEOR, and ROUGE-L into a composite score that balances semantics, lexical clarity, and structural alignment. Evaluated on 5,000+ refactorings from Defects4J and SF110 (GPT-4o and Mistral-Large), CTSES reduces false negatives and provides more interpretable signals than individual metrics. Our emerging results illustrate that CTSES offers a proof-of-concept for composite approaches, showing their promise in bridging automated metrics and developer judgments.

cs.SE↗

arXiv2025

Human-Aligned Code Readability Assessment with Large Language Models

Code readability is crucial for software comprehension and maintenance, yet difficult to assess at scale. Traditional static metrics often fail to capture the subjective, context-sensitive nature of human judgments. Large Language Models (LLMs) offer a scalable alternative, but their behavior as readability evaluators remains underexplored. We introduce CoReEval, the first large-scale benchmark for evaluating LLM-based code readability assessment, comprising over 1.4 million model-snippet-prompt evaluations across 10 state of the art LLMs. The benchmark spans 3 programming languages (Java, Python, CUDA), 2 code types (functional code and unit tests), 4 prompting strategies (ZSL, FSL, CoT, ToT), 9 decoding settings, and developer-guided prompts tailored to junior and senior personas. We compare LLM outputs against human annotations and a validated static model, analyzing numerical alignment (MAE, Pearson's, Spearman's) and justification quality (sentiment, aspect coverage, semantic clustering). Our findings show that developer-guided prompting grounded in human-defined readability dimensions improves alignment in structured contexts, enhances explanation quality, and enables lightweight personalization through persona framing. However, increased score variability highlights trade-offs between alignment, stability, and interpretability. CoReEval provides a robust foundation for prompt engineering, model alignment studies, and human in the loop evaluation, with applications in education, onboarding, and CI/CD pipelines where LLMs can serve as explainable, adaptable reviewers.

cs.SE↗

arXiv2025

Investigations of MWISP Clumps: 13CO Clump Source Catalog and Physical Properties

We present the first comprehensive catalogs of $^{13}$CO clumps from the Milky Way Imaging Scroll Painting (MWISP) project. By developing an equivalent global detection scheme integrated with the FacetClumps algorithm, we successfully extract 71,661 molecular clumps across a high-resolution $^{13}$CO data cube spanning 2310 deg$^2$ from the MWISP Phase I survey. To determine accurate distances, we design an automatic hierarchical distance decision method using signal regions as fundamental objects, effectively resolving the kinematic distance ambiguity problem and obtaining reliable measurements for 97.94% of the sample. Statistical analysis reveals that 65.3% of clumps are gravitationally bound, accounting for approximately 96.3% of the statistical total mass. Scaling relation analysis across multiple surveys reveals universal power-law behaviors in clump populations. Maser-associated clumps exhibit modified parameter distributions and scaling relations, revealing how active star formation alters clump dynamics and structure. These extensive catalogs establish a foundation for investigating molecular clump properties, star formation processes, and Galactic evolution.

astro-ph.GA↗

arXiv2025

An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks

Large Language Models (LLMs) and other automated techniques have been increasingly used to support software developers by generating software artifacts such as code snippets, patches, and comments. However, accurately assessing the correctness of these generated artifacts remains a significant challenge. On one hand, human evaluation provides high accuracy but is labor-intensive and lacks scalability. On the other hand, many automatic evaluation metrics are scalable and require minimal human effort, but they often fail to accurately reflect the actual correctness of generated software artifacts. In this paper, we present SE-Jury, the first evaluation metric for LLM-as-Ensemble-Judge specifically designed to accurately assess the correctness of generated software artifacts. SE-Jury first defines five distinct evaluation strategies, each implemented by an independent judge. A dynamic team selection mechanism then identifies the most appropriate subset of judges as a team to produce a final correctness score through ensembling. We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks that span three popular SE tasks: code generation, automated program repair, and code summarization. Results demonstrate that SE-Jury consistently achieves a higher correlation with human judgments, with improvements ranging from 29.6% to 140.8% over existing automatic metrics. SE-Jury reaches agreement levels with human annotators that are close to inter-annotator agreement in code generation and program repair. These findings underscore SE-Jury's potential as a scalable and reliable alternative to human evaluation in these SE tasks.

cs.SE↗

arXiv2025

A Simultaneous Synergistic Protection Mechanism in Hybrid Perovskite-Organic Multi-junctions Enables Long-Term Stable and Efficient Tandem Solar Cells

Perovskite-organic tandem solar cells (P-O TSCs) hold great promise for next-generation thin-film photovoltaics, with steadily improving power conversion efficiency (PCE). However, the development of optimal interconnecting layers (ICLs) remains one major challenge for further efficiency gains, and progress in understanding the improved long-term stability of P-O tandem configuration has been lagging. In this study, we experimentally investigate the enhanced stability of p-i-n P-O TSCs employing a simplified C60/atomic-layer-deposition (ALD) SnOx/PEDOT: PSS ICL without an additional charge recombination layer (CRL), which achieve an averaged efficiency of 25.12% and a hero efficiency of 25.5%. Our finding discovers that the recrystallization of C60, a widely used electron transport layer in perovskite photovoltaics, leads to the formation of grain boundaries during operation, which act as migration channels for the interdiffusion of halide and Ag ions. Critically, we demonstrate for the first time that the tandem device architecture, incorporating organic semiconductor layers, effectively suppresses the bi-directional ion diffusion and mitigates electrode corrosion. Thus, the P-O TSC establishes a mutual protection system: the organic layers stabilize the perovskite sub-cell by suppressing ion diffusion-induced degradation, and the perovskite layer shields the organic sub-cell from spectrally induced degradation. The simultaneous synergistic protection mechanism enables P-O TSCs to achieve exceptional long-term operational stability, retaining over 91% of their initial efficiency after 1000 hours of continuous metal-halide lamp illumination, and to exhibit minimal fatigue after 86 cycles (2067 hours) of long-term diurnal (12/12-hour) testing. These results demonstrate that tandem cells significantly outperform their single-junction counterparts in both efficiency and stability.

physics.app-ph↗

arXiv2025

Brevity is the Soul of Wit: Condensing Code Changes to Improve Commit Message Generation

Commit messages are valuable resources for describing why code changes are committed to repositories in version control systems (e.g., Git). They effectively help developers understand code changes and better perform software maintenance tasks. Unfortunately, developers often neglect to write high-quality commit messages in practice. Therefore, a growing body of work is proposed to generate commit messages automatically. These works all demonstrated that how to organize and represent code changes is vital in generating good commit messages, including the use of fine-grained graphs or embeddings to better represent code changes. In this study, we choose an alternative way to condense code changes before generation, i.e., proposing brief yet concise text templates consisting of the following three parts: (1) summarized code changes, (2) elicited comments, and (3) emphasized code identifiers. Specifically, we first condense code changes by using our proposed templates with the help of a heuristic-based tool named ChangeScribe, and then fine-tune CodeLlama-7B on the pairs of our proposed templates and corresponding commit messages. Our proposed templates better utilize pre-trained language models, while being naturally brief and readable to complement generated commit messages for developers. Our evaluation based on a widely used dataset showed that our approach can outperform six baselines in terms of BLEU-Norm, METEOR, and ROUGE-L, with average improvements of 51.7%, 78.7%, and 62.5%, respectively. The ablation study and human evaluation also provide further insights into the effectiveness of our approach.

cs.SE↗

arXiv2025

ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledge

We introduce ESGenius, a comprehensive benchmark for evaluating and enhancing the proficiency of Large Language Models (LLMs) in Environmental, Social, and Governance (ESG) and sustainability-focused question answering. ESGenius comprises two key components: (i) ESGenius-QA, a collection of 1,136 Multiple-Choice Questions (MCQs) generated by LLMs and rigorously validated by domain experts, covering a broad range of ESG pillars and sustainability topics. Each question is systematically linked to its corresponding source text, enabling transparent evaluation and supporting Retrieval-Augmented Generation (RAG) methods; and (ii) ESGenius-Corpus, a meticulously curated repository of 231 foundational frameworks, standards, reports, and recommendation documents from 7 authoritative sources. Moreover, to fully assess the capabilities and adaptation potential of LLMs, we implement a rigorous two-stage evaluation protocol -- Zero-Shot and RAG. Extensive experiments across 50 LLMs (0.5B to 671B) demonstrate that state-of-the-art models achieve only moderate performance in zero-shot settings, with accuracies around 55--70%, highlighting a significant knowledge gap for LLMs in this specialized, interdisciplinary domain. However, models employing RAG demonstrate significant performance improvements, particularly for smaller models. For example, DeepSeek-R1-Distill-Qwen-14B improves from 63.82% (zero-shot) to 80.46% with RAG. These results demonstrate the necessity of grounding responses in authoritative sources for enhanced ESG understanding. To the best of our knowledge, ESGenius is the first comprehensive QA benchmark designed to rigorously evaluate LLMs on ESG and sustainability knowledge, providing a critical tool to advance trustworthy AI in this vital domain.

cs.CL↗

arXiv2025

VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems

Visual documentation is an effective tool for reducing the cognitive barrier developers face when understanding unfamiliar code, enabling more intuitive comprehension. Compared to textual documentation, it provides a higher-level understanding of the system structure and data flow. Developers usually prefer visual representations over lengthy textual descriptions for large software systems. Visual documentation is both difficult to produce and challenging to evaluate. Manually creating it is time-consuming, and currently, no existing approach can automatically generate high-level visual documentation directly from code. Its evaluation is often subjective, making it difficult to standardize and automate. To address these challenges, this paper presents the first exploration of using agentic LLM systems to automatically generate visual documentation. We introduce VisDocSketcher, the first agent-based approach that combines static analysis with LLM agents to identify key elements in the code and produce corresponding visual representations. We propose a novel evaluation framework, AutoSketchEval, for assessing the quality of generated visual documentation using code-level metrics. The experimental results show that our approach can valid visual documentation for 74.4% of the samples. It shows an improvement of 26.7-39.8% over a simple template-based baseline. Our evaluation framework can reliably distinguish high-quality (code-aligned) visual documentation from low-quality (non-aligned) ones, achieving an AUC exceeding 0.87. Our work lays the foundation for future research on automated visual documentation by introducing practical tools that not only generate valid visual representations but also reliably assess their quality.

cs.SE↗

arXiv2025

Contrastive Network Representation Learning

Network representation learning seeks to embed networks into a low-dimensional space while preserving the structural and semantic properties, thereby facilitating downstream tasks such as classification, trait prediction, edge identification, and community detection. Motivated by challenges in brain connectivity data analysis that is characterized by subject-specific, high-dimensional, and sparse networks that lack node or edge covariates, we propose a novel contrastive learning-based statistical approach for network edge embedding, which we name as Adaptive Contrastive Edge Representation Learning (ACERL). It builds on two key components: contrastive learning of augmented network pairs, and a data-driven adaptive random masking mechanism. We establish the non-asymptotic error bounds, and show that our method achieves the minimax optimal convergence rate for edge representation learning. We further demonstrate the applicability of the learned representation in multiple downstream tasks, including network classification, important edge detection, and community detection, and establish the corresponding theoretical guarantees. We validate our method through both synthetic data and real brain connectivities studies, and show its competitive performance compared to the baseline method of sparse principal components analysis.

stat.ML↗

arXiv2025

GeV Gamma-Rays from Molecular Clouds Illuminated by Particles Diffusing from the Adjacent Supernova Remnant G335.2+0.1 Confined in an Expanding Bubble

We report the detection of GeV gamma-ray emission likely associated with supernova remnant (SNR) G335.2+0.1 and the finding of a molecular cloud ($\sim20$--$30^\prime$ in angular size) that is very likely in physical contact with the SNR and responsible for the gamma-ray emission. Using the 16.8 yr Fermi-LAT data, an extended emission, with a significance of 13.5 $σ$ and a radius 0.24° in 0.2--500 GeV in the uniform-disk model, was found to the adjacent east of the SNR. With archival Mopra CO-line data, a large molecular clump at local-standard-of-rest velocity $\sim-48$ to $-43$ km s$^{-1}$ was revealed appearing coincident with the gamma-ray source. The SNR was found located in a cavity encircled by a 'C'-shaped ring-like molecular shell at $-45$ to $-43$ km s$^{-1}$. This morphological agreement, together with the position-velociy diagrams made along lines cutting across the cavity, suggests that the SNR was evolving in the expanding molecular bubble created by the stellar wind of the progenitor with a mass $\gtrsim 15 M_{\mathrm{sun}}$. The giant molecular cloud, visible at around $-46$ km s$^{-1}$, and the associated SNR are thus estimated to lie at a kinematic distance of 3.1 kpc, with the HI absorption taken into account. We suggest that the SNR has entered the radiative phase after the blastwave recently struck the cavity wall. With the evolutionary scenario of the SNR, we demonstrate that the gamma-ray emission reported here can be naturally interpreted by the hadronic interaction between the accelerated protons that escaped from the SNR shock and the eastern large molecular clump.

astro-ph.HE↗

arXiv2025

FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce $\textbf{FutureX}$, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.

cs.AI↗

arXiv2025

Dynamically encircling an exceptional point through phase-tracked closed-loop control

The intricate complex eigenvalues of non-Hermitian Hamiltonians manifest as Riemann surfaces in control parameter spaces. At the exceptional points (EPs), the degeneracy of both eigenvalues and eigenvectors introduces noteworthy topological features, particularly during the encirclement of the EPs. Traditional methods for probing the state information on the Riemann surfaces involve static measurements; however, realizing continuous encircling remains a formidable challenge due to non-adiabatic transitions that disrupt the transport paths. Here we propose an approach leveraging the phase-locked loop (PLL) technique to facilitate smooth, dynamic encircling of EPs while maintaining resonance. Our methodology strategically ties the excitation frequencies of steady states to their response phases, enabling controlled traversal along the Riemann surfaces of real eigenvalues. This study advances the concept of phase-tracked dynamical encircling and explores its practical implementation within a fully electrically controlled non-Hermitian microelectromechanical system, highlighting robust in-situ tunability and providing methods for exploring non-Hermitian topologies.

quant-ph↗

arXiv2025

Curiosity-Driven Testing for Sequential Decision-Making Process

Sequential decision-making processes (SDPs) are fundamental for complex real-world challenges, such as autonomous driving, robotic control, and traffic management. While recent advances in Deep Learning (DL) have led to mature solutions for solving these complex problems, SDMs remain vulnerable to learning unsafe behaviors, posing significant risks in safety-critical applications. However, developing a testing framework for SDMs that can identify a diverse set of crash-triggering scenarios remains an open challenge. To address this, we propose CureFuzz, a novel curiosity-driven black-box fuzz testing approach for SDMs. CureFuzz proposes a curiosity mechanism that allows a fuzzer to effectively explore novel and diverse scenarios, leading to improved detection of crashtriggering scenarios. Additionally, we introduce a multi-objective seed selection technique to balance the exploration of novel scenarios and the generation of crash-triggering scenarios, thereby optimizing the fuzzing process. We evaluate CureFuzz on various SDMs and experimental results demonstrate that CureFuzz outperforms the state-of-the-art method by a substantial margin in the total number of faults and distinct types of crash-triggering scenarios. We also demonstrate that the crash-triggering scenarios found by CureFuzz can repair SDMs, highlighting CureFuzz as a valuable tool for testing SDMs and optimizing their performance.

cs.SE↗

arXiv2025

Nonreciprocal Entanglement by Dynamically Encircling a Nexus

Nonreciprocal entanglement, characterized by inherently robust operation, is a cornerstone for quantum information processing and communications. However, it remains a great challenge to achieve nonreciprocal entanglement characterized by stability and robustness against environmental fluctuations. Here, we propose a universal nonlinear mechanism to engineer magnetic-free nonreciprocity in dissipative optomechanics by utilizing bistability, a phenomenon ubiquitous across nonlinear physical systems. By dynamically encircling the nexus of bistability, a cusp converged by the bistable surfaces, we obtain nonreciprocal displacement and then utilize it to achieve robust nonreciprocal entanglement. Owing to the unique landscape of bistability, our nonreciprocal displacement and entanglements exhibit stability and robustness through closed-loop operations. Our work presents a foundational framework for leveraging nonlinearity to achieve nonreciprocal quantum information processing. It paves new avenues for exploring nonreciprocal quantum information processing and designing backaction-immune quantum metrology with nonlinearity.

quant-ph↗

arXiv2025

TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling

Recent advancements in aligning large language models via reinforcement learning have achieved remarkable gains in solving complex reasoning problems, but at the cost of expensive on-policy rollouts and limited exploration of diverse reasoning paths. In this work, we introduce TreePO, involving a self-guided rollout algorithm that views sequence generation as a tree-structured searching process. Composed of dynamic tree sampling policy and fixed-length segment decoding, TreePO leverages local uncertainty to warrant additional branches. By amortizing computation across common prefixes and pruning low-value paths early, TreePO essentially reduces the per-update compute burden while preserving or enhancing exploration diversity. Key contributions include: (1) a segment-wise sampling algorithm that alleviates the KV cache burden through contiguous segments and spawns new branches along with an early-stop mechanism; (2) a tree-based segment-level advantage estimation that considers both global and local proximal policy optimization. and (3) analysis on the effectiveness of probability and quality-driven dynamic divergence and fallback strategy. We empirically validate the performance gain of TreePO on a set reasoning benchmarks and the efficiency saving of GPU hours from 22\% up to 43\% of the sampling design for the trained models, meanwhile showing up to 40\% reduction at trajectory-level and 35\% at token-level sampling compute for the existing models. While offering a free lunch of inference efficiency, TreePO reveals a practical path toward scaling RL-based post-training with fewer samples and less compute. Home page locates at https://m-a-p.ai/TreePO.

cs.LG↗

arXiv2025

Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception

Automatically generated unit tests-from search-based tools like EvoSuite or LLMs-vary significantly in structure and readability. Yet most evaluations rely on metrics like Cyclomatic Complexity and Cognitive Complexity, designed for functional code rather than test code. Recent studies have shown that SonarSource's Cognitive Complexity metric assigns near-zero scores to LLM-generated tests, yet its behavior on EvoSuite-generated tests and its applicability to test-specific code structures remain unexplored. We introduce CCTR, a Test-Aware Cognitive Complexity metric tailored for unit tests. CCTR integrates structural and semantic features like assertion density, annotation roles, and test composition patterns-dimensions ignored by traditional complexity models but critical for understanding test code. We evaluate 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large-1024 across 350 classes from Defects4J and SF110. Results show CCTR effectively discriminates between structured and fragmented test suites, producing interpretable scores that better reflect developer-perceived effort. By bridging structural analysis and test readability, CCTR provides a foundation for more reliable evaluation and improvement of generated tests. We publicly release all data, prompts, and evaluation scripts to support replication.

cs.SE↗