arXiv subjects

Zhiyu Fan

Publications and source records attributed to Zhiyu Fan.

12 recordsLinked to original sources

arXiv2026

Can Large Language Models Generate Observability-Aware Code?

Recent advances in coding agents have enabled the generation of increasingly complex software systems. While existing evaluations primarily focus on functional correctness, production systems must expose failure evidence to support observability. In this paper, we present a systematic study of observability in agent-generated systems. We examine whether agents can reconstruct source-level diagnostic semantics by restoring observability artifacts in 10 open-source and 8 industrial repositories. We also evaluate whether these artifacts translate into effective fault signals at runtime through 200 generated microservice systems deployed on Kubernetes with 13 injected faults. Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals (i.e., explicit, fault-specific evidence) at runtime. At the source level, agents partially recover observability artifacts but struggle to capture key diagnostic semantics. At runtime, generated systems expose fault signals for only a small fraction of failures (up to 13.99\%), despite the presence of logging, suggesting that the generated observability artifacts may lack the failure-specific semantics needed to effectively expose faults. We further introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure, but the gains remain limited, indicating that the gap is not easily addressed. More broadly, our findings suggest that current evaluations focusing primarily on functional correctness may overlook observability as an important dimension of practical software quality.

cs.SE↗

arXiv2026

Understanding the (In)Security of Vibe-Coded Applications

Recent advances in large language models (LLMs) have enabled vibe coding, an emerging software development paradigm in which users create applications primarily through natural-language interactions with AI agents. Due to its low barrier to entry, vibe coding is rapidly gaining adoption in practice. Unlike conventional AI-assisted programming, where developers remain responsible for implementation and code review, vibe coding delegates a substantial portion of the development process to AI systems. This shift raises a fundamental question: how (in)secure are applications developed through vibe coding? In this paper, we conduct a systematic study of the security of real-world vibe-coded applications. We collect 9,041 open-source applications developed using popular AI agents (Claude Code and Lovable), and audit 200 publicly deployed applications, uncovering 1,186 vulnerabilities. Our study of these applications and vulnerabilities reveals several key findings: (1) insecurity is the norm rather than the exception: 91.0\% of audited applications contain at least one vulnerability, and 65.77\% of identified vulnerabilities are rated Critical or High severity, concentrated in broken access control, injection, and authentication failures; (2) these vulnerabilities are traceable to eight recurring failure modes rooted in three systematic limitations of AI agents: memory defects, objective defects, and knowledge defects; and (3) while improved agent harness and prompting strategies can reduce the incidence of vulnerabilities, they do not eliminate the underlying security risks. Overall, our study provides an empirical understanding of the security landscape of vibe-coded applications and lays the groundwork for addressing the security risks in the growing delegation of software development to AI systems.

cs.CR↗

arXiv2025

A universal description of Mott insulators: Characterizing quantum phases beyond broken symmetries

Using Mott insulators as a prototypical example, we demonstrate a dynamics-based characterization of quantum phases of matter through a general N-body renormalization group framework. The essential "Mott-ness" turns out to be characterized by a change of size-scaling of the effective intra- momentum repulsions between long-lived emergent "eigen-particles" that encodes the dynamics of two-body bound states in the high-energy sector. This directly offers a universal characterization at long space-time scale for the corresponding class of Mott insulators through a uniform single occupation of all momenta, and otherwise Mott metals. This universal description naturally paves the way to topological Mott insulators and is straightforward to extend to bosonic Mott systems. More generally, this demonstration exemplifies a generic paradigm of characterizing quantum phases of matter through their distinct dynamics beyond broken symmetries.

cond-mat.str-el↗

arXiv2025

Bogoliubov quasi-particles in superconductors are integer-charged particles inapplicable for braiding quantum information

We present a rigorous proof that under a number-conserving Hamiltonian, one-body quasi-particles generally possess quantized charge and inertial mass identical to the bare particles. It follows that, Bogoliubov zero modes in the vortex (or on the edge) of superconductors $\textit{cannot}$ be their own anti-particles capable of braiding quantum information. As such, the heavily pursued Majorana zero mode-based route for quantum computation requires a serious re-consideration. This study further reveals the conceptual challenge in preparing and manipulating braid-able quantum states via physical thermalization or slow external fields. These profound results should reignite the long-standing quest for a number-conserving theory of superconductivity and superfluidity without fictitiously breaking global U(1) symmetry.

cond-mat.str-el↗

arXiv2025

SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints

The advancement of large language models (LLMs) and code agents has demonstrated significant potential to assist software engineering (SWE) tasks, such as autonomous issue resolution and feature addition. Existing AI for software engineering leaderboards (e.g., SWE-bench) focus solely on solution accuracy, ignoring the crucial factor of effectiveness in a resource-constrained world. This is a universal problem that also exists beyond software engineering tasks: any AI system should be more than correct - it must also be cost-effective. To address this gap, we introduce SWE-Effi, a set of new metrics to re-evaluate AI systems in terms of holistic effectiveness scores. We define effectiveness as the balance between the accuracy of outcome (e.g., issue resolve rate) and the resources consumed (e.g., token and time). In this paper, we specifically focus on the software engineering scenario by re-ranking popular AI systems for issue resolution on a subset of the SWE-bench benchmark using our new multi-dimensional metrics. We found that AI system's effectiveness depends not just on the scaffold itself, but on how well it integrates with the base model, which is key to achieving strong performance in a resource-efficient manner. We also identified systematic challenges such as the "token snowball" effect and, more significantly, a pattern of "expensive failures". In these cases, agents consume excessive resources while stuck on unsolvable tasks - an issue that not only limits practical deployment but also drives up the cost of failed rollouts during RL training. Lastly, we observed a clear trade-off between effectiveness under the token budget and effectiveness under the time budget, which plays a crucial role in managing project budgets and enabling scalable reinforcement learning, where fast responses are essential.

cs.SE↗

arXiv2025

Pressure-induced trans-proximate correlation in La$_4$Ni$_3$O$_{10}$ and possible routes to enhance its superconductivity

We report an unexpected trans-proximate interlayer correlation (stronger correlation between disjoint layers than the adjacent ones) in the high-pressure phase of the recently discovered La$_4$Ni$_3$O$_{10}$ superconductors. Accompanied by an unusual pressure-induced fractionalization of Ni$^{2+}$ ionic spin from the standard spin-1 to spin-$\frac{1}{2}$, this trans-proximate correlation results from the emergence of a cross-layer trimer in our multi-energy-scale derivation of the electron dynamics. The resulting low-energy effective description resembles that of the cuprates and suggests a universal superconducting mechanism in all existing nickelate and cuprate superconductors. The rare trans-proximate correlation not only explains the weaker superconductivity in comparison with the related La$_3$Ni$_2$O$_7$ samples, but it also indicates a viable strategy to improve superconductivity in this trilayer nickelate by lowering layer symmetry. Such pressure-induced trans-proximate correlation is expected in many materials and examplifies the engineering of rich uncharted quantum states of matter through pressure.

cond-mat.supr-con↗

arXiv2024

Codexity: Secure AI-assisted Code Generation

Despite the impressive performance of Large Language Models (LLMs) in software development activities, recent studies show the concern of introducing vulnerabilities into software codebase by AI programming assistants (e.g., Copilot, CodeWhisperer). In this work, we present Codexity, a security-focused code generation framework integrated with five LLMs. Codexity leverages the feedback of static analysis tools such as Infer and CppCheck to mitigate security vulnerabilities in LLM-generated programs. Our evaluation in a real-world benchmark with 751 automatically generated vulnerable subjects demonstrates Codexity can prevent 60% of the vulnerabilities being exposed to the software developer.

cs.SE↗

arXiv2024

AutoCodeRover: Autonomous Program Improvement

Researchers have made significant progress in automating the software development process in the past decades. Recent progress in Large Language Models (LLMs) has significantly impacted the development process, where developers can use LLM-based programming assistants to achieve automated coding. Nevertheless, software engineering involves the process of program improvement apart from coding, specifically to enable software maintenance (e.g. bug fixing) and software evolution (e.g. feature additions). In this paper, we propose an automated approach for solving GitHub issues to autonomously achieve program improvement. In our approach called AutoCodeRover, LLMs are combined with sophisticated code search capabilities, ultimately leading to a program modification or patch. In contrast to recent LLM agent approaches from AI researchers and practitioners, our outlook is more software engineering oriented. We work on a program representation (abstract syntax tree) as opposed to viewing a software project as a mere collection of files. Our code search exploits the program structure in the form of classes/methods to enhance LLM's understanding of the issue's root cause, and effectively retrieve a context via iterative search. The use of spectrum-based fault localization using tests, further sharpens the context, as long as a test-suite is available. Experiments on SWE-bench-lite (300 real-life GitHub issues) show increased efficacy in solving GitHub issues (19% on SWE-bench-lite), which is higher than the efficacy of the recently reported SWE-agent. In addition, AutoCodeRover achieved this efficacy with significantly lower cost (on average, $0.43 USD), compared to other baselines. We posit that our workflow enables autonomous software engineering, where, in future, auto-generated code from LLMs can be autonomously improved.

cs.SE↗

arXiv2024

Pressure Driven Fractionalization of Ionic Spins Results in Cupratelike High-$T_c$ Superconductivity in La$_3$Ni$_2$O$_7$

Beyond 14GPa of pressure, bi-layered La$_3$Ni$_2$O$_7$ was recently found to develop strong superconductivity above the liquid nitrogen boiling temperature. An immediate essential question is the pressure-induced qualitative change of electronic structure that enables the exciting high-temperature superconductivity. We investigate this timely question via a numerical multi-scale derivation of effective many-body physics. At the atomic scale, we first clarify that the system has a strong charge transfer nature with itinerant carriers residing mainly in the in-plane oxygen between spin-1 Ni$^{2+}$ ions. We then elucidate in eV- and sub-eV-scale the key physical effect of the applied pressure: It induces a cupratelike electronic structure through partially screening the Ni spin from 1 to 1/2. This suggests a high-temperature superconductivity in La$_3$Ni$_2$O$_7$ with microscopic mechanism and ($d$-wave) symmetry similar to that in the cuprates.

cond-mat.supr-con↗

arXiv2023

Software Engineering Educational Experience in Building an Intelligent Tutoring System

The growing number of students enrolling in Computer Science (CS) programmes is pushing CS educators to their limits. This poses significant challenges to computing education, particularly the teaching of introductory programming and advanced software engineering (SE) courses. First-year programming courses often face overwhelming enrollments, including interdisciplinary students who are not CS majors. The high teacher-to-student ratio makes it challenging to provide timely and high-quality feedback. Meanwhile, software engineering education comes with inherent difficulties like acquiring industry partners and the dilemma that such software projects are often under or over-specified and one-time efforts within one team or one course. To address these challenges, we designed a novel foundational SE course. This SE course envisions building a full-fledged Intelligent Tutoring System (ITS) of Programming Assignments to provide automated, real-time feedback for novice students in programming courses over multiple years. Each year, SE students contribute to specific short-running SE projects that improve the existing ITS implementation, while at the same time, we can deploy the ITS for usage by students for learning programming. This project setup builds awareness among SE students about their contribution to a "to-be-deployed" software project. In this multi-year teaching effort, we have incrementally built an ITS that is now deployed in various programming courses. This paper discusses the Intelligent Tutoring System architecture, our teaching concept in the SE course, our experience with the built ITS, and our view of future computing education.

cs.SE↗

arXiv2023

Automated Repair of Programs from Large Language Models

Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.

cs.SE↗

arXiv2021

Verifix: Verified Repair of Programming Assignments

Automated feedback generation for introductory programming assignments is useful for programming education. Most works try to generate feedback to correct a student program by comparing its behavior with an instructor's reference program on selected tests. In this work, our aim is to generate verifiably correct program repairs as student feedback. The student assignment is aligned and composed with a reference solution in terms of control flow, and differences in data variables are automatically summarized via predicates to relate the variable names. Failed verification attempts for the equivalence of the two programs are exploited to obtain a collection of maxSMT queries, whose solutions point to repairs of the student assignment. We have conducted experiments on student assignments curated from a widely deployed intelligent tutoring system. Our results indicate that we can generate verified feedback in up to 58% of the assignments. More importantly, our system indicates when it is able to generate a verified feedback, which is then usable by novice students with high confidence.

cs.SE↗