arXiv subjects

Suzhen Zhong

Publications and source records attributed to Suzhen Zhong.

3 recordsLinked to original sources

arXiv2026

Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality

Large Language Models (LLMs) are increasingly used by developers for conversational coding assistance, including code generation, technical questions, and iterative problem solving. However, limited empirical evidence exists on the topics developers discuss with LLMs, how interaction patterns vary across tasks, and how the quality of generated code changes across conversational turns. To address this, we construct CodeChat, a large-scale dataset derived from WildChat. CodeChat contains 587,568 real-world developer-LLM conversations and 1.7 million code snippets across more than 20 programming languages. We identify the topics developers raise and examine how their engagement varies across topics. We then assess the quality of generated code across five programming languages (i.e., Python, JavaScript, C++, Java, and C#) and analyze how quality issues evolve across conversational turns. We find that developers most frequently seek help with web design (9.6% of conversations) and machine learning (8.7%), and that conversations often grow long because developers repeatedly shift use cases. LLM-generated code exhibits prevalent, language-specific issues whose prevalence does not consistently decrease in interactions lasting at least five turns. Among successful syntax-error resolutions, prompts that explicitly identify the error and request a fix are the most frequently observed. Our findings suggest that conversational assistants should track evolving developer intent and monitor code quality across turns.

cs.SE↗

arXiv2026

From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

Code review helps maintain software quality before code integration, but it also imposes a substantial workload on human reviewers. As generative artificial intelligence becomes part of software development, code review is shifting from a primarily human review process toward AI-supported review processes in which large language model (LLM) reviewers and AI agent reviewers participate alongside human reviewers. However, we still lack empirical evidence on how this transition affects review efficiency and review quality. In this paper, we study 1.02 million reviewed pull requests from 207 GitHub projects that transition across three code review eras: human-centric review, LLM-assisted review, and agentic code review. We identify three AI reviewer adoption practices: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption. We further model pull request review discussions as reviewer interaction sequences to characterize how human, LLM, and AI agent reviewers collaborate during the review process. Our results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption. However, these efficiency gains do not translate into better review quality. We also find that review activity and pull request type remain important across eras, while human-AI collaboration patterns become the strongest explanatory factor for review efficiency once LLM and AI agent reviewers participate. These findings provide empirical guidance for designing AI-supported code review processes that improve efficiency without weakening review quality.

cs.SE↗

arXiv2026

Human-AI Synergy in Agentic Code Review

Code review is a critical software engineering practice where developers review code changes before integration to ensure code quality, detect defects, and improve maintainability. In recent years, AI agents that can understand code context, plan review actions, and interact with development environments have been increasingly integrated into the code review process. However, there is limited empirical evidence to compare the effectiveness of AI agents and human reviewers in collaborative workflows. To address this gap, we conduct a large-scale empirical analysis of 278,790 code review conversations across 300 open-source GitHub projects. In our study, we aim to compare the feedback differences provided by human reviewers and AI agents. We investigate human-AI collaboration patterns in review conversations to understand how interaction shapes review outcomes. Moreover, we analyze the adoption of code suggestions provided by human reviewers and AI agents into the codebase and how adopted suggestions change code quality. We find that human reviewers provide additional feedback than AI agents, including understanding, testing, and knowledge transfer. Human reviewers exchange 11.8% more rounds when reviewing AI-generated code than human-written code. Moreover, code suggestions made by AI agents are adopted into the codebase at a significantly lower rate than suggestions proposed by human reviewers. Over half of unadopted suggestions from AI agents are either incorrect or addressed through alternative fixes by developers. When adopted, suggestions provided by AI agents produce significantly larger increases in code complexity and code size than suggestions provided by human reviewers. Our findings suggest that while AI agents can scale defect screening, human oversight remains critical for ensuring suggestion quality and providing contextual feedback that AI agents lack.

cs.SE↗