arXiv subjects

Chris Brown

Publications and source records attributed to Chris Brown.

At least 19 recordsLinked to original sources

arXiv2026

Repository-Aware Metamorphic Relation Generation for Augmented Reality Applications using Large Language Models

Metamorphic Testing (MT) provides a promising approach for testing software without defined test oracles by specifying expected relations between inputs and outputs, instead of relying on exact outputs. For example, testing Augmented Reality (AR) applications is challenging due to dynamic interactions between virtual content, physical environments, and code, which make traditional test oracles difficult to define. However, formulating metamorphic relations (MRs) is time-consuming and burdensome. We introduce a context-aware pipeline that generates and refines MRs using repository-level context and reasoning orchestration, evaluated on a dataset of 142 mobile AR system repositories. Across three context configurations generating 14,916 candidate MRs, hierarchical context yielded the broadest coverage (7,004 MRs across 142 repositories and 5,167 class--method pairs) and lower redundancy. An agentic deliberation process then reconciled conflicting candidates---observed in 79.0% of cases---reducing duplication and selecting context-aware relations in 88.2% of outcomes. A manual oracle study shows refined relations (n = 141) are both logically valid and sufficiently concrete to be directly translated into test assertions, and a preliminary case study reveals converting generated MRs (n = 5) into executable tests can detect non-equivalent mutations in real-world code. Overall, our results show that combining repository-aware MR generation with reasoning-based refinement enables scalable construction of reliable, domain-relevant test oracles.

cs.SE↗

arXiv2026

TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation

Foundation models (FMs) can generate plausible unit tests, but determining whether those tests are correct, useful, maintainable, and worth integrating remains difficult. Generated tests must be mapped to the code they target, inserted into real projects, built, executed, measured against the baseline suite, repaired when necessary, and compared across models and generation strategies. This validation process is fragmented across build systems, test runners, coverage tools, mutation tools, static analyzers, and experiment scripts. The problem is especially important because generated tests are both code artifacts and validation artifacts: they must themselves be validated before they can be trusted as evidence about the system under test. This paper presents TestMap, an open-source infrastructure prototype that automates evidence-backed foundation-model-assisted test generation for C#/.NET repositories. TestMap supports repository analysis, source-test mapping, baseline execution, code metric collection, test smell detection, coverage measurement, mutation testing, model-guided test generation, validation, repair, and repository-specific experiment tracking. Rather than reporting only final passing tests, TestMap records the lifecycle of each generated candidate, including failed, repaired, low-impact, and evidence positive outcomes. These intermediate outcomes can reveal model limitations, missing context, repair cost, toolchain inefficiencies, or possible faults in the system under test. Using TestMap as a design case, we describe the architecture and evidence model needed to make generated tests observable, repeatable, and comparable across repositories, models, prompts, and generation strategies. We conclude with lessons learned and open challenges, including oracle and assertion quality, metric attribution, test maintainability, flakiness, execution cost, and developer acceptance.

cs.SE↗

arXiv2026

Integrating Log-Based Security Analytics in Agile Workflows: A Real-World Experience Report

Modern organizations increasingly rely on log data and monitoring signals to protect products against account takeovers and abuse, yet integrating security analytics into fast-moving Agile workflows remains challenging. While it is important to understand how security practices are developed and sustained within Agile, real-world case studies of such integrations remain scarce. This experience report provides insights on developer perceptions of an effort to integrate log-based fraud detection within an organization, known as the "Red Flag Project". A cross-functional team of eight members (including one author) iterated weekly to implement a proof-of-concept log-based system that alerts stakeholders when accounts exhibit suspicious activity patterns. Through semi-structured interviews, we investigate developer perceptions of log-based fraud detection integration-exploring their willingness to adopt the system, challenges encountered, and the overall impact on day-to-day development activities and security perceptions. Our findings highlight key lessons, mitigation techniques, and best practices for embedding security analytics into Agile workflows. We provide insights for practitioners and researchers seeking to incorporate security practices into modern development processes while maintaining both speed and resilience in software delivery.

cs.SE↗

arXiv2026

From Papers to Progress: Rethinking Knowledge Accumulation in Software Engineering

Software engineering research has experienced rapid growth in both output and participation over the past decades. Yet concerns persist about the field's ability to accumulate, integrate, and reuse knowledge in ways that support long-term progress. To better understand how the community itself perceives these challenges, we analyze responses from the ICSE 2026 Future of Software Engineering pre-survey, which captures perspectives from 280 globally distributed and highly experienced researchers. Our analysis reveals a tension between increasing research productivity and the limited mechanisms available for synthesizing results, tracking evolving claims, and supporting cumulative understanding over time. Building on these observations, we diagnose four interrelated structural breakdowns: papers function as isolated knowledge units with claims embedded in prose; context and provenance are lost as knowledge moves through the publication pipeline; claims evolve without systematic tracking; and incentive structures favor novelty over consolidation. We argue that addressing these barriers requires rethinking the fundamental properties of research artifacts. We articulate four technology-agnostic principles for future research artifacts: structured and interpretable representations of claims and evidence; inspectable and provenance-aware documentation of methodological decisions; long-lived and reusable substrates that evolve beyond publication; and governance mechanisms that align individual incentives with collective knowledge-building goals. We discuss implications for research practice, publication norms, and community infrastructure, positioning FOSE as a venue for experimenting with alternative artifact designs that support cumulative scientific progress.

cs.SE↗

arXiv2026

Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering

Large Language Models (LLMs) have shown impressive capabilities across software engineering tasks, including question answering (QA). However, most studies and benchmarks focus on isolated functions or single-file snippets, overlooking the challenges of real-world program comprehension, which often spans multiple files and system-level dependencies. In this work, we introduce StackRepoQA, the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects. Using this dataset, we systematically evaluate two widely used LLMs (Claude 3.5 Sonnet and GPT-4o) under both direct prompting and agentic configurations. We compare baseline performance with retrieval-augmented generation methods that leverage file-level retrieval and graph-based representations of structural dependencies. Our results show that LLMs achieve moderate accuracy at baseline, with performance improving when structural signals are incorporated. Nonetheless, overall accuracy remains limited for repository-scale comprehension. The analysis reveals that high scores often result from verbatim reproduction of Stack Overflow answers rather than genuine reasoning. To our knowledge, this is the first empirical study to provide such evidence in repository-level QA. We release StackRepoQA to encourage further research into benchmarks, evaluation protocols, and augmentation strategies that disentangle memorization from reasoning, advancing LLMs as reliable tool for repository-scale program comprehension.

cs.SE↗

arXiv2026

Political and Ideological Pressure in Software Engineering Research: The Case of DEI Backlash

Political and ideological pressures shape global research. Recently, these pressures have become particularly visible in research related to diversity, equity, and inclusion (DEI). Drastic changes in national funding and governmental guidance, especially in the US, have affected the global software engineering research ecosystem. The impacts of these pressures on research are not always direct, as they operate at multiple levels. However, what is clear is that these pressures affect every field, including software engineering (SE), despite the belief that our field is politically and ideologically neutral. In this position paper, we examine cases of political and ideological pressures on the SE research ecosystem. We investigate the community's perceptions of political and ideological pressures by analyzing community survey responses and outlining case examples of DEI backlash in SE research across three levels: macro, meso, and micro. Our research shows how recent political and ideological pressures have affected SE research across these levels, and, as a result, we propose actionable steps for the community to address these issues at different levels.

cs.SE↗

arXiv2026

Human-Human-AI Triadic Programming: Uncovering the Role of AI Agent and the Value of Human Partner in Collaborative Learning

As AI assistance becomes embedded in programming practice, researchers have increasingly examined how these systems help learners generate code and work more efficiently. However, these studies often position AI as a replacement for human collaboration and overlook the social and learning-oriented aspects that emerge in collaborative programming. Our work introduces human-human-AI (HHAI) triadic programming, where an AI agent serves as an additional collaborator rather than a substitute for a human partner. Through a within-subjects study with 20 participants, we show that triadic collaboration enhances collaborative learning and social presence compared to the dyadic human-AI (HAI) baseline. In the triadic HHAI conditions, participants relied significantly less on AI-generated code in their work. This effect was strongest in the HHAI-shared condition, where participants had an increased sense of responsibility to understand AI suggestions before applying them. These findings demonstrate how triadic settings activate socially shared regulation of learning by making AI use visible and accountable to a human peer, suggesting that AI systems that augment rather than automate peer collaboration can better preserve the learning processes that collaborative programming relies on.

cs.HC↗

arXiv2025

Parse Tree Tracking Through Time for Programming Process Analysis at Scale

Background and Context: Programming process data can be utilized to understand the processes students use to write computer programming assignments. Keystroke- and line-level event logs have been used in the past in various ways, primarily in high-level descriptive statistics (e.g., timings, character deletion rate, etc). Analysis of behavior in context (e.g., how much time students spend working on loops) has been cumbersome because of our inability to automatically track high-level code representations, such as abstract syntax trees, through time and unparseable states. Objective: Our study has two goals. The first is to design the first algorithm that tracks parse tree nodes through time. Second, we utilize this algorithm to perform a partial replication study of prior work that used manual tracking of code representations, as well as other novel analyses of student programming behavior that can now be done at scale. Method: We use two algorithms presented in this paper to track parse tree nodes through time and construct tree representations for unparseable code states. We apply these algorithms to a public keystroke data from student coursework in a 2021 CS1 course and conduct analysis on the resulting parse trees. Findings: We discover newly observable statistics at scale, including that code is deleted at similar rates inside and outside of conditionals and loops, a third of commented out code is eventually restored, and that frequency with which students jump around in their code may not be indicative of struggle. Implications: The ability to track parse trees through time opens the door to understanding new dimensions of student programming, such as best practices of structural development of code over time, quantitative measurement of what syntactic constructs students struggle most with, refactoring behavior, and attention shifting within the code.

cs.SE↗

arXiv2025

Designing Conversational AI to Support Think-Aloud Practice in Technical Interview Preparation for CS Students

One challenge in technical interviews is the think-aloud process, where candidates verbalize their thought processes while solving coding tasks. Despite its importance, opportunities for structured practice remain limited. Conversational AI offers potential assistance, but limited research explores user perceptions of its role in think-aloud practice. To address this gap, we conducted a study with 17 participants using an LLM-based technical interview practice tool. Participants valued AI's role in simulation, feedback, and learning from generated examples. Key design recommendations include promoting social presence in conversational AI for technical interview simulation, providing feedback beyond verbal content analysis, and enabling crowdsourced think-aloud examples through human-AI collaboration. Beyond feature design, we examined broader considerations, including intersectional challenges and potential strategies to address them, how AI-driven interview preparation could promote equitable learning in computing careers, and the need to rethink AI's role in interview practice by suggesting a research direction that integrates human-AI collaboration.

cs.HC↗

arXiv2025

How do Software Engineering Candidates Prepare for Technical Interviews?

To obtain employment, aspiring software engineers must complete technical interviews -- a hiring process which involves candidates writing code while communicating to an audience. However, the complexities of tech interviews are difficult to prepare for and seldom faced in computing curricula. To this end, we seek to understand how candidates prepare for technical interviews, investigating the effects of preparation methods and the role of education. We distributed a survey to candidates (n = 131) actively preparing for technical interviews. Our results suggest candidates rarely train in authentic settings and courses fail to support preparation efforts -- leading to stress and unpreparedness. Based on our findings, we provide implications for stakeholders to enhance tech interview preparation for candidates pursuing software engineering roles.

cs.SE↗

arXiv2025

Exploring the Impact of Integrating UI Testing in CI/CD Workflows on GitHub

Background: User interface (UI) testing, which is used to verify the behavior of interactive elements in applications, plays an important role in software development and quality assurance. However, little is known about the adoption of UI testing frameworks in continuous integration and continuous delivery (CI/CD) workflows and their impact on open-source software development processes. Objective: We aim to investigate the current usage of popular UI testing frameworks-Selenium, Playwright and Cypress-in CI/CD pipelines among GitHub repositories. Our goal is to understand how UI testing tools are used in CI/CD processes and assess their potential impacts on open-source development activity and CI/CD workflows. Method: We propose an empirical study to examine GitHub repositories that incorporate UI testing in CI/CD workflows. Our exploratory evaluation will collect repositories that implement UI testing frameworks in configuration files for GitHub Actions workflows to inspect UI testing-related and non-UI testing-related workflows. Moreover, we further plan to collect metrics related to repository development activity and GitHub Actions workflows to conduct comparative and time series analyses exploring whether UI testing integration and usage within CI/CD processes has an impact on open-source development.

cs.SE↗

arXiv2025

Are We on the Same Page? Examining Developer Perception Alignment in Open Source Code Reviews

Code reviews are a critical aspect of open-source software (OSS) development, ensuring quality and fostering collaboration. This study examines perceptions, challenges, and biases in OSS code review processes, focusing on the perspectives of Contributors and Maintainers. Through surveys (n=289), interviews (n=23), and repository analysis (n=81), we identify key areas of alignment and disparity. While both groups share common objectives, differences emerge in priorities, e.g, with Maintainers emphasizing alignment with project goals while Contributors overestimated the value of novelty. Bias, particularly familiarity bias, disproportionately affects underrepresented groups, discouraging participation and limiting community growth. Misinterpretation of approach differences as bias further complicates reviews. Our findings underscore the need for improved documentation, better tools, and automated solutions to address delays and enhance inclusivity. This work provides actionable strategies to promote fairness and sustain the long-term innovation of OSS ecosystems.

cs.SE↗

arXiv2025

Automatic Bias Detection in Source Code Review

Bias is an inherent threat to human decision-making, including in decisions made during software development. Extensive research has demonstrated the presence of biases at various stages of the software development life-cycle. Notably, code reviews are highly susceptible to prejudice-induced biases, and individuals are often unaware of these biases as they occur. Developing methods to automatically detect these biases is crucial for addressing the associated challenges. Recent advancements in visual data analytics have shown promising results in detecting potential biases by analyzing user interaction patterns. In this project, we propose a controlled experiment to extend this approach to detect potentially biased outcomes in code reviews by observing how reviewers interact with the code. We employ the "spotlight model of attention", a cognitive framework where a reviewer's gaze is tracked to determine their focus areas on the review screen. This focus, identified through gaze tracking, serves as an indicator of the reviewer's areas of interest or concern. We plan to analyze the sequence of gaze focus using advanced sequence modeling techniques, including Markov Models, Recurrent Neural Networks (RNNs), and Conditional Random Fields (CRF). These techniques will help us identify patterns that may suggest biased interactions. We anticipate that the ability to automatically detect potentially biased interactions in code reviews will significantly reduce unnecessary push-backs, enhance operational efficiency, and foster greater diversity and inclusion in software development. This approach not only helps in identifying biases but also in creating a more equitable development environment by mitigating these biases effectively

cs.SE↗

arXiv2025

Towards Evidence-Based Tech Hiring Pipelines

Software engineers are responsible for developing, maintaining, and innovating software. To hire software engineers, organizations employ a tech hiring pipeline. This process typically consists of a series of steps to evaluate the extent to which applicants meet job requirements and can effectively contribute to a development team -- such as resume screenings and technical interviews. However, research highlights substantial flaws with current tech hiring practices -- such as bias from stress-inducing assessments. As the landscape of software engineering (SE) is dramatically changing, assessing the technical proficiency and abilities of software engineers is an increasingly crucial task to meet technological needs and demands. In this paper, we outline challenges in current hiring practices and present future directions to promote fair and evidence-based evaluations in tech hiring pipelines. Our vision aims to enhance outcomes for candidates and assessments for employers to enhance the workforce in the tech industry.

cs.SE↗

arXiv2025

Integrating DAST in Kanban and CI/CD: A Real World Security Case Study

Modern development methodologies, such as Kanban and continuous integration and continuous deployment (CI/CD), are critical for web application development -- as software products must adapt to changing requirements and deploy products to users quickly. As web application attacks and exploited vulnerabilities are rising, it is increasingly crucial to integrate security into modern development practices. Yet, the iterative and incremental nature of these processes can clash with the sequential nature of security engineering. Thus, it is challenging to adopt security practices and activities in modern development practices. Dynamic Application Security Testing (DAST) is a security practice within software development frameworks that bolsters system security. This study delves into the intersection of Agile development and DAST, exploring how a software organization attempted to integrate DAST into their Kanban workflows and CI/CD pipelines to identify and mitigate security vulnerabilities within the development process. Through an action research case study incorporating interviews among team members, this research elucidates the challenges, mitigation techniques, and best practices associated with incorporating DAST into Agile methodologies from developers' perspectives. We provide insights into integrating security practices with modern development, ensuring both speed and security in software delivery.

cs.SE↗

arXiv2025

Towards Decoding Developer Cognition in the Age of AI Assistants

Background: The increasing adoption of AI assistants in programming has led to numerous studies exploring their benefits. While developers consistently report significant productivity gains from these tools, empirical measurements often show more modest improvements. While prior research has documented self-reported experiences with AI-assisted programming tools, little to no work has been done to understand their usage patterns and the actual cognitive load imposed in practice. Objective: In this exploratory study, we aim to investigate the role AI assistants play in developer productivity. Specifically, we are interested in how developers' expertise levels influence their AI usage patterns, and how these patterns impact their actual cognitive load and productivity during development tasks. We also seek to better understand how this relates to their perceived productivity. Method: We propose a controlled observational study combining physiological measurements (EEG and eye tracking) with interaction data to examine developers' use of AI-assisted programming tools. We will recruit professional developers to complete programming tasks both with and without AI assistance while measuring their cognitive load and task completion time. Through pre- and post-task questionnaires, we will collect data on perceived productivity and cognitive load using NASA-TLX.

cs.HC↗

arXiv2024

Generative Co-Learners: Enhancing Cognitive and Social Presence of Students in Asynchronous Learning with Generative AI

Cognitive presence and social presence are crucial for a comprehensive learning experience. Despite the flexibility of asynchronous learning environments to accommodate individual schedules, the inherent constraints of asynchronous environments make augmenting cognitive and social presence particularly challenging. Students often face challenges such as a lack of timely feedback and support, a lack of non-verbal cues in communication, and a sense of isolation. To address this challenge, this paper introduces Generative Co-Learners, a system designed to leverage generative AI-powered agents, simulating co-learners supporting multimodal interactions, to improve cognitive and social presence in asynchronous learning environments. We conducted a study involving 12 student participants who used our system to engage with online programming tutorials to assess the system's effectiveness. The results show that by implementing features to support textual and visual communication and simulate an interactive learning environment with generative agents, our system enhances the cognitive and social presence in the asynchronous learning environment. These results suggest the potential to use generative AI to support students learning at scale and transform asynchronous learning into a more inclusive, engaging, and efficacious educational approach.

cs.HC↗

arXiv2024

The Evolution of Information Seeking in Software Development: Understanding the Role and Impact of AI Assistants

About 32% of a software practitioners' day involves seeking and using information to support task completion. Although the information needs of software practitioners have been studied extensively, the impact of AI-assisted tools on their needs and information-seeking behaviors remains largely unexplored. To addresses this gap, we conducted a mixed-method study to understand AI-assisted information seeking behavior of practitioners and its impact on their perceived productivity and skill development. We found that developers are increasingly using AI tools to support their information seeking, citing increased efficiency as a key benefit. Our findings also amplify caveats that come with effectively using AI tools for information seeking, especially for learning and skill development, such as the importance of foundational developer knowledge that can guide and inform the information provided by AI tools. Our efforts have implications for the effective integration of AI tools into developer workflows as information retrieval systems and learning aids.

cs.SE↗