Searcharxiv⌕ Search

arXiv subjects

John Grundy

Publications and source records attributed to John Grundy.

At least 19 recordsLinked to original sources

EmpathiSEr: Validated Scales for Measuring Empathy Towards Software Engineering Practitioners and Users

Empathy plays a critical role in software engineering (SE), for instance in situations where developers interpret non-technical users' frustration with system usability or where product owners account for the technical constraints experienced by engineers during implementation. Such interactions shape collaboration, communication, and user-centred design outcomes. Although SE research has increasingly recognised empathy as a key human aspect, there remains no validated instrument specifically designed to measure it within the unique socio-technical contexts of SE. Existing generic empathy scales, while well-established in psychology and healthcare, often rely on language, scenarios, and assumptions that are not meaningful or interpretable for software practitioners. These scales fail to account for the diverse, role-specific, and domain-bound expressions of empathy in SE, such as understanding a non-technical user's frustrations or another practitioner's technical constraints, which differ substantially from empathy in clinical or everyday contexts. To address this gap, we developed and validated two domain-specific empathy scales: EmpathiSEr-P, assessing empathy among practitioners, and EmpathiSEr-U, capturing practitioner empathy towards users. Grounded in a practitioner-informed conceptual framework, the scales encompass three dimensions of empathy: cognitive empathy, affective empathy, and empathic responses. We followed a rigorous, multi-phase methodology, including expert evaluation, cognitive interviews, and two practitioner surveys. The resulting instruments represent the first psychometrically validated empathy scales tailored to SE, offering researchers and practitioners a tool for assessing empathy and designing empathy-enhancing interventions in software teams and user interactions.

cs.SE↗

Identifying and Mitigating API Misuse in Large Language Models

API misuse in code generated by large language models (LLMs) presents a serious and growing challenge in software development, as although LLMs demonstrate impressive code generation capabilities, their interactions with complex library APIs are often error-prone and can lead to software failures and vulnerabilities. In this paper, we conduct a large-scale study of API misuse patterns in LLM-generated code by analyzing both method selection and parameter usage across Python and Java, using three representative LLMs: StarCoder-7B, Qwen2.5-Coder-7B, and GitHub Copilot. Based on extensive manual annotation of 3,209 method-level and 3,492 parameter-level misuses, we identify and categorize four recurring misuse types by building on and refining prior API misuse taxonomies. Our evaluation of the three LLMs reveals persistent challenges in API usage, particularly hallucination and intent misalignment. To address these issues, we propose Dr.Fix, an LLM-based automatic repair approach guided by our taxonomy, which improves repair accuracy compared to baseline prompting and existing repair methods, achieving gains of up to 38.4 BLEU and up to 72 more exact matches on benchmark datasets. This work offers important insights into the current limitations of LLMs in API usage and points to directions for improving automated misuse repair in code generation systems.

cs.SE↗

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.

cs.SE↗

From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines

Retrieval-Augmented Generation (RAG) pipelines are widely used for domain-specific question answering, yet locating faults remains manual and time-consuming. Fault localization is challenging because RAG pipelines consist of interdependent stages, where faults can propagate downstream, masking their origin, or remain undetected even when the final answer is correct. Existing evaluation approaches often focus on individual stages and rely on reference answers or code instrumentation. We present RAGLog, which detects and localizes faults across RAG stages using execution logs, without requiring reference answers or code instrumentation. RAGLog transforms heterogeneous API logs into a common RAG schema and combines rule-based, embedding-based, and LLM-based checks to identify faulty stages. We introduce a live fault injector and RAGLog Faults, a dataset of 3,134 runs from five RAG pipelines with stage-level fault labels. With Claude Haiku 5.5, RAGLog achieves a pooled F1 score of 0.73 on runs with naturally occurring faults, outperforming Claude Code (0.58) and Codex (0.52), and comparable effectiveness on runs with injected faults (0.84 versus 0.80 and 0.82, respectively). It analyzes runs in approximately five seconds at US\$0.005 each, with overall lower cost and latency than agent baselines. RAGLog is available as an open-source tool for debugging, evaluating, and monitoring RAG pipelines.

cs.AI↗

Monitoring Machine Learning Systems in Production: A Multivocal Literature Review

Monitoring is essential for maintaining reliable machine learning (ML) systems after deployment. However, knowledge about its practical implementation and effectiveness remains fragmented across academic and gray literature. We present a multivocal literature review of 94 sources (37 academic and 57 gray) covering both traditional ML and generative AI systems. We develop taxonomies covering monitored objects and properties, techniques, input signals, tools, automation, frequency, evaluation, and challenges. Our findings reveal an imbalanced monitoring process. Existing approaches provide broad and largely automated support for observing system behavior, detecting problems, and communicating findings, while monitor configuration, investigation, and response retain substantial human involvement. Monitoring is concentrated on inference, with no source covering the entire production ML system, limiting end-to-end diagnosis. Evidence of effectiveness is also limited: among 78 sources presenting an approach or tool, 45 provide no formal evaluation, 24 report offline or controlled evaluations, and only nine evaluate monitoring in live production. This gap is particularly evident for generative AI, where evidence is dominated by gray literature and vendor claims. We propose six directions for improving monitor design, cross-layer diagnosis, proxy validation, corrective responses, production-oriented evaluation, and organizational practices. Our review provides a structured basis for improving the detection, diagnosis, and resolution of production ML problems.

cs.SE↗

Developer Task Delegation to LLMs and the Trust Factor: A Mixed Methods Study

With the rise of Large Language Models (LLMs), software developers are presented with a powerful new approach: delegating software development tasks to LLMs. Existing studies have explored task delegation for a subset of tasks and factors. Critically, what tasks developers delegate to LLMs, to what extent, and why they delegate them, is not yet well understood. We explore these questions through 22 interviews and a survey with 160 software practitioners. Developers reported employing four modes of task delegation to LLMs: full delegation, partial delegation, self-delegation, and task avoidance. Drawing on our qualitative findings, we discuss the rationales behind these choices and explore how task difficulty, task risk, developer motivation, and trust appear to be associated with software development task delegation to LLMs. Our statistical analysis reveals that trust is a significant factor associated with task delegation to LLMs. Finally, we share actionable implications for human-AI collaboration in software research, and for software practitioners to make the most of their newly found powers of delegation.

cs.SE↗

AI Policies: Help or Hindrance? A Software Developer's Perspective

AI policies introduced by software organisations to mitigate LLM-related risks such as sensitive information leaks and unauthorised usage are not useful if software developers do not engage with them. We draw on 19 software developer interviews to show how AI policies help and hinder developers. We suggest approaches to support managers and decision makers with a developer-centric approach to introducing AI policies.

cs.SE↗

Human-Centred Requirements Engineering for Critical Systems: Insights from Disaster Early Warning Applications

Critical systems, such as those used in healthcare, defence, transportation, and disaster management, require rigorous requirements engineering to ensure safety and reliability. However, this rigour has traditionally focused on technical assurance, with less attention to the human and social contexts in which these systems are used. This paper argues that human-centricity is an essential dimension of dependability and presents a human-centred requirements engineering process for making vulnerable-user needs explicit and traceable from inclusive design guidelines to requirements, prototype features, and validation evidence. Drawing on a structured review of inclusive design literature, we identified 62 guidelines relevant to four vulnerable communities: older adults, low-digital-literacy users, rural users, and colour-blind users. These guidelines were translated into a catalogue of 67 functional and non-functional requirements for inclusive early warning systems. To our knowledge, this is one of the first studies in software engineering to consolidate and empirically validate inclusive design requirements for disaster early warning systems, where accessibility and usability failures can have serious safety consequences. The requirements were operationalised through an adaptive disaster early warning prototype and evaluated through six interviews and eight cognitive walkthroughs. The evaluation provided strong positive evidence across the four vulnerable groups, with particularly encouraging results for elderly and rural users, whose requirements achieved full validation coverage and high positive validation rates. The paper concludes by positioning human-centricity not as an ethical add-on, but as a traceable quality concern in the design of safe and equitable critical systems.

cs.SE↗

Walking the Tightrope of LLMs for Software Development: A Practitioners' Perspective

Background: Large Language Models emerged with the potential of provoking a revolution in software development (e.g., automating processes, workforce transformation). Although studies have started to investigate the perceived impact of LLMs for software development, there is a need for empirical studies to comprehend how to balance forward and backward effects of using LLMs. Objective: We investigated how LLMs impact software development and how to manage the impact from a software developer's perspective. Method: We conducted 22 interviews with software practitioners across 3 rounds of data collection and analysis, between October (2024) and September (2025). We employed Socio-Technical Grounded Theory for Data Analysis (STGT4DA) to rigorously analyse interview participants' responses. Results: We identified the benefits (e.g., maintain developer flow, improve developer mental models, and foster entrepreneurship) and challenges (e.g., damage to developers' reputation) of using LLMs at individual, team, organisation, and society levels; as well as actionable guidances into how mitigate these challenges. Conclusion: Critically, we present the trade-offs that software practitioners, teams, and organisations face in working with LLMs. Our findings are particularly useful for software team leaders and IT managers to assess the viability of LLMs within their specific context.

cs.SE↗

Engineering for Crisis Management: A User-Centred Analysis of Disaster Mobile Applications

Disaster mobile apps play an increasingly important role in disseminating hazard information and supporting communities during emergency situations. This study presents a comprehensive analysis of these mobile applications, focusing on their features, user-reported challenges, and opportunities for improvement. We first examined the landscape of disaster mobile apps by analysing 70 apps identified through a combination of methods, including those from the literature, the Google Play Store, and the App Store. The analysis categorised apps based on disaster focus, geographic coverage, popularity, monetisation strategies, and features across the disaster lifecycle. We then extracted, translated and analysed user reviews using topic modelling and sentiment analysis to identify key concerns and recurring issues. The results show that most applications prioritise response-related functionalities, with limited support for preparedness and recovery. User feedback highlights critical challenges related to technical reliability, usability, accessibility, and information clarity. Based on these findings, we propose a set of recommendations for developers and emergency management agencies to improve the reliability, inclusiveness, and overall effectiveness of disaster mobile apps. These include adopting lifecycle-oriented design approaches, strengthening multilingual support, improving technical robustness, and integrating user feedback into development processes. This work contributes to the growing body of research on human-centred disaster risk reduction by providing empirical insights and actionable guidance for the design of more reliable and inclusive disaster communication systems.

cs.SE↗

SSG: Logit-Balanced Vocabulary Partitioning for LLM Watermarking

Watermarking has emerged as a promising technique for tracing the authorship of content generated by large language models (LLMs). Among existing approaches, the KGW scheme is particularly attractive due to its versatility, efficiency, and effectiveness in natural language generation. However, KGW's effectiveness degrades significantly under low-entropy settings such as code generation and mathematical reasoning. A crucial step in the KGW method is random vocabulary partitioning, which enables adjustments to token selection based on specific preferences. Our study revealed that the next-token probability distribution plays an critical role in determining how much, or even whether, we can modify token selection and, consequently, the effectiveness of watermarking. We refer to this characteristic, associated with the probability distribution of each token prediction, as \emph{watermark strength.} In cases of random vocabulary partitioning, the lower bound of watermark strength is dictated by the next-token probability distribution. However, we found that, by redesigning the vocabulary partitioning algorithm, we can potentially raise this lower bound. In this paper, we propose SSG (\textbf{S}ort-then-\textbf{S}plit by \textbf{G}roups), a method that partitions the vocabulary into two logit-balanced subsets. This design lifts the lower bound of watermark strength for each token prediction, thereby improving watermark detectability. Experiments on code generation and mathematical reasoning datasets demonstrate the effectiveness of SSG.

cs.CR↗

Requirements Perception Gap across Stakeholders: A Comparative Survey of Aged Care Digital Health Software

We sought to explore and compare the perspectives of three key stakeholder groups: older adults, caregivers (formal health providers and informal caregivers), and digital health software developers on key functional and non-functional requirements. We conducted a survey, designed based on the findings from an existing systematic review, to gather and analyse data related to the three stakeholder groups' (dis)satisfaction with current aged care digital health software and their views on key future aged care software requirements. A mixed-methods survey approach integrated quantitative questionnaire data and qualitative open-ended responses from a total sample of 249, comprised of older adults (103), formal and informal caregivers (41), and software developers (105). Data analysis utilised a mixed methods approach, employing inferential statistics to compare group satisfaction levels and thematic analysis for qualitative open-ended responses. Our analysis reveals a significant "Requirements Gap". Software developers tend to prioritise advanced features and functional requirements, significantly overestimating user satisfaction with core NFRs such as ease of use and responsiveness. Conversely, developers were more critical of existing functional features compared to older adults and caregivers, who prioritised simplicity and reliability over feature density. By combining quantitative and qualitative analysis, we identified where stakeholder priorities align and where they diverge across functional and non-functional requirements in both the current designs they used and the future designs they desire. Our findings present a stakeholder gap analysis that can guide future co-design processes, near-term product decisions, and privacy-by-design recommendations in aged care digital health.

cs.SE↗

Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering

How software developers interact with Artificial Intelligence (AI)-powered tools, including Large Language Models (LLMs), plays a vital role in how these AI-powered tools impact them. While overreliance on AI may lead to long-term negative consequences (e.g., atrophy of critical thinking skills); underreliance might deprive software developers of potential gains in productivity and quality. Based on twenty-two interviews with software developers on using LLMs for software development, we propose a preliminary reliance-control framework where the level of control can be used as a way to identify AI overreliance and underreliance. We also use it to recommend future research to further explore the different control levels supported by the current and emergent LLM-driven tools. Our paper contributes to the emerging discourse on AI overreliance and provides an understanding of the appropriate degree of reliance as essential to developers making the most of these powerful technologies. Our findings can help practitioners, educators, and policymakers promote responsible and effective use of AI tools.

cs.SE↗

Towards Explainable Stakeholder-Aware Requirements Prioritisation in Aged-Care Digital Health

Requirements engineering for aged-care digital health must account for human aspects, because requirement priorities are shaped not only by technical functionality but also by stakeholders' health conditions, socioeconomics, and lived experience. Knowing which human aspects matter most, and for whom, is critical for inclusive and evidence-based requirements prioritisation. Yet in practice, while some studies have examined human aspects in RE, they have largely relied on expert judgement or model-driven analysis rather than large-scale user studies with meaningful human-in-the-loop validation to determine which aspects matter most and why. To address this gap, we conducted a mixed-methods study with 103 older adults, 105 developers, and 41 caregivers. We first applied an explainable machine learning to identify the human aspects most strongly associated with requirement priorities across 8 aged-care digital health themes, and then conducted 12 semi-structured interviews to validate and interpret the quantitative patterns. The results identify the key human aspects shaping requirement priorities, reveal their directional effects, and expose substantial misalignment across stakeholder groups. Together, these findings show that human-centric requirements analysis should engage stakeholder groups explicitly rather than collapsing their perspectives into a single aggregate view. This paper contributes an identification of the key human aspects driving requirement priorities in aged-care digital health and an explainable, human-centric RE framework that combines ML-derived importance rankings with qualitative validation to surface the stakeholder misalignments that inclusive requirements engineering must address.

cs.SE↗

Auto-Generating Personas from User Reviews in VR App Stores

Personas are a valuable tool for discussing accessibility requirements in software design and development practices. However, the use of personas for accessibility-focused requirements elicitation in VR projects remains limited and is accompanied by several challenges. To fill this gap, we developed an auto-generated persona system in a VR course, where the personas were used to facilitate discussions on accessibility requirements and to guide VR design and development. Our findings indicate that the auto-generated persona system enabled students to develop empathy more efficiently. This study demonstrates the use of automatically generated personas in VR course settings as a means of eliciting latent accessibility requirements.

cs.HC↗

Role of Graphics in Disaster Communication: Practitioner Perspectives on Use, Challenges, and Inclusivity

Information graphics, such as hazard maps, evacuation diagrams, and pictorial action guides, are widely used in disaster risk communication. These visuals are important because they convey hazard information quickly, reduce reliance on lengthy text, and support decision-making in time-critical situations. However, despite their importance, disaster information graphics do not work equally well for all audiences. In practice, many graphics remain difficult to interpret, and their accessibility for vulnerable populations is still uneven and underexplored. Despite their central role, there has been little empirical work examining how graphics shape disaster communication, what challenges practitioners face in using them, and, most importantly, how inclusive current disaster graphics are in real-world settings. To address this gap, we examine how information graphics are currently produced and used in disaster communication, what issues emerge in practice, and how inclusivity is addressed. We conducted semi-structured interviews with disaster communication practitioners and researchers to examine the role of graphics across preparedness, warning, and response contexts, as well as the barriers experienced by vulnerable communities. Our findings show that graphics are widely expected and heavily relied upon, yet significant accessibility gaps persist for groups such as people with vision impairments, older adults, and culturally and linguistically diverse communities. Participants also highlighted that inclusive adaptations are difficult to achieve during unfolding emergencies due to operational constraints, limited guidance, and resource barriers. Based on these findings, we outline recommendations for disaster management agencies and graphic designers and identify research directions for technological and adaptive support to make disaster graphics more inclusive at scale.

cs.GR↗

SOGPTSpotter: Detecting ChatGPT-Generated Answers on Stack Overflow

Stack Overflow is a popular Q&A platform where users ask technical questions and receive answers from a community of experts. Recently, there has been a significant increase in the number of answers generated by ChatGPT, which can lead to incorrect and unreliable information being posted on the site. While Stack Overflow has banned such AI-generated content, detecting whether a post is ChatGPT-generated remains a challenging task. We introduce a novel approach, SOGPTSpotter, that employs Siamese Neural Networks, leveraging the BigBird model and the Triplet loss, to detect ChatGPT-generated answers on Stack Overflow. We use triplets of human answers, reference answers, and ChatGPT answers. Our empirical evaluation reveals that our approach outperforms well-established baselines like GPTZero, DetectGPT, GLTR, BERT, RoBERTa, and GPT-2 in identifying ChatGPT-synthesized Stack Overflow responses. We also conducted an ablation study to show the effectiveness of our model. Additional experiments were conducted to assess various factors, including the impact of text length, the model's robustness against adversarial attacks, and its generalization capabilities across different domains and large language models. We also conducted a real-world case study on Stack Overflow. Using our tool's recommendations, Stack Overflow moderators were able to identify and take down ChatGPT-suspected generated answers, demonstrating the practical applicability and effectiveness of our approach.

cs.SE↗

VRARE: Using Virtual Reality to Understand Accessibility Requirements of Color Blindness and Weakness

In this paper, we developed a virtual reality (VR) system that can simulate color blindness and weakness. We built an immersive 3D web view interface where participants can discuss accessibility requirements for a fitness website projects within a virtual fitness environment. We conducted a pilot experiment involving 24 participants from six software teams, who used both VR and non-VR methods to understand color blindness and weakness requirements in a website project. Our findings indicate that using VR can provide several benefits for requirements activities, such as an improved user experience and reduced workload.

cs.HC↗