Searcharxiv⌕ Search

arXiv subjects

Sherlock A. Licorish

Publications and source records attributed to Sherlock A. Licorish.

At least 19 recordsLinked to original sources

Using machine learning approaches to classify false-positives generated by static analysis tools - An empirical study using small datasets

Static Code Analysis (SCA) tools are vital for software development to reduce the cost and time required for manual code reviews. It was established in our previous study that the best tools reported in the software engineering community return high false-positive and false-negative rates when used for code analysis. Several approaches have been taken in existing studies to identify false-positive alarms generated by SCA tools, in helping to enhance the utility of these tools. However, existing studies reveal several limitations, chiefly among which is the tendency to focus on a limited number of alarm types or failing to evaluate a large breadth of Machine-Learning (ML) techniques comprehensively. This study has addressed this opportunity. We identified the best Code Representation Learning (CRL) techniques and ML algorithms for identifying false-positives, where results show that performance outcomes vary based on the characteristics of the training datasets and the goal of the model building. Given the evidence established in this study, we recommend further investigations targeting specific configurations of advanced CRLs and ML Algorithms, including simple ML models.

cs.SE↗

Beyond Benchmarks: Evaluating Large Language Models and Agentic Patterns for Static Code Analysis Warning Classification

Context: Static Code Analysis (SCA) tools support software verification and validation, reducing the time and cost of code reviews. However, the best SCA tools in the practitioner community generate a high volume of false positives, which developers tend to ignore, leading to genuine warnings being overlooked. Accordingly, studies increasingly adopt Large Language Models (LLMs) to identify and reduce false positives. Existing studies have produced promising results on public datasets, but underperformed on unseen datasets. Moreover, researchers do not leverage the full potential of Agentic patterns in LLMs or incorporate established code quality standards for warning classification. Objectives: This study utilises four manually annotated datasets to evaluate the performance of three contemporary LLMs for SCA warning classification in real-world applications, and incorporates Agentic patterns and established coding standards to enhance models' performance, addressing gaps identified in existing studies. Method: Three LLMs were selected using the LiveCodeBench benchmark, and the Structured Chain-of-Thought (SCoT) prompting technique was used to evaluate these LLMs. Furthermore, we incorporated Agentic patterns, such as Debating and Self-Consistency, to potentially improve performance in SCA warning classification. Results: Our results show that the performance of different LLMs varies with the characteristics of the dataset, ranging from 0.52 to 0.96 F1-Score for the best models. Additionally, the use of Agentic patterns does not improve LLMs' performance in most scenarios. Conclusions: SCA warning classification using LLMs works best only with certain warning types. We recommend evaluating LLMs in real-world applications rather than using publicly available datasets/benchmarks to understand their full utility.

cs.SE↗

Usage Patterns of Programming Languages and Tools: A Systematic Literature Review

Technology and the popularity of languages and tools are rapidly evolving. Thus, it is important to constantly assess the most popular languages and tools used in industry, furthering the collaboration between research and the software industry. In this study, we use a systematic literature review and investigate 37 studies published between 2015 -- 2024, which explored the programming languages, IDEs and CI tools used in industry. We observed that Java is most often listed among the most popular languages, and there are limited studies that investigate specific IDE usage. Also, Travis CI and GitHub Actions are the most common CI tools identified as being top CI tools used in industry. Overall, evidence shows that there is a lack of research focused on popular languages and tools used in industry, largely as there are many limitations with the possible methodology approaches. For example, lack of access to closed-source projects and sampling biases are potential issues that limit our understanding of popular languages and tools used in industry. In this regard, there is a need for more recent studies into development tools used in industry, involving multiple research approaches, to balance the limitations of each approach.

cs.SE↗

At What Cost? Software Developers' Well-Being in the Age of GenAI

Generative Artificial Intelligence (GenAI) is rapidly reshaping software development, with growing emphasis on accelerating productivity and optimizing performance. However, excessive focus on such dimensions risks overlooking the critical implications for developer well-being. GenAI tools can amplify cognitive load, introduce new forms of oversight labor, and escalate expectations around output and pace, contributing to stress, burnout, and diminished work-life balance. The GenAI movement is also transforming professional norms, altering career entry points, demanding continuous adaptation, and deepening inequalities in access and support. This position paper calls for a reorientation of the GenAI research agenda in software development and proposes a theoretical framework to move beyond narrow performance metrics toward investigations that also center on human experience, social context, and sustainable productivity.

cs.SE↗

Geographic Variation in Stack Overflow Code Quality: Evidence from a Cross-Regional Study of Coding Practices

Developers frequently reuse Stack Overflow code snippets, yet the quality of these snippets remains unevenly understood, particularly across programming languages and geographic contexts. This study investigates code quality in Stack Overflow answers from contributors located in the United States, focusing on SQL, JavaScript, Python, Ruby, and Java snippets. We evaluate four quality dimensions: reliability, readability, performance, and security. Using language-specific linting and static analysis tools, we quantify violations across states and cities, compute violation densities to enable fair regional comparison, and examine relationships between code quality and state-level diversity indicators. We further conduct inductive content analysis on code snippets from California, Utah, and North Dakota to identify qualitative patterns in code quality violations. Results show that readability violations are the most prevalent across all languages, followed by reliability, performance, and security. Common issues include improper whitespace, inconsistent formatting, program-flow errors, inefficient resource use, unsanitised inputs, and insecure dynamic evaluation. Regional analysis indicates that major technology hubs produce more parsable snippets but do not necessarily exhibit higher violation densities. States with broader access to computing devices, Internet subscriptions, higher income, and more equitable wealth distribution tend to show fewer code quality violations. Qualitative findings suggest that established technology regions often produce more complex violations, while less mature technology regions display more fundamental errors. These findings highlight the socio-technical nature of code quality in community question-answering platforms and suggest that developers should exercise caution when reusing online code snippets.

cs.SE↗

A Pilot Study on Detecting Software Design Patterns with Large Language Models: An Empirical Evaluation

Design patterns provide reusable solutions to recurring software design problems. Automatically detecting these patterns in source code can help bootstrap new developers' understanding of unfamiliar software system architectures, and can help experienced developers to quickly identify and rectify potential quality issues. While many prior research works have explored graph based and machine-learning based detection techniques, this work evaluates the design pattern recognition capabilities of four Large Language Models and two ensemble approaches consisting three out of the four models. We also compare their performance when prompted with a) Source code, b) PlantUML representation of source code, and c) Text-based descriptions of the source code. We investigate the detection of five design patterns: singleton, adapter, bridge, composite and decorator. Our preliminary results indicate that LLMs show promise for automatically detecting design patterns, with NextCoder and Gemma 3 demonstrating comparatively higher accuracy than other models evaluated, and the ensemble approaches enhancing the overall efficiency of design pattern detection. We identify several directions for future work.

cs.SE↗

The Future of Software Engineering Conferences: A New Zealand Perspective

Software engineering (SE) conferences are vital for knowledge exchange and collaboration, yet can also involve significant barriers for researchers in geographically distant regions such as New Zealand. We identify barriers such as high travel costs, misaligned academic calendars, and limited representation, and propose strategies including hybrid participation, cost-conscious venues, and governance reforms. We make recommendations to promote equitable global participation and strengthen the SE research community.

cs.SE↗

Why Do You Contribute to Stack Overflow? Understanding Cross-Cultural Motivations and Usage Patterns before the Age of LLMs

Understanding motivations of contributors for participating in community question and answer platforms is crucial for sustaining knowledge-sharing ecosystem, which is necessary to advance the discipline while also ensuring its longevity. This is particularly necessary in the age of LLMs, where data from such portals are used to train these models. Limited insights exist regarding how motivations of contributors vary across different national cultures. This research investigates Stack Overflow contributor motivations, analysing regional differences and relations to platform activity. We combined qualitative content analysis of Stack Overflow profiles with quantitative linguistic analysis of data from the United States, China, and Russia. Using deductive content analysis, we identified 17 motivational categories. We applied correlation analysis to identify associations between stated motivations and platform activities. Contributors are primarily motivated by advertising opportunities and altruistic problem solving desires. American contributors demonstrated stronger self promotional behaviours while Chinese contributors exhibited greater learning oriented engagement. Our correlation analysis showed that those with more detailed profiles tend to engage in advertising and social activities, while learning oriented users maintain minimal self presentation. Understanding these variations can inform strategies for enhancing cross cultural participation in software engineering.

cs.SE↗

Taking a Closer Look at Warnings Generated by PMD and SonarQube, their Rules and Compliance to Established Coding Standards

Context: Static code analysis (SCA) tools play a vital role in software development, reducing the cost and time required for code reviews. However, high false-positive and false-negative rates are reported for the best tools in the community. Accordingly, studies often aim to develop datasets for learning SCA warning patterns to reduce false results. These datasets are meant to possess high-quality and high-volume in covering the full range of faults/rules that typically result in false warnings and be compliant with established coding standards. However, existing studies have not utilised such datasets or identified the breadth of rules that are prone to false positives and their compliance to coding standards. Objectives: We analysed code from Stack Overflow and Apache Tomcat to capture variations in code length and style in detecting false-positive warnings from best-performing tools PMD and SonarQube, addressing this gap. Method: In deriving false-positive warnings, outcomes from the tools were labelled using established coding standards. Deeper analyses were then conducted to identify the rules that are prone to false-positives, reasons for these, and agreement/gaps between SCA rules and established standards. Results: Among our main outcomes, we observe that only a few SCA rules generate false positives, ranging from 4.64% to 18.45% across four datasets. Additionally, eliminating rules that contradict established standards significantly reduce the false-positive rate. Additionally, our findings reveal discrepancies between tools and established standards. Conclusion: Given the evidence established in this study, we recommend further investigations into gaps between tools and established standards, including the use of machine learning approaches to annotate larger datasets.

cs.SE↗

On the need to perform comprehensive evaluations of automated program repair benchmarks: Sorald case study

In supporting the development of high-quality software, especially necessary in the era of LLMs, automated program repair (APR) tools aim to improve code quality by automatically addressing violations detected by static analysis profilers. Previous research tends to evaluate APR tools only for their ability to clear violations, neglecting their potential introduction of new (sometimes severe) violations, changes to code functionality and degrading of code structure. There is thus a need for research to develop and assess comprehensive evaluation frameworks for APR tools. This study addresses this research gap, and evaluates Sorald (a state-of-the-art APR tool) as a proof of concept. Sorald's effectiveness was evaluated in repairing 3,529 SonarQube violations across 30 rules within 2,393 Java code snippets extracted from Stack Overflow. Outcomes show that while Sorald fixes specific rule violations, it introduced 2,120 new faults (32 bugs, 2088 code smells), reduced code functional correctness--as evidenced by a 24% unit test failure rate--and degraded code structure, demonstrating the utility of our framework. Findings emphasize the need for evaluation methodologies that capture the full spectrum of APR tool effects, including side effects, to ensure their safe and effective adoption.

cs.SE↗

Static Analysis as a Feedback Loop: Enhancing LLM-Generated Code Beyond Correctness

Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% to 11%, and reliability warnings from >50% to 11% within ten iterations. These results demonstrate that LLMs, when guided by static analysis feedback, can significantly enhance code quality beyond functional correctness.

cs.SE↗

Predictive Analytics for Collaborators Answers, Code Quality, and Dropout on Stack Overflow

Previous studies that used data from Stack Overflow to develop predictive models often employed limited benchmarks of 3-5 models or adopted arbitrary selection methods. Despite being insightful, their limited scope suggests the need to benchmark more models to avoid overlooking untested algorithms. Our study evaluates 21 algorithms across three tasks: predicting the number of question a user is likely to answer, their code quality violations, and their dropout status. We employed normalisation, standardisation, as well as logarithmic and power transformations paired with Bayesian hyperparameter optimisation and genetic algorithms. CodeBERT, a pre-trained language model for both natural and programming languages, was fine-tuned to classify user dropout given their posts (questions and answers) and code snippets. We found Bagging ensemble models combined with standardisation achieved the highest R2 value (0.821) in predicting user answers. The Stochastic Gradient Descent regressor, followed by Bagging and Epsilon Support Vector Machine models, consistently demonstrated superior performance to other benchmarked algorithms in predicting user code quality across multiple quality dimensions and languages. Extreme Gradient Boosting paired with log-transformation exhibited the highest F1-score (0.825) in predicting user dropout. CodeBERT was able to classify user dropout with a final F1-score of 0.809, validating the performance of Extreme Gradient Boosting that was solely based on numerical data. Overall, our benchmarking of 21 algorithms provides multiple insights. Researchers can leverage findings regarding the most suitable models for specific target variables, and practitioners can utilise the identified optimal hyperparameters to reduce the initial search space during their own hyperparameter tuning processes.

cs.SE↗

GBM Returns the Best Prediction Performance among Regression Approaches: A Case Study of Stack Overflow Code Quality

Practitioners are increasingly dependent on publicly available resources for supporting their knowledge needs during software development. This has thus caused a spotlight to be paced on these resources, where researchers have reported mixed outcomes around the quality of these resources. Stack Overflow, in particular, has been studied extensively, with evidence showing that code resources on this platform can be of poor quality at times. Limited research has explored the variables or factors that predict code quality on Stack Overflow, but instead has focused on ranking content, identifying defects and predicting future content. In many instances approaches used for prediction are not evaluated to identify the best techniques. Contextualizing the Stack Overflow code quality problem as regression-based, we examined the variables that predict Stack Overflow (Java) code quality, and the regression approach that provides the best predictive power. Six approaches were considered in our evaluation, where Gradient Boosting Machine (GBM) stood out. In addition, longer Stack Overflow code tended to have more code violations, questions that were scored higher also attracted more views and the more answers that are added to questions on Stack Overflow the more errors were typically observed in the code that was provided. Outcomes here point to the value of the GBM ensemble learning mechanism, and the need for the practitioner community to be prudent when contributing and reusing Stack Overflow Java coding resource.

cs.SE↗

Comparing Human and LLM Generated Code: The Jury is Still Out!

Much is promised in relation to AI-supported software development. However, there has been limited evaluation effort in the research domain aimed at validating the true utility of such techniques, especially when compared to human coding outputs. We bridge this gap, where a benchmark dataset comprising 72 distinct software engineering tasks is used to compare the effectiveness of large language models (LLMs) and human programmers in producing Python software code. GPT-4 is used as a representative LLM, where for the code generated by humans and this LLM, we evaluate code quality and adherence to Python coding standards, code security and vulnerabilities, code complexity and functional correctness. We use various static analysis benchmarks, including Pylint, Radon, Bandit and test cases. Among the notable outcomes, results show that human-generated code recorded higher ratings for adhering to coding standards than GPT-4. We observe security flaws in code generated by both humans and GPT-4, however, code generated by humans shows a greater variety of problems, but GPT-4 code included more severe outliers. Our results show that although GPT-4 is capable of producing coding solutions, it frequently produces more complex code that may need more reworking to ensure maintainability. On the contrary however, our outcomes show that a higher number of test cases passed for code generated by GPT-4 across a range of tasks than code that was generated by humans. That said, GPT-4 frequently struggles with complex problem-solving that involve in-depth domain knowledge. This study highlights the potential utility of LLMs for supporting software development, however, tasks requiring comprehensive, innovative or unconventional solutions, and careful debugging and error correction seem to be better developed by human programmers. We plot an agenda for the software engineering community.

cs.SE↗

Features that Predict the Acceptability of Java and JavaScript Answers on Stack Overflow

Context: Stack Overflow is a popular community question and answer portal used by practitioners to solve problems during software development. Developers can focus their attention on answers that have been accepted or where members have recorded high votes in judging good answers when searching for help. However, the latter mechanism (votes) can be unreliable, and there is currently no way to differentiate between an answer that is likely to be accepted and those that will not be accepted by looking at the answer's characteristics. Objective: In potentially providing a mechanism to identify acceptable answers, this study examines the features that distinguish an accepted answer from an unaccepted answer. Methods: We studied the Stack Overflow dataset by analyzing questions and answers for the two most popular tags (Java and JavaScript). Our dataset comprised 249,588 posts drawn from 2014-2016. We use random forest and neural network models to predict accepted answers, and study the features with the highest predictive power in those two models. Results: Our findings reveal that the length of code in answers, reputation of users, similarity of the text between questions and answers, and the time lag between questions and answers have the highest predictive power for differentiating accepted and unaccepted answers. Conclusion: Tools may leverage these findings in supporting developers and reducing the effort they must dedicate to searching for suitable answers on Stack Overflow.

cs.SE↗

Studying the Characteristics of SQL-related Development Tasks: An Empirical Study

A key function of a software system is its ability to facilitate the manipulation of data, which is often implemented using a flavour of the Structured Query Language (SQL). To develop the data operations of software (i.e, creating, retrieving, updating, and deleting data), developers are required to excel in writing and combining both SQL and application code. The problem is that writing SQL code in itself is already challenging (e.g., SQL anti-patterns are commonplace) and combining SQL with application code (i.e., for SQL development tasks) is even more demanding. Meanwhile, we have little empirical understanding regarding the characteristics of SQL development tasks. Do SQL development tasks typically need more code changes? Do they typically have a longer time-to-completion? Answers to such questions would prepare the community for the potential challenges associated with such tasks. Our results obtained from 20 Apache projects reveal that SQL development tasks have a significantly longer time-to-completion than SQL-unrelated tasks and require significantly more code changes. Through our qualitative analyses, we observe that SQL development tasks require more spread out changes, effort in reviews and documentation. Our results also corroborate previous research highlighting the prevalence of SQL anti-patterns. The software engineering community should make provision for the peculiarities of SQL coding, in the delivery of safe and secure interactive software.

cs.SE↗

A Systematic Mapping Study Addressing the Reliability of Mobile Applications: The Need to Move Beyond Testing Reliability

Intense competition in the mobile apps market means it is important to maintain high levels of app reliability to avoid losing users. Yet despite its importance, app reliability is underexplored in the research literature. To address this need, we identify, analyse, and classify the state-of-the-art in the field of mobile apps' reliability through a systematic mapping study. From the results of such a study, researchers in the field can identify pressing research gaps, and developers can gain knowledge about existing solutions, to potentially leverage them in practice. We found 87 relevant papers which were then analysed and classified based on their research focus, research type, contribution, research method, study settings, data, quality attributes and metrics used. Results indicate that there is a lack of research on understanding reliability with regard to context-awareness, self-healing, ageing and rejuvenation, and runtime event handling. These aspects have rarely been studied, or if studied, there is limited evaluation. We also identified several other research gaps including the need to conduct more research in real-world industrial projects. Furthermore, little attention has been paid towards quality standards while conducting research. Outcomes here show numerous opportunities for greater research depth and breadth on mobile app reliability.

cs.SE↗

Software Engineering in Australasia

Six months ago an important call was made for researchers globally to provide insights into the way Software Engineering is done in their region. Heeding this call we hereby outline the position Software Engineering in Australasia (New Zealand and Australia). This article first considers the software development methods practices and tools that are popular in the Australasian software engineering community. We then briefly review the particular strengths of software engineering researchers in Australasia. Finally we make an open call for collaborators by reflecting on our current position and identifying future opportunities

cs.SE↗