Searcharxiv⌕ Search

arXiv subjects

Gordon Fraser

Publications and source records attributed to Gordon Fraser.

At least 55 records · Page 3Linked to original sources
arXiv2023

Improving Testing Behavior by Gamifying IntelliJ

Testing is an important aspect of software development, but unfortunately, it is often neglected. While test quality analyses such as code coverage or mutation analysis inform developers about the quality of their tests, such reports are viewed only sporadically during continuous integration or code review, if they are considered at all, and their impact on the developers' testing behavior therefore tends to be negligible. To actually influence developer behavior, it may rather be necessary to motivate developers directly within their programming environment, while they are coding. We introduce IntelliGame, a gamified plugin for the popular IntelliJ Java Integrated Development Environment, which rewards developers for positive testing behavior using a multi-level achievement system: A total of 27 different achievements, each with incremental levels, provide affirming feedback when developers exhibit commendable testing behavior, and provide an incentive to further continue and improve this behavior. A controlled experiment with 49 participants given a Java programming task reveals substantial differences in the testing behavior triggered by IntelliGame: Incentivized developers write more tests, achieve higher coverage and mutation scores, run their tests more often, and achieve functionality earlier.

cs.SE↗
arXiv2023

Do Automatic Test Generation Tools Generate Flaky Tests?

Non-deterministic test behavior, or flakiness, is common and dreaded among developers. Researchers have studied the issue and proposed approaches to mitigate it. However, the vast majority of previous work has only considered developer-written tests. The prevalence and nature of flaky tests produced by test generation tools remain largely unknown. We ask whether such tools also produce flaky tests and how these differ from developer-written ones. Furthermore, we evaluate mechanisms that suppress flaky test generation. We sample 6 356 projects written in Java or Python. For each project, we generate tests using EvoSuite (Java) and Pynguin (Python), and execute each test 200 times, looking for inconsistent outcomes. Our results show that flakiness is at least as common in generated tests as in developer-written tests. Nevertheless, existing flakiness suppression mechanisms implemented in EvoSuite are effective in alleviating this issue (71.7 % fewer flaky tests). Compared to developer-written flaky tests, the causes of generated flaky tests are distributed differently. Their non-deterministic behavior is more frequently caused by randomness, rather than by networking and concurrency. Using flakiness suppression, the remaining flaky tests differ significantly from any flakiness previously reported, where most are attributable to runtime optimizations and EvoSuite-internal resource thresholds. These insights, with the accompanying dataset, can help maintainers to improve test generation tools, give recommendations for developers using these tools, and serve as a foundation for future research in test flakiness or test generation.

cs.SE↗
arXiv2023

NuzzleBug: Debugging Block-Based Programs in Scratch

While professional integrated programming environments support developers with advanced debugging functionality, block-based programming environments for young learners often provide no support for debugging at all, thus inhibiting debugging and preventing debugging education. In this paper we introduce NuzzleBug, an extension of the popular block-based programming environment Scratch that provides the missing debugging support. NuzzleBug allows controlling the executions of Scratch programs with classical debugging functionality such as stepping and breakpoints, and it is an omniscient debugger that also allows reverse stepping. To support learners in deriving hypotheses that guide debugging, NuzzleBug is an interrogative debugger that enables to ask questions about executions and provides answers explaining the behavior in question. In order to evaluate NuzzleBug, we survey the opinions of teachers, and study the effects on learners in terms of debugging effectiveness and efficiency. We find that teachers consider NuzzleBug to be useful, and children can use it to debug faulty programs effectively. However, systematic debugging requires dedicated training, and even when NuzzleBug can provide correct answers learners may require further help to comprehend faults and necessary fixes, thus calling for further research on improving debugging techniques and the information they provide.

cs.SE↗
arXiv2023

A Survey on What Developers Think About Testing

Software is infamous for its poor quality and frequent occurrence of bugs. While there is no doubt that thorough testing is an appropriate answer to ensure sufficient quality, the poor state of software generally suggests that developers may not always engage as thoroughly with testing as they should. This observation aligns with the prevailing belief that developers simply do not like writing tests. In order to determine the truth of this belief, we conducted a comprehensive survey with 21 questions aimed at (1) assessing developers' current engagement with testing and (2) identifying factors influencing their inclination toward testing; that is, whether they would actually like to test more but are inhibited by their work environment, or whether they would really prefer to test even less if given the choice. Drawing on 284 responses from professional software developers, we uncover reasons that positively and negatively impact developers' motivation to test. Notably, reasons for motivation to write more tests encompass not only a general pursuit of software quality but also personal satisfaction. However, developers nevertheless perceive testing as mundane and tend to prioritize other tasks. One approach emerging from the responses to mitigate these negative factors is by providing better recognition for developers' testing efforts.

cs.SE↗
arXiv2023

Exploring Programming Task Creation of Primary School Teachers in Training

Introducing computational thinking in primary school curricula implies that teachers have to prepare appropriate lesson material. Typically this includes creating programming tasks, which may overwhelm primary school teachers with lacking programming subject knowledge. Inadequate resulting example code may negatively affect learning, and students might adopt bad programming habits or misconceptions. To avoid this problem, automated program analysis tools have the potential to help scaffolding task creation processes. For example, static program analysis tools can automatically detect both good and bad code patterns, and provide hints on improving the code. To explore how teachers generally proceed when creating programming tasks, whether tool support can help, and how it is perceived by teachers, we performed a pre-study with 26 and a main study with 59 teachers in training and the LitterBox static analysis tool for Scratch. We find that teachers in training (1) often start with brainstorming thematic ideas rather than setting learning objectives, (2) write code before the task text, (3) give more hints in their task texts and create fewer bugs when supported by LitterBox, and (4) mention both positive aspects of the tool and suggestions for improvement. These findings provide an improved understanding of how to inform teacher training with respect to support needed by teachers when creating programming tasks.

cs.CY↗
arXiv2023

Debugging Flaky Tests using Spectrum-based Fault Localization

Non-deterministically behaving (i.e., flaky) tests hamper regression testing as they destroy trust and waste computational and human resources. Eradicating flakiness in test suites is therefore an important goal, but automated debugging tools are needed to support developers when trying to understand the causes of flakiness. A popular example for an automated approach to support regular debugging is spectrum-based fault localization (SFL), a technique that identifies software components that are most likely the causes of failures. While it is possible to also apply SFL for locating likely sources of flakiness in code, unfortunately the flakiness makes SFL both imprecise and non-deterministic. In this paper we introduce SFFL (Spectrum-based Flaky Fault Localization), an extension of traditional coverage-based SFL that exploits our observation that 80% of flaky tests exhibit varying coverage behavior between different runs. By distinguishing between stable and flaky coverage, SFFL is able to locate the sources of flakiness more precisely and keeps the localization itself deterministic. An evaluation on 101 flaky tests taken from 48 open-source Python projects demonstrates that SFFL is effective: Of five prominent SFL formulas, DStar, Ochiai, and Op2 yield the best overall performance. On average, they are able to narrow down the fault's location to 3.5 % of the project's code base, which is 18.7 % better than traditional SFL (for DStar). SFFL's effectiveness, however, depends on the root causes of flakiness: The source of non-order-dependent flaky tests can be located far more precisely than order-dependent faults.

cs.SE↗
arXiv2023

FlaPy: Mining Flaky Python Tests at Scale

Flaky tests obstruct software development, and studying and proposing mitigations against them has therefore become an important focus of software engineering research. To conduct sound investigations on test flakiness, it is crucial to have large, diverse, and unbiased datasets of flaky tests. A common method to build such datasets is by rerunning the test suites of selected projects multiple times and checking for tests that produce different outcomes. While using this technique on a single project is mostly straightforward, applying it to a large and diverse set of projects raises several implementation challenges such as (1) isolating the test executions, (2) supporting multiple build mechanisms, (3) achieving feasible run times on large datasets, and (4) analyzing and presenting the test outcomes. To address these challenges we introduce FlaPy, a framework for researchers to mine flaky tests in a given or automatically sampled set of Python projects by rerunning their test suites. FlaPy isolates the test executions using containerization and fresh execution environments to simulate real-world CI conditions and to achieve accurate results. By supporting multiple dependency installation strategies, it promotes diversity among the studied projects. FlaPy supports parallelizing the test executions using SLURM, making it feasible to scan thousands of projects for test flakiness. Finally, FlaPy analyzes the test outcomes to determine which tests are flaky and depicts the results in a concise table. A demo video of FlaPy is available at https://youtu.be/ejy-be-FvDY

cs.SE↗
arXiv2023

The ABC of Pair Programming: Gender-dependent Attitude, Behavior and Code of Young Learners

Young learners are increasingly introduced to programming, and one of the main challenges for educators is to achieve learning success while also creating enthusiasm. As it is particularly difficult to achieve this enthusiasm initially in young females, prior work has identified gender-specific differences in the programming behavior of young learners. Since pair programming, which turns programming into a more sociable activity, has been proposed as an approach to support programming education, in this paper we aim to investigate whether similar gender-specific characteristics can also be observed during pair programming. Therefore, we designed a gender-neutral introductory SCRATCH programming course tailored for integrating pair programming principles, and conducted it with a total of 139 students aged between 8 and 14 years. To identify gender-dependent differences and similarities, we measure the attitude towards programming and the course setting, observe the behavior of the students while programming, and analyze the code of the programs for different gender-combinations. Overall, our study demonstrates that pair programming is well suited for young learners and results in a positive attitude. While the resulting programs are similar in quality and complexity independent of gender, differences are evident when it comes to the compliance to pair programming roles, the exploration of code, and the creative customization of programs. These findings contribute to an in-depth understanding of social and technical gender specifics of pair programming, and provide educators with resources and guidance for implementing gender-sensitive pair programming in the classroom.

cs.SE↗
arXiv2023

Exposing Software Engineering Students to Stressful Projects: Does Diversity Matter?

Software development teams have to face stress caused by deadlines, staff turnover, or individual differences in commitment, expertise, and time zones. While students are typically taught the theory of software project management, their exposure to such stress factors is usually limited. However, preparing students for the stress they will have to endure once they work in project teams is important for their own sake, as well as for the sake of team performance in the face of stress. Team performance has been linked to the diversity of software development teams, but little is known about how diversity influences the stress experienced in teams. In order to shed light on this aspect, we provided students with the opportunity to self-experience the basics of project management in self-organizing teams, and studied the impact of six diversity dimensions on team performance, coping with stressors, and positive perceived learning effects. Three controlled experiments at two universities with a total of 65 participants suggest that the social background impacts the perceived stressors the most, while age and work experience have the highest impact on perceived learnings. Most diversity dimensions have a medium correlation with the quality of work, yet no significant relation to the team performance. This lays the foundation to improve students' training for software engineering teamwork based on their diversity-related needs and to create diversity-sensitive awareness among educators, employers and researchers.

cs.SE↗
arXiv2023

Learning by Viewing: Generating Test Inputs for Games by Integrating Human Gameplay Traces in Neuroevolution

Although automated test generation is common in many programming domains, games still challenge test generators due to their heavy randomisation and hard-to-reach program states. Neuroevolution combined with search-based software testing principles has been shown to be a promising approach for testing games, but the co-evolutionary search for optimal network topologies and weights involves unreasonably long search durations. In this paper, we aim to improve the evolutionary search for game input generators by integrating knowledge about human gameplay behaviour. To this end, we propose a novel way of systematically recording human gameplay traces, and integrating these traces into the evolutionary search for networks using traditional gradient descent as a mutation operator. Experiments conducted on eight diverse Scratch games demonstrate that the proposed approach reduces the required search time from five hours down to only 52 minutes.

cs.SE↗
arXiv2023

Code Critters: A Block-Based Testing Game

Learning to program has become common in schools, higher education and individual learning. Although testing is an important aspect of programming, it is often neglected in education due to a perceived lack of time and knowledge, or simply because testing is considered less important or fun. To make testing more engaging, we therefore introduce Code Critters, a Tower Defense game based on testing concepts: The aim of the game is to place magic mines along the route taken by small "critters" from their home to a tower, such that the mines distinguish between critters executing correct code from those executing buggy code. Code is shown and edited using a block-based language to make the game accessible for younger learners. The mines encode test inputs as well as test oracles, thus making testing an integral and fun component of the game.

cs.SE↗
arXiv2023

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

cs.SE↗
arXiv2023

On the Applicability of Language Models to Block-Based Programs

Block-based programming languages like Scratch are increasingly popular for programming education and end-user programming. Recent program analyses build on the insight that source code can be modelled using techniques from natural language processing. Many of the regularities of source code that support this approach are due to the syntactic overhead imposed by textual programming languages. This syntactic overhead, however, is precisely what block-based languages remove in order to simplify programming. Consequently, it is unclear how well this modelling approach performs on block-based programming languages. In this paper, we investigate the applicability of language models for the popular block-based programming language Scratch. We model Scratch programs using n-gram models, the most essential type of language model, and transformers, a popular deep learning model. Evaluation on the example tasks of code completion and bug finding confirm that blocks inhibit predictability, but the use of language models is nevertheless feasible. Our findings serve as foundation for improving tooling and analyses for block-based languages.

cs.PL↗
arXiv2022

Neuroevolution-Based Generation of Tests and Oracles for Games

Game-like programs have become increasingly popular in many software engineering domains such as mobile apps, web applications, or programming education. However, creating tests for programs that have the purpose of challenging human players is a daunting task for automatic test generators. Even if test generation succeeds in finding a relevant sequence of events to exercise a program, the randomized nature of games means that it may neither be possible to reproduce the exact program behavior underlying this sequence, nor to create test assertions checking if observed randomized game behavior is correct. To overcome these problems, we propose Neatest, a novel test generator based on the NeuroEvolution of Augmenting Topologies (NEAT) algorithm. Neatest systematically explores a program's statements, and creates neural networks that operate the program in order to reliably reach each statement -- that is, Neatest learns to play the game in a way to reliably cover different parts of the code. As the networks learn the actual game behavior, they can also serve as test oracles by evaluating how surprising the observed behavior of a program under test is compared to a supposedly correct version of the program. We evaluate this approach in the context of Scratch, an educational programming environment. Our empirical study on 25 non-trivial Scratch games demonstrates that our approach can successfully train neural networks that are not only far more resilient to random influences than traditional test suites consisting of static input sequences, but are also highly effective with an average mutation score of more than 65%.

cs.SE↗
arXiv2022

Common Patterns in Block-Based Robot Programs

Programmable robots are engaging and fun to play with, interact with the real world, and are therefore well suited to introduce young learners to programming. Introductory robot programming languages often extend existing block-based languages such as Scratch. While teaching programming with such languages is well established, the interaction with the real world in robot programs leads to specific challenges, for which learners and educators may require assistance and feedback. A practical approach to provide this feedback is by identifying and pointing out patterns in the code that are indicative of good or bad solutions. While such patterns have been defined for regular block-based programs, robot-specific programming aspects have not been considered so far. The aim of this paper is therefore to identify patterns specific to robot programming for the Scratch-based mBlock programming language, which is used for the popular mBot and Codey Rocky robots. We identify: (1) 26 bug patterns, which indicate erroneous code; (2) three code smells, which indicate code that may work but is written in a confusing or difficult to understand way; and (3) 18 code perfumes, which indicate aspects of code that are likely good. We extend the LitterBox analysis framework to automatically identify these patterns in mBlock programs. Evaluated on a dataset of 3,540 mBlock programs, we find a total of 6,129 instances of bug patterns, 592 code smells and 14,495 code perfumes. This demonstrates the potential of our approach to provide feedback and assistance to learners and educators alike for their mBlock robot programs.

cs.SE↗
arXiv2022

Common Problems and Effects of Feedback on Fun When Programming Ozobots in Primary School

Computational thinking is increasingly introduced at primary school level, usually with some form of programming activity. In particular, educational robots provide an opportunity for engaging students with programming through hands-on experiences. However, primary school teachers might not be adequately prepared for teaching computer science related topics, and giving feedback to students can often be challenging: Besides the content of the feedback (e.g., what problems have to be handled), the way the feedback is given is also important, as it can lead to negative emotional effects. To support teachers with the way of giving feedback on common problems when teaching programming with robotics, we conducted a study consisting of seven workshops with three third and four fourth grade primary school classes. Within seven different activities, the 116 primary school children first programmed the Ozobot Evo robot in the pen-and-paper mode and then on a digital device. Throughout these activities we collected data on the problems the students encountered, the feedback given, and the fun they experienced. Our analysis reveals eight categories of problems, which we summarise in this paper together with corresponding possible feedback. We observed that problems that are urgent or can harm the students' self-efficacy have a negative impact on how enjoyable an activity is perceived. While direct instruction significantly decreased the experienced fun, hints had a positive effect. Generally, we found programming the Ozobot Evo to be encouraging for both girls and boys. To support teachers, we discuss ideas for giving encouraging feedback on common problems of Ozobot Evo programming activities and how our findings transfer to other robots.

cs.SE↗
arXiv2022

Gender-dependent Contribution, Code and Creativity in a Virtual Programming Course

Since computer science is still mainly male dominated, academia, industry and education jointly seek ways to motivate and inspire girls, for example by introducing them to programming at an early age. The recent COVID-19 pandemic has forced many such endeavours to move to an online setting. While the gender-dependent differences in programming courses have been studied previously, for example revealing that girls may feel safer in same-sex groups, much less is known about gender-specific differences in online programming courses. In order to investigate whether gender-specific differences can be observed in online courses, we conducted an online introductory programming course for Scratch, in which we observed the gender-specific characteristics of participants with respect to how they interact, their enjoyment, the code they produce, and the creativity exposed by their programs. Overall, we observed no significant differences between how girls participated in all-female vs. mixed groups, and girls generally engaged with the course more actively than boys. This suggests that online courses can be a useful means to avoid gender-dependent group dynamics. However, when encouraging creative freedom in programming, girls and boys seem to fall back to socially inherited stereotypical behavior also in an online setting, influencing the choice of programming concepts applied. This may inhibit learning and is a challenge that needs to be addressed independently of whether courses are held online.

cs.SE↗
arXiv2022

Automated Test Generation for Scratch Programs

The importance of programming education has lead to dedicated educational programming environments, where users visually arrange block-based programming constructs that typically control graphical, interactive game-like programs. The Scratch programming environment is particularly popular, with more than 70 million registered users at the time of this writing. While the block-based nature of Scratch helps learners by preventing syntactical mistakes, there nevertheless remains a need to provide feedback and support in order to implement desired functionality. To support individual learning and classroom settings, this feedback and support should ideally be provided in an automated fashion, which requires tests to enable dynamic program analysis. The Whisker framework enables automated testing of Scratch programs, but creating these automated tests for Scratch programs is challenging. In this paper, we therefore investigate how to automatically generate Whisker tests. This raises important challenges: First, game-like programs are typically randomised, leading to flaky tests. Second, Scratch programs usually consist of animations and interactions with long delays, inhibiting the application of classical test generation approaches. Evaluation on common programming exercises, a random sample of 1000 Scratch user programs, and the 1000 most popular Scratch programs demonstrates that our approach enables Whisker to reliably accelerate test executions, and even though many Scratch programs are small and easy to cover, there are many unique challenges for which advanced search-based test generation using many-objective algorithms is needed in order to achieve high coverage.

cs.SE↗