Searcharxiv⌕ Search

arXiv subjects

Jeffrey C. Carver

Publications and source records attributed to Jeffrey C. Carver.

At least 19 recordsLinked to original sources

Operationalizing Research Software for Supply Chain Security

Empirical studies of research software are hard to compare because the literature operationalizes ``research software'' inconsistently. Motivated by the research software supply chain (RSSC) and its security risks, we introduce an RSSC-oriented taxonomy that makes scope and operational boundaries explicit for empirical research software security studies. We conduct a targeted scoping review of recent repository mining and dataset construction studies, extracting each work's definition, inclusion criteria, unit of analysis, and identification heuristics. We synthesize these into a harmonized taxonomy and a mapping that translates prior approaches into shared taxonomy dimensions. We operationalize the taxonomy on a large community-curated corpus from the Research Software Encyclopedia (RSE), producing an annotated dataset, a labeling codebook, and a reproducible labeling pipeline. Finally, we apply OpenSSF Scorecard as a preliminary security analysis to show how repository-centric security signals differ across taxonomy-defined clusters and why taxonomy-aware stratification is necessary for interpreting RSSC security measurements.

cs.SE↗

Peer Code Review in Research Software Development: The Research Software Engineer Perspective

Background: Research software is crucial for enabling research discoveries and supporting data analysis, simulation, and interpretation across domains. However, evolving requirements, complex inputs, and legacy dependencies hinder the software quality and maintainability. While peer code review can improve software quality, its adoption by research software engineers (RSEs) remains unexplored. Aims: This study explores RSE perspectives on peer code review, focusing on their practices, challenges, and potential improvements. Building on prior work, it aims to uncover how RSEs insights differ from those of other research software developers and identify factors that can enhance code review adoption in this domain. Method: We surveyed RSEs to gather insights into their perspectives on peer code review. The survey design aligned with previous research to enable comparative analysis while including additional questions tailored to RSEs. Results: We received 61 valid responses from the survey. The findings align with prior research while uncovering unique insights about the challenges and practices of RSEs compared to broader developer groups. Conclusions: Peer code review is vital in improving research software's quality, maintainability, and reliability. Despite the unique challenges RSEs face, addressing these through structured processes, improved tools, and targeted training can enhance peer review adoption and effectiveness in research software development.

cs.SE↗

ChatGPT in Introductory Programming: Counterbalanced Evaluation of Code Quality, Conceptual Learning, and Student Perceptions

Background: Large language models (LLMs) such as ChatGPT are increasingly used in introductory programming courses to provide real-time code generation, debugging, and explanations. While these tools can boost productivity and code quality, concerns remain about over-reliance and potential impacts on conceptual learning. Objective: To investigate how ChatGPT access affects code quality, conceptual understanding, task completion times, and student perceptions in a CS1 course. Methods: We conducted a counterbalanced, quasi-experimental study in which students alternated between ChatGPT and non-ChatGPT conditions across two programming assignments in C (functions and structures). We evaluated their code submissions using multidimensional rubrics, conceptual post-surveys, and task completion time. Results: Students who had access to ChatGPT produced significantly higher rubric scores for code quality and completed tasks in less time compared to those without access. However, gains in conceptual understanding were mixed, lower for the functions topic but higher for the structures topic. Students reported positive experiences with ChatGPT, citing its value for debugging and practice, while expressing concerns about accuracy and long-term skill development. Conclusions: ChatGPT can enhance code quality and efficiency for novice programmers, but may not uniformly improve conceptual understanding. Structured integration and complementary instructional strategies are recommended to foster independent problem-solving skills.

cs.SE↗

Enhancing Software Testing Education: Understanding Where Students Struggle

Effective software testing is critical for producing reliable and secure software, yet many computer science students struggle to master the foundational concepts required to construct comprehensive test suites. While automated feedback tools are widely used to support student learning, it remains unclear which testing concepts are most frequently misunderstood and how these misunderstandings are reflected in students' test suite revisions. This study examines the specific testing concepts that lead students to make ineffective changes, those that fail to improve code coverage, during test suite development. Leveraging an automated feedback tool in a senior-level software testing course, we analyzed student submissions from two assignments to identify prevalent conceptual gaps and patterns of unproductive modification. Our results reveal that decision coverage and exception handling are persistent challenges, and that students most often make superficial or method-level changes that do not enhance coverage. These findings provide actionable insights for educators, researchers, and tool designers. By pinpointing the concepts that most often contribute to poor testing outcomes, we can refine feedback systems, target instruction to address persistent misconceptions, and more effectively support students in developing robust, maintainable test suites.

cs.SE↗

Ten Simple Rules for Catalyzing Collaborations and Building Bridges between Research Software Engineers and Software Engineering Researchers

In the evolving landscape of scientific and scholarly research, effective collaboration between Research Software Engineers (RSEs) and Software Engineering Researchers (SERs) is pivotal for advancing innovation and ensuring the integrity of computational methodologies. This paper presents ten strategic guidelines aimed at fostering productive partnerships between these two distinct yet complementary communities. The guidelines emphasize the importance of recognizing and respecting the cultural and operational differences between RSEs and SERs, proactively initiating and nurturing collaborations, and engaging within each other's professional environments. They advocate for identifying shared challenges, maintaining openness to emerging problems, ensuring mutual benefits, and serving as advocates for one another. Additionally, the guidelines highlight the necessity of vigilance in monitoring collaboration dynamics, securing institutional support, and defining clear, shared objectives. By adhering to these principles, RSEs and SERs can build synergistic relationships that enhance the quality and impact of research outcomes.

cs.SE↗

Testing Research Software: An In-Depth Survey of Practices, Methods, and Tools

Context: Research software is essential for developing advanced tools and models to solve complex research problems and drive innovation across domains. Therefore, it is essential to ensure its correctness. Software testing plays a vital role in this task. However, testing research software is challenging due to the software's complexity and to the unique culture of the research software community. Aims: Building on previous research, this study provides an in-depth investigation of testing practices in research software, focusing on test case design, challenges with expected outputs, use of quality metrics, execution methods, tools, and desired tool features. Additionally, we explore whether demographic factors influence testing processes. Method: We survey research software developers to understand how they design test cases, handle output challenges, use metrics, execute tests, and select tools. Results: Research software testing varies widely. The primary challenges are test case design, evaluating test quality, and evaluating the correctness of test outputs. Overall, research software developers are not familiar with existing testing tools and have a need for new tools to support their specific needs. Conclusion: Allocating human resources to testing and providing developers with knowledge about effective testing techniques are important steps toward improving the testing process of research software. While many industrial testing tools exist, they are inadequate for testing research software due to its complexity, specialized algorithms, continuous updates, and need for flexible, custom testing approaches. Access to a standard set of testing tools that address these special characteristics will increase level of testing in research software development and reduce the overhead of distributing knowledge about software testing.

cs.SE↗

Thoughts on Learning Human and Programming Languages

This is a virtual dialog between Jeffrey C. Carver and Daniel S. Katz on how people learn programming languages. It's based on a talk Jeff gave at the first US-RSE Conference (US-RSE'23), which led Dan to think about human languages versus computer languages. Dan discussed this with Jeff at the conference, and this discussion continued asynchronous, with this column being a record of the discussion.

cs.SE↗

Human Error Management in Requirements Engineering: Should We Fix the People, the Processes, or the Environment?

Context: Software development is human-centric and vulnerable to human error. Human errors are errors in the human thought process. To ensure software quality, practitioners must understand how to manage these human errors. Organizations often change the requirements engineering process to prevent human errors from occurring or to mitigate the harm caused when those errors do occur. While there are studies on human error management in other disciplines, research on the prevention and mitigation of human errors in software engineering, and requirements engineering specifically, are limited. The software engineering studies do not provide strong results about the types of changes that are most effective in requirements engineering. Objective: The goal of this paper is to develop a taxonomy of human error prevention and mitigation strategies based on data from requirements engineering professionals. Method: We performed a qualitative analysis of two practitioner surveys on requirements engineering practices to identify and classify strategies for the prevention and mitigation of human errors. Results: We organized the human error management strategies into a taxonomy based on whether they primarily affect People, Processes, or the Environment. Inside each high-level category, we further organized the strategies into low-level classes. More than 50% of the reported strategies require a change in Process, 23% require a change in Environment, 21% require a change in People, with the remaining 5% too ambiguous to classify. In addition, more than 50\% of the strategies focus on Management activities. Conclusions: The Human Error Management Taxonomy provides a systematic classification and organization of strategies for prevention and mitigation of human errors in requirements engineering. This systematic organization provides a foundation upon which research can build.

cs.SE↗

Community Smells -- The Sources of Social Debt: A Systematic Literature Review

Context: Social debt describes the accumulation of unforeseen project costs (or potential costs) from sub-optimal software development processes. Community smells are sociotechnical anti-patterns and one source of social debt that impact software teams, development processes, outcomes, and organizations. Objective: To provide an overview of community smells based on published literature, and describe future research. Method: We conducted a systematic literature review (SLR) to identify properties, understand origins and evolution, and describe the emergence of community smells. This SLR explains the impact of community smells on teamwork and team performance. Results: We include 25 studies. Social debt describes the impacts of poor socio-technical decisions on work environments, people, software products, and society. For each of the 30 identified community smells, we provide a description, management approaches, organizational strategies, and mitigation effectiveness. We identify five groups of management approaches: organizational strategies, frameworks, models, tools, and guidelines. We describe 11 properties of community smells. We develop the Community Smell Stages Framework to concisely describe the origin and evolution of community smells. We describe the causes and effects for each community smell. We identify and describe 8 types of causes and 11 types of effects for community smells. Finally, we provide 8 Sankey diagrams that offer insights into threats the community smells pose to teamwork factors and team performance. Conclusion: Community smells explain the influence work conditions have on software developers. The literature is scarce and focuses on a small number of community smells. Thus, community smells still need more research. This review organizes the state of the art about community smells and provides motivation for future research along with educational material.

cs.SE↗

Testing Research Software: A Survey

Background: Research software plays an important role in solving real-life problems, empowering scientific innovations, and handling emergency situations. Therefore, the correctness and trustworthiness of research software are of absolute importance. Software testing is an important activity for identifying problematic code and helping to produce high-quality software. However, testing of research software is difficult due to the complexity of the underlying science, relatively unknown results from scientific algorithms, and the culture of the research software community. Aims: The goal of this paper is to better understand current testing practices, identify challenges, and provide recommendations on how to improve the testing process for research software development. Method: We surveyed members of the research software developer community to collect information regarding their knowledge about and use of software testing in their projects. Results: We analysed 120 responses and identified that even though research software developers report they have an average level of knowledge about software testing, they still find it difficult due to the numerous challenges involved. However, there are a number of ways, such as proper training, that can improve the testing process for research software. Conclusions: Testing can be challenging for any type of software. This difficulty is especially present in the development of research software, where software engineering activities are typically given less attention. To produce trustworthy results from research software, there is a need for a culture change so that testing is valued and teams devote appropriate effort to writing and executing tests.

cs.SE↗

Assessing Expert System-Assisted Literature Reviews With a Case Study

Given the large number of publications in software engineering, frequent literature reviews are required to keep current on work in specific areas. One tedious work in literature reviews is to find relevant studies amongst thousands of non-relevant search results. In theory, expert systems can assist in finding relevant work but those systems have primarily been tested in simulations rather than in application to actual literature reviews. Hence, few researchers have faith in such expert systems. Accordingly, using a realistic case study, this paper assesses how well our state-of-the-art expert system can help with literature reviews. The assessed literature review aimed at identifying test case prioritization techniques for automated UI testing, specifically from 8,349 papers on IEEE Xplore. This corpus was studied with an expert system that incorporates an incrementally updated human-in-the-loop active learning tool. Using that expert system, in three hours, we found 242 relevant papers from which we identified 12 techniques representing the state-of-the-art in test case prioritization when source code information is not available. These results were then validated by six other graduate students manually exploring the same corpus. Without the expert system, this task would have required 53 hours and would have found 27 additional papers. That is, our expert system achieved 90% recall with 6% of the human effort cost when compared to a conventional manual method. Significantly, the same 12 state-of-the-art test case prioritization techniques were identified by both the expert system and the manual method. That is, the 27 papers missed by the expert system would not have changed the conclusion of the literature review. Hence, if this result generalizes, it endorses the use of our expert system to assist in literature reviews.

cs.SE↗

Training Computing Educators to Become Computing Education Researchers

The computing education community endeavors to consistently move forward, improving the educational experience of our students. As new innovations in computing education practice are learned and shared, however, these papers may not exhibit the desired qualities that move simple experience reports to true Scholarship of Teaching and Learning (SoTL). We report on our six years of experience in running professional development for computing educators in empirical research methods for social and behavioral studies in the classroom. Our goal is to have a direct impact on instructors who are in the beginning stages of transitioning their educational innovations from anecdotal to empirical results that can be replicated by instructors at other institutions. To achieve this, we created a year-long mentoring experience, beginning with a multi-day workshop on empirical research methods during the summer, followed by regular mentoring sessions with participants, and culminating in a follow-up session at the following year's SIGCSE Technical Symposium. From survey results and as evidenced by eventual research results and publications from participants, we believe that our method of structuring empirical research professional development was successful and could be a model for similar programs in other areas.

cs.CY↗

Developers Perception of Peer Code Review in Research Software Development

Background: Research software is software developed by and/or used by researchers, across a wide variety of domains, to perform their research. Because of the complexity of research software, developers cannot conduct exhaustive testing. As a result, researchers have lower confidence in the correctness of the output of the software. Peer code review, a standard software engineering practice, has helped address this problem in other types of software. Aims: Peer code review is less prevalent in research software than it is in other types of software. In addition, the literature does not contain any studies about the use of peer code review in research software. Therefore, through analyzing developers perceptions, the goal of this work is to understand the current practice of peer code review in the development of research software, identify challenges and barriers associated with peer code review in research software, and present approaches to improve the peer code review in research software. Method: We conducted interviews and a community survey of research software developers to collect information about their current peer code review practices, difficulties they face, and how they address those difficulties. Results: We received 84 unique responses from the interviews and surveys. The results show that while research software teams review a large amount of their code, they lack formal process, proper organization, and adequate people to perform the reviews. Conclusions: Use of peer code review is promising for improving the quality of research software and thereby improving the trustworthiness of the underlying research results. In addition, by using peer code review, research software developers produce more readable and understandable code, which will be easier to maintain.

cs.SE↗

A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature

Computing Education Research (CER) is critical for supporting the increasing number of students who need to learn computing skills. To systematically advance knowledge, publications must be clear enough to support replications, meta-analyses, and theory-building. The goal of this study is to characterize the reporting of empiricism in CER literature by identifying whether publications include information to support replications, meta-analyses, and theory building. The research questions are: RQ1) What percentage of papers in CER venues have empirical evaluation? RQ2) What are the characteristics of the empirical evaluation? RQ3) Do the papers with empirical evaluation follow reporting norms (both for inclusion and for labeling of key information)? We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues: SIGCSE TS, ICER, ITiCSE, TOCE, and CSE. We developed and applied the CER Empiricism Assessment Rubric. Over 80% of papers had some form of empirical evaluation. Quantitative evaluation methods were the most frequent. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other data set or baseline. Many papers lacked properly reported research objectives, goals, research questions, or hypotheses, description of participants, study design, data collection, and threats to validity. CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.

cs.CY↗

Understanding Peer Review of Software Engineering Papers

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect. We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper. We first conducted a series of in-person interviews with well-respected reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts. We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: 45% of award-winners are reviewing 20+ conference papers a year, while 28% of non-award winners conduct that many. 88% of reviewers are taking more than two hours on journal reviews. We also report on qualitative results. To write a good review, the important criteria were it should be factual and helpful, ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from those with overly grandiose claims. The main recommendation for authors is to make the contribution of the work very clear in their paper. In addition, reviewers viewed data availability and its consistency as being important.

cs.SE↗

Addressing Research Software Sustainability via Institutes

Research software is essential to modern research, but it requires ongoing human effort to sustain: to continually adapt to changes in dependencies, to fix bugs, and to add new features. Software sustainability institutes, amongst others, develop, maintain, and disseminate best practices for research software sustainability, and build community around them. These practices can both reduce the amount of effort that is needed and create an environment where the effort is appreciated and rewarded. The UK SSI is such an institute, and the US URSSI and the Australian AuSSI are planning to become institutes, and this extended abstract discusses them and the strengths and weaknesses of this approach.

cs.SE↗

Sustaining Research Software via Research Software Engineers and Professional Associations

Research software is a class of software developed to support research. Today a wealth of such software is created daily in universities, government, and commercial research enterprises worldwide. The sustainability of this software faces particular challenges due, at least in part, to the type of people who develop it. These Research Software Engineers (RSEs) face challenges in developing and sustaining software that differ from those faced by the developers of traditional software. As a result, professional associations have begun to provide support, advocacy, and resources for RSEs. These benefits are critical to sustaining RSEs, especially in environments where their contributions are often undervalued and not rewarded. This paper focuses on how professional associations, such as the United States Research Software Engineer Association (US-RSE), can provide this.

cs.SE↗

How do Practitioners Perceive the Relevance of Requirements Engineering Research?

The relevance of Requirements Engineering (RE) research to practitioners is vital for a long-term dissemination of research results to everyday practice. Some authors have speculated about a mismatch between research and practice in the RE discipline. However, there is not much evidence to support or refute this perception. This paper presents the results of a study aimed at gathering evidence from practitioners about their perception of the relevance of RE research and at understanding the factors that influence that perception. We conducted a questionnaire-based survey of industry practitioners with expertise in RE. The participants rated the perceived relevance of 435 scientific papers presented at five top RE-related conferences. The 153 participants provided a total of 2,164 ratings. The practitioners rated RE research as essential or worthwhile in a majority of cases. However, the percentage of non-positive ratings is still higher than we would like. Among the factors that affect the perception of relevance are the research's links to industry, the research method used, and respondents' roles. The reasons for positive perceptions were primarily related to the relevance of the problem and the soundness of the solution, while the causes for negative perceptions were more varied. The respondents also provided suggestions for future research, including topics researchers have studied for decades, like elicitation or requirement quality criteria.

cs.SE↗