Ahmed Binjahlan | Searcharxiv

arXiv2024

How do Software Engineering Researchers Use GitHub? An Empirical Study of Artifacts & Impact

Millions of developers share their code on open-source platforms like GitHub, which offer social coding opportunities such as distributed collaboration and popularity-based ranking. Software engineering researchers have joined in as well, hosting their research artifacts (tools, replication package & datasets) in repositories, an action often marked as part of the publications contribution. Yet a decade after the first such paper-with-GitHub-link, little is known about the fate of such repositories in practice. Do research repositories ever gain the interest of the developer community, or other researchers? If so, how often and why (not)? Does effort invested on GitHub pay off with research impact? In short: we ask whether and how authors engage in social coding related to their research. We conduct a broad empirical investigation of repositories from published work, starting with ten thousand papers in top SE research venues, hand-annotating their 3449 GitHub (and Zenodo) links, and studying 309 paper-related repositories in detail. We find a wide distribution in popularity and impact, some strongly correlated with publication venue. These were often heavily informed by the authors investment in terms of timely responsiveness and upkeep, which was often remarkably subpar by GitHubs standards, if not absent altogether. Yet we also offer hope: popular repositories often go hand-in-hand with well-citepd papers and achieve broad impact. Our findings suggest the need to rethink the research incentives and reward structure around research products requiring such sustained contributions.

cs.SE↗

arXiv2024

Language Models are Better Bug Detector Through Code-Pair Classification

Large language models (LLMs) such as GPT-3.5 and CodeLlama are powerful models for code generation and understanding. Fine-tuning these models comes with a high computational cost and requires a large labeled dataset. Alternatively, in-context learning techniques allow models to learn downstream tasks with only a few examples. Recently, researchers have shown how in-context learning performs well in bug detection and repair. In this paper, we propose code-pair classification task in which both the buggy and non-buggy versions are given to the model, and the model identifies the buggy ones. We evaluate our task in real-world dataset of bug detection and two most powerful LLMs. Our experiments indicate that an LLM can often pick the buggy from the non-buggy version of the code, and the code-pair classification task is much easier compared to be given a snippet and deciding if and where a bug exists.

cs.SE↗