arXiv subjects

Lucas Albuquerque

Publications and source records attributed to Lucas Albuquerque.

3 recordsLinked to original sources

arXiv2026

An Empirical Study of Foundation Models for Variability-Induced Compilation Errors in Configurable C Code

In configurable systems, conditional compilation can hide compilation errors under untested feature combinations. We investigate foundation models for detecting such errors and, in a controlled setting, restoring compilability in configurable C code. Study I evaluates GPT-OSS-20B on 5,000 synthetic snippets generated by ChatGPT-5.2 from 30 curated seeds and exhaustively compiled under all Boolean feature assignments; it also compares TypeChef and evaluates Gemini 3.6 Flash on a stratified sample. GPT-OSS-20B achieved 84.7% micro-precision and 52.1% micro-recall for affected configurations. Coverage depended on reporting style: presence conditions covered 99.4% of failing configurations, whereas explicit enumerations covered 29.5% under a prompt requesting only a minimal justifiable set. GPT-OSS-20B restored compilability for 1,930 of 2,665 faulty snippets (72.4%), while Gemini 3.6 Flash did so for 182 of 190 sampled faulty snippets (95.8%). A paired counterfactual audit found no evidence that an identified label-correlated #define property materially influenced GPT-OSS-20B's predictions. Study II evaluates Codex-GPT5.5 on 100 faulty file-level subjects from five mature configurable systems and reports target-fault-aligned problems in 94 subjects, including four of five historical bugs. Overall, foundation models can support localized detection, explanation, and triage, but should complement compiler-based and variability-aware analyses; compiler acceptance does not establish semantic correctness

cs.SE↗

arXiv2025

Code Generation with Small Language Models: A Codeforces-Based Study

Large Language Models (LLMs) demonstrate capabilities in code generation, potentially boosting developer productivity. However, their adoption remains limited by high computational costs, among other factors. Small Language Models (SLMs) present a lightweight alternative. While LLMs have been evaluated on competitive programming tasks, prior work often emphasizes metrics like Elo or pass rates, neglecting failure analysis. The potential of SLMs in this space remains underexplored. In this study, we benchmark three open SLMs - Llama-3.2-3B, Gemma-3-12B, and Phi-4-14B - across 280 Codeforces problems spanning Elo ratings from 800 to 2100 and covering 36 distinct topics. All models were tasked with generating Python solutions. Phi-4-14B achieved the best SLM performance with a pass@3 of 63.6%, nearing o3-mini-high (86.8%). Combining Python and C++ outputs increased Phi-4-14B's pass@6 to 73.6%. A qualitative analysis revealed some failures stemmed from minor implementation issues rather than reasoning flaws.

cs.SE↗

arXiv2024

Evaluating the Capability of LLMs in Identifying Compilation Errors in Configurable Systems

Compilation is an important process in developing configurable systems, such as Linux. However, identifying compilation errors in configurable systems is not straightforward because traditional compilers are not variability-aware. Previous approaches that detect some of these compilation errors often rely on advanced techniques that require significant effort from programmers. This study evaluates the efficacy of Large Language Models (LLMs), specifically ChatGPT4, Le Chat Mistral and Gemini Advanced 1.5, in identifying compilation errors in configurable systems. Initially, we evaluate 50 small products in C++, Java, and C languages, followed by 30 small configurable systems in C, covering 17 different types of compilation errors. ChatGPT4 successfully identified most compilation errors in individual products and in configurable systems, while Le Chat Mistral and Gemini Advanced 1.5 detected some of them. LLMs have shown potential in assisting developers in identifying compilation errors in configurable systems.

cs.SE↗