arXiv · 2611.20033
The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering
Abstract
Context: Large Language Models (LLMs) are increasingly embedded in empirical software engineering (ESE) workflows, supporting tasks such as classification, coding, extraction, synthesis, and evaluation. Their probabilistic and opaque behavior, however, introduces methodological concerns that may affect empirical rigor. Aims: This paper investigates threats to validity introduced by LLM-assisted empirical workflows and examines how LLM interactions influence empirical studies. Method: We conducted a secondary analysis of 50 primary studies comprising 69 LLM-assisted research tasks identified through a systematic literature review. We examined the extent to which LLM-related threats were reported, how these threats were characterized, and how they related to established validity dimensions. Results: LLM-related threats remain substantially underreported. We identified recurring threats including classification errors, prompt-dependent interpretation, workflow instability, evaluator bias, downstream propagation of outputs, and model dependence. These threats can affect multiple validity dimensions depending on where and how LLM outputs enter the workflow. Conclusion: We introduce the concept of hidden methodological layer as a conceptual lens for understanding how LLMs influence empirical software engineering workflows. We argue that rigorous LLM-assisted ESE requires making prompts, model configurations, interaction protocols, validation procedures, and downstream reuse of LLM-generated outputs visible as part of the research method.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Victória Gomes, Fabio Palomba, Robert Feldt, Thomas Zimmermann, David Lo, Rodrigo Spinola. 2026-11-15. The Hidden Methodological Layer: Uncovering Threats to Validity in LLM-Assisted Empirical Software Engineering. https://searcharxiv.com/abs/2611.20033
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.