Comment on Scientific production in the era of large language models

Abstract

Kusumegi et al. (2025) study whether researchers' preprint output rises after adopting large language models (LLMs), dating adoption as the first month in which at least one submitted abstract exceeds an LLM-detection threshold. We show that this treatment-timing rule is mechanically related to output. The probability that at least one paper is flagged in a month is increasing in the number of papers submitted in that month, so detected-adoption months are disproportionately high-output months. An event study centered on first detection can therefore display positive post-event dynamics even when the flagging rule contains no information about true LLM adoption, because the omitted pre-treatment period is selected from months with no prior detection. We demonstrate this in a simulation: with i.i.d. productivity and no causal effect, first-detection timing generates a spurious positive post-treatment path. We also replicate the stacked event study of Kusumegi et al. (2025) and show that three placebo exercises (random paper-level assignment, neutral keyword flags, and a pre-ChatGPT observation window) each produce a similarly positive post-treatment pattern.