When Language Models Misinterpret Uncertainty in Earth System Mechanism Attribution

Abstract

Mechanistic attribution in Earth system science requires quantifying uncertainty and communicating its implications. Here, we develop an uncertainty-aware mechanistic attribution pipeline that constructs probabilistic fingerprints from ensemble estimates, including covariance-aware separation of competing mechanisms, and an evidence ladder that systematically varies the information provided to language models (LLMs). We evaluate this framework in an idealized stochastic system and a coupled El Niño model using controlled interventions and ensemble data assimilation. In the idealized system, numerical separation increases GPT-4o's intervention-label agreement from 0.80 to 0.98, but not GPT-4o-mini's. In the coupled system, both LLMs identify seven of eight fingerprint-ambiguous events when given full fingerprints with verbal interpretations; GPT-4o's false ambiguity reports decrease from 10/82 to 1/82, without consistent gains in intervention-label agreement. For eight historical ENSO events, the uncertainty-aware condition agrees with the model-inferred mechanism labels in seven cases. These findings distinguish uncertainty assessment from its communication.