arXiv · 2611.18623
Evaluating the Robustness of Fact-Checking LLMs on Multilingual Evidence
Abstract
Automated fact-checking benchmarks typically pair claims with clean, curated evidence in high-resource languages, but real-world evidence is often incomplete, misleading, or written in a language where models are weaker. We evaluate the robustness of fact-checking LLMs for different languages. Building on EuroVerdict, we construct gold evidence sets for 190 claims in each of English, German, Greek, and Romanian, then apply four corruption strategies at two severity levels. We test LLMs on a set of claims, providing them with evidence from the sets above. A rubric-based LLM-as-a-judge scores reasoning quality. Verdict accuracy proves highly sensitive to both factors: Misleading edit cut Llama 3.1 8B's English accuracy from 80.00% to 16.84%, and losses widen as language resource level decreases. Machine-translating Greek and Romanian claims into English recovers much, but not all, of the low-resource gap. Judge scores decline more slowly than accuracy, suggesting that automated reasoning evaluation underestimates degradation. These results caution against deploying LLM fact-checkers on noisy multilingual evidence, and we release our evidence datasets and evaluation code to support future work.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Radhika Shah. 2026-09-17. Evaluating the Robustness of Fact-Checking LLMs on Multilingual Evidence. https://searcharxiv.com/abs/2611.18623
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.